update docs

docs/10_Installation.md

+# Installation
+
+## Extract files
+
+On a server the client must be deployed i.e. in `/opt/deployment-client/`
+
+Use git clone if you feel familiar with git. Otherwise download the archive and extract it.
+
+The files should be owned by root:root because the main script will be executed as root
+and it wants to chown the extracted files to your application user.
+
+The result in your target directory is:
+
+```text
+.
+├── bin
+│   ├── create_config.sh
+│   ├── getfile.sh
+│   ├── getfile.sh.cfg
+│   ├── getfile.sh.cfg.dist
+│   └── preinstall_cleanup.sh
+├── deploy_app.sh
+├── docs
+│   └── ...
+├── profiles
+│   └── example
+│       ├── config.sh.dist
+│       ├── replace.txt
+│       ├── tasks_config.sh
+│       ├── tasks_postinstall.sh
+│       └── tasks_preinstall.sh
+└── readme.md
+```
+
+## Create directory for logfiles
+
+Then create a folder for log data. As root:
+
+```shell
+mkdir -p /var/log/imldeployment-client/
+```

docs/20_Configuration.md

+# Global configuration
+
+## Set up access to package server
+
+This setting is for all projects on the server. It has to be done once.
+
+* in ./bin directory copy getfile.sh.cfg.dist to getfile.sh.cfg
+* edit getfile.sh.cfg and define software endpoint and set the phase:
+
+```bash
+IMLCI_URL=https://software.example.com
+IMLCI_PKG_SECRET=put-secret-here
+IMLCI_PHASE=preview
+```
+
+name              | type   | description
+---               | ---    | ---
+IMLCI_PHASE       | string | name of the phase; one of preview|stage|live
+IMLCI_PKG_SECRET  | string | Secret to download software packages; recommendation: do NOT set a shared secret for all applications but an individual one
+IMLCI_URL         | string | Url to server with software packages
+
+# Set up software rollout
+
+The deployment client can handle multipe profiles. One profile is for one application.
+
+In the subdir ./profile is an example profile:
+
+```txt
+.
+└── example
+    ├── config.sh.dist
+    ├── replace.txt
+    ├── tasks_config.sh
+    ├── tasks_postinstall.sh
+    └── tasks_preinstall.sh
+```
+
+Files:
+
+File                  | Description
+---                   | ---
+config.sh.dist        | Configuration file for the application rollout
+replace*.txt          | Files containing replacement data. You can create multiple files eg. for different responsible groups like sysadmin, developers
+tasks_config.sh       | 
+tasks_postinstall.sh  | 
+tasks_preinstall.sh   | 
+
+## Create a profile
+
+* Create a subdirectory in ./profiles/ for each product
+* The example subdir gives an orientation and can be copied, i.e. `cp -r example myapp`
+* create a config file named `./profiles/myapp/config.sh` (copy the config.sh.dist from example profile)
+
+```bash
+# my install dir
+installdir=/var/www/myapp
+
+# fileowner
+# appowner="user:www-data"
+
+# ----- settings for CI server software package 
+
+
+IMLCI_PROJECT=id-in-ci-server
+
+# override global value 
+# IMLCI_PHASE=preview
+
+# cleanup after pre install tasks and bevore extracting data
+# set both to 0 .. or only one of them to 1
+cleanup_preview=0
+cleanup_force=0
+```
+
+Config variables:
+
+name            | type   | description
+---             | ---    | ---
+installdir      | string | target directory of your application
+appowner        | string | if not empty a chown -R will be applied in target directory by `chwon -R ${appowner} ${installdir}`; appowner is the parameter behind -R. It is something like "myuser." or "myuser:mygroup". For a web application it should be the user of your webservice (www-data/ apache/ nginx). The command chown requires to run the deploy script as root.
+IMLCI_PROJECT   | string | Project id in IML CI server
+IMLCI_PHASE     | string | optional: override the global IMLCI_PHASE in ./bin/getfile.sh.cfg; it is one of preview\|stage\|live
+cleanup_preview | 0 or 1 | Cleanup preview - shows diff between downloaded TGZ and ${installdir}.
+cleanup_force   | 0 or 1 | Run cleanup: it deletes all files in target directory that aren't in the last downloaded tgz. To keep runtime data like logs or uploads you can add a file `.keep` in the directory.
+
+Make a testrun: `./deploy_app.sh` in application root.
+It should download the software package and extract it and install it into you `${installdir}`.
+
+## Add hooks
+
+If needed you can create hook scripts. The working directory is `${installdir}` that you can use relative pathes to point to your files in the the extracted sources.
+
+To access other ressources you can user these variables:
+
+```txt
+${selfdir}     application root of deployment scripts
+${profiledir}  project config dir i.e. [path]/profiles/myapp
+```
+
+For hooks you can create files with pre defined names. A hook script must have executable rights. You get a hint message if it does not exist or has no x permission. A missing hook script does not result in an error.
+
+* profiles/myapp/tasks_preinstall.sh - do something before extracting the archive.
+* profiles/myapp/tasks_config.sh - replace config files (see below)
+* profiles/myapp/tasks_postinstall.sh - do postinstall actions before finishing
+
+## Create configs for an application: tasks_config.sh
+
+The script `./bin/create_config.sh` can read config templates and create an output file.
+
+You need to reference the template, output file and a file for replacement data.
+
+```bash
+# ----------------------------------------------------------------------
+# TASKS :: GENERATE CONFIGS
+# ----------------------------------------------------------------------
+
+#               create_config.sh    template_file                                 target_file                       replacements (can be multiple files)
+#               |                   |                                             |                                 |
+#               v                   v                                             v                                 v
+"${selfdir}/bin/create_config.sh"   hooks/templates/mytemplate.erb                config/target.php                 ${profiledir}/replace.txt
+```
+
+During execution the working directory is the target directory of your application.
+
+* "${selfdir}/bin/create_config.sh" is the script that replaces config data in a template.
+* In your software repsoitory put a template into ./hooks/templates/. It will be found if the archive was extracted.
+* the target file is relative to your application root.
+* files with replacement data.
+
+A sample replacement file is profiles/example/replace.txt:
+
+```txt
+# ----------------------------------------------------------------------
+# REPLACEMENTS
+# ----------------------------------------------------------------------
+#
+# SYNTAX:
+# key=value
+#
+# - no spaces around first "="
+# - value can contain any char, no quoting
+# ----------------------------------------------------------------------
+
+dbhost=localhost
+dbuser=fred
+dbpassword=vom-jupiter
+dbport=3306
+
+api-key=12345678
+
+# ----------------------------------------------------------------------
+```
+
+The idea for several replacent files is that different groups can set
+their replacement data:
+
+* sysadmins add connections and credentials to needed services/ database connections
+* developers add api keys and other internal values
+* ...
+
+## Pre and post install actions
+
+Example: A simple tasks_postinstall.sh can contain the start of a script that is delivered in the tgz archive.
+
+```bash
+# ----------------------------------------------------------------------
+# TASKS :: POST INSTALL ACTIONS
+# ----------------------------------------------------------------------
+hooks/ondeploy
+```

docs/30_How_does_it_work.md

+# Execution plan
+
+If fully configured the deployment script executes the following steps in that fixed sequence:
+
+* loop over profiles ... and per profile ...
+* set profile data by sourcing [profile]/config.sh
+* download Software archive
+* detect if download is newer than the last one
+* jump into installation dir of your application
+* optional: execute pre installation tasks (1)
+* optional: run cleanup (1)
+* extract software archive (1)
+* create config files
+* optional: execute post installation tasks (1)
+* optional: execute post config change tasks (1) or (2)
+
+(1) if a new software version was downloaded
+(2) if one of the config files was created or changed
+
+## Graphical view
+
+```mermaid
+graph TD
+    START(START) --> Profile
+    Profile(source profilename/config.sh) --> DL
+
+    DL(Download software) --> IsNew
+    IsNew{new download?}
+    IsNew -->|no| new0(Update only flag=0)
+    IsNew -->|yes| new1(new installation flag=1)
+
+
+    new1 --> preInstall
+    preInstall(optional: Pre install tasks) --> preCleanup
+    preCleanup(optional: Cleanup files) --> extract
+    extract(extract software archive) --> cont1
+    
+    new0 --> cont1
+
+    cont1(( )) --> cfg
+    cfg(Create config files) --> DoPostInstall{new download?}
+    DoPostInstall --> |yes| PostInstall
+    DoPostInstall --> |no| cont2
+    PostInstall(Run post install script) --> cont2
+
+    cont2(( )) --> DoPostChange{Changed config?}
+    DoPostChange --> |yes| PostConfig
+    DoPostChange --> |no| cont3
+    PostConfig(Run post change script) --> cont3
+
+    cont3(( )) --> anotherProfile{Another profile?}
+    anotherProfile --> |yes|Profile
+    anotherProfile -->|no|END
+
+    END(END)
+
+```

docs/40_Usage.md

+
+# Scripts
+
+## deploy_app.sh
+
+This is the main deployment script.
+
+```txt
+./deploy_app.sh [PROFILENAME]
+```
+
+If you start it without parameter it will loop over all existing profiles.
+You can add an existing profile name to limit the execution to that profile only.
+
+## ./bin/create_config.sh
+
+The script is used to read a template (*.erb) to replace simple placeholders
+by regex `<\%\=\ *\@replace\[\"$key\"\]\ *\%>` with a value coming from one or more
+replacement data files. The output will be written to a target file.
+
+```txt
+
+===== IML DEPLOYMENT :: replace variables in erb syntax =====
+
+template    : 
+output      : 
+replacements: replace*.txt
+
+ERROR: missing params.
+create_config.sh TEMPLATE-FILE OUTFILE [optional: REPLACE-DATA-FILE]
+```
+
+## ./bin/getfile.sh
+
+With getfile.sh you can access the software archive.
+
+* read phase names
+* read ci project ids available in a given phase
+* list files of a given projects of a phase
+* download a single file to a custom target file
+* download ALL files to current working directory
+
+```txt
+SYNTAX:
+
+  getfile.sh [OPTIONS]
+
+OPTIONS:
+
+  -c CFGFILE  load custom config file after defaults in getfile.sh.cfg
+  -d          enable debug infos
+  -e PHASE    phase; overrides env variable IMLCI_PHASE
+  -f FILE     filename to get (without path); overrides env variable IMLCI_FILE
+  -l ITEM     list
+  -o OUTFILE  optional output file
+  -p PROJECT  ci project id; overrides env variable IMLCI_PROJECT
+  -s SECRET   override secret in IMLCI_PKG_SECRET
+  -u URL      URL of iml ci server without trailing /; overrides env variable IMLCI_URL
+
+VALUES:
+
+  CFGFILE     custom config file. It is useful to handle files of different 
+              projects on a server.
+  PHASE       is a phase of the ci server; one of preview|stage|live
+  FILE        is a filename without path that was created by ci server.
+  OUTFILE     Output file. It can countain a path. If none is given the filename
+              will be taken from FILE and stored in current directory
+  PROJECT     project id of the ci server
+  SECRET      secret to access project data on package server. Your given secret
+              must match the secret on package server to get access to any url.
+  ITEM        type what to list; one of phases|projects|files
+              To list projects a phase must be set.
+              To list files a phase and a project must be set.
+
+DEFAULTS:
+
+  You don't need to set all values by command line. Use a config to set defaults
+  ./getfile.sh.cfg
+
+EXAMPLES:
+
+  If url, secret, project and phase are set in the config you can operate by
+  setting the filename to request.
+
+  getfile.sh -f FILE 
+    downloads FILE to the current dir.
+
+  getfile.sh -f FILE -o my-own-filename.tgz 
+    downloads FILE as my-own-filename.tgz
+
+  getfile.sh -f ALL 
+    there is a special file ALL; it fetches all filenames by executing a directory 
+    listing and then downloads all remote files with their original name
+
+  getfile.sh -e preview -l projects
+    list existing projects in phase preview
+
+  getfile.sh -l files
+    list existing files of current project
+
+  Remark: The directory listing can be turned off on the package server and
+  results in a 403 status.
+```

docs/_index.md

@@ -15,262 +15,28 @@ GNU GPL 3.0
 
 URL: <https://git-repo.iml.unibe.ch/iml-open-source/imldeployment-client/>
 
-## Installation
+## Requirements
 
-On a server the client must be deployed i.e. in `/opt/deployment-client/`
+A few Linux standard tools are required. It should run on any GNU Linux.
 
-Use git clone if you feel familiar with git. Otherwise download the archive and extract it.
+* Bash
+* Curl
+* sed
 
-## Execution plan
+Tested environments:
 
-If fully configured the deployment script executes the following steps in that fixed sequence:
+* CentOS
+* Debian
+* Manjaro
+* Ubuntu
 
-* loop over profiles ... and per profile ...
-* set profile data by sourcing [profile]/config.js
-* download Software archive
-* detect if download is newer than the last one
-* jump into installation dir of your application
-* optional: execute pre installation tasks (1)
-* optional: run cleanup (1)
-* extract software archive (1)
-* create config files
-* optional: execute post installation tasks (1)
-* optional: execute post config change tasks (1) or (2)
+## Features
 
-(1) if a new software version was downloaded
-(2) if one of the config files was created or changed
-
-## Set up access to package server
-
-This setting is for all projects on the server. It has to be done once.
-
-* in ./bin directory copy getfile.sh.cfg.dist to getfile.sh.cfg
-* edit getfile.sh.cfg and define software endpoint and set the phase:
-
-```bash
-IMLCI_URL=https://software.example.com
-IMLCI_PKG_SECRET=put-secret-here
-IMLCI_PHASE=preview
-```
-
-## Set up software rollout
-
-### Create a profile
-
-* Create a subdirectory in ./profiles/ for each rollout
-* The example subdir gives an orientation and can be copied, i.e. `cp -r example myapp`
-* create a config file named `./profiles/myapp/config.sh` (copy the config.sh.dist from example profile)
-
-```bash
-# my install dir
-installdir=/var/www/myapp
-
-# fileowner
-# appowner="user:www-data"
-
-# ----- settings for CI server software package 
-
-
-IMLCI_PROJECT=id-in-ci-server
-
-# override global value 
-# IMLCI_PHASE=preview
-
-# cleanup after pre install tasks and bevore extracting data
-# set both to 0 .. or only one of them to 1
-cleanup_preview=0
-cleanup_force=0
-```
-
-Config variables:
-
-name            | type   | description
----             | ---    | ---
-installdir      | string | target directory of your application
-appowner        | string | if not empty a chown -R will be applied in target directory by `chwon -R ${appowner} ${installdir}`; appowner is the parameter behind -R. It is something like "myuser." or "myuser:mygroup". For a web application it should be the user of your webservice (www-data/ apache/ nginx). The command chown requires to run the deploy script as root.
-IMLCI_PROJECT   | string | Project id in IML CI server
-IMLCI_PHASE     | string | optional: override the global IMLCI_PHASE in ./bin/getfile.sh.cfg; it is one of preview\|stage\|live
-cleanup_preview | 0 or 1 | Cleanup preview - shows diff between downloaded TGZ and ${installdir}.
-cleanup_force   | 0 or 1 | Run cleanup: it deletes all files in target directory that aren't in the last downloaded tgz. To keep runtime data like logs or uploads you can add a file `.keep` in the directory.
-
-Make a testrun: `./deploy_app.sh` in application root.
-It should download the software package and extract it and install it into you `${installdir}`.
-
-### Add hooks
-
-If needed you can create hook scripts. The working directory is `${installdir}` that you can use relative pathes to point to your files in the the extracted sources.
-
-To access other ressources you can user these variables:
-
-```txt
-${selfdir}     application root of deployment scripts
-${profiledir}  project config dir i.e. [path]/profiles/myapp
-```
-
-For hooks you can create files with pre defined names. A hook script must have executable rights. You get a hint message if it does not exist or has no x permission. A missing hook script does not result in an error.
-
-* profiles/myapp/tasks_preinstall.sh - do something before extracting the archive.
-* profiles/myapp/tasks_config.sh - replace config files (see below)
-* profiles/myapp/tasks_postinstall.sh - do postinstall actions before finishing
-
-#### Create configs
-
-The script `./bin/create_config.sh` can read config templates and create an output file.
-
-You need to reference the template, output file and a file for replacement data.
-
-```bash
-# ----------------------------------------------------------------------
-# TASKS :: GENERATE CONFIGS
-# ----------------------------------------------------------------------
-
-#               create_config.sh    template_file                                 target_file                       replacements (can be multiple files)
-#               |                   |                                             |                                 |
-#               v                   v                                             v                                 v
-"${selfdir}/bin/create_config.sh"   hooks/templates/mytemplate.erb                config/target.php                 ${profiledir}/replace.txt
-```
-
-#### Pre and post install actions
-
-Example: A simple tasks_postinstall.sh can contain the start of a script that is delivered in the tgz archive.
-
-```bash
-# ----------------------------------------------------------------------
-# TASKS :: POST INSTALL ACTIONS
-# ----------------------------------------------------------------------
-hooks/ondeploy
-```
-
-## Scripts
-
-### deploy_app.sh
-
-This is the main deployment script.
-
-```txt
-./deploy_app.sh [PROFILENAME]
-```
-
-If you start it without parameter it will loop over all existing profiles.
-You can add an existing profile name to limit the execution to that profile only.
-
-### ./bin/create_config.sh
-
-The script is used to read a template (*.erb) to replace simple placeholders
-by regex `<\%\=\ *\@replace\[\"$key\"\]\ *\%>` with a value coming from one or more
-replacement data files. The output will be written to a target file.
-
-```txt
-
-===== IML DEPLOYMENT :: replace variables in erb syntax =====
-
-template    : 
-output      : 
-replacements: replace*.txt
-
-ERROR: missing params.
-create_config.sh TEMPLATE-FILE OUTFILE [optional: REPLACE-DATA-FILE]
-```
-
-A sample replacement file is profiles/example/replace.txt:
-
-```txt
-# ----------------------------------------------------------------------
-# REPLACEMENTS
-# ----------------------------------------------------------------------
-#
-# SYNTAX:
-# key=value
-#
-# - no spaces around first "="
-# - value can contain any char, no quoting
-# ----------------------------------------------------------------------
-
-dbhost=localhost
-dbuser=fred
-dbpassword=vom-jupiter
-dbport=3306
-
-api-key=12345678
-
-# ----------------------------------------------------------------------
-```
-
-The idea for several replacent files is that different groups can set
-their replacement data:
-
-* sysadmins add connections and credentials to needed services/ database connections
-* developers add api keys and other internal values
-* ...
-
-### ./bin/getfile.sh
-
-With getfile.sh you can access the software archive.
-
-* read phase names
-* read ci project ids available in a given phase
-* list files of a given projects of a phase
-* download a single file to a custom target file
-* download ALL files to current working directory
-
-```txt
-SYNTAX:
-
-  getfile.sh [OPTIONS]
-
-OPTIONS:
-
-  -c CFGFILE  load custom config file after defaults in getfile.sh.cfg
-  -d          enable debug infos
-  -e PHASE    phase; overrides env variable IMLCI_PHASE
-  -f FILE     filename to get (without path); overrides env variable IMLCI_FILE
-  -l ITEM     list
-  -o OUTFILE  optional output file
-  -p PROJECT  ci project id; overrides env variable IMLCI_PROJECT
-  -s SECRET   override secret in IMLCI_PKG_SECRET
-  -u URL      URL of iml ci server without trailing /; overrides env variable IMLCI_URL
-
-VALUES:
-
-  CFGFILE     custom config file. It is useful to handle files of different 
-              projects on a server.
-  PHASE       is a phase of the ci server; one of preview|stage|live
-  FILE        is a filename without path that was created by ci server.
-  OUTFILE     Output file. It can countain a path. If none is given the filename
-              will be taken from FILE and stored in current directory
-  PROJECT     project id of the ci server
-  SECRET      secret to access project data on package server. Your given secret
-              must match the secret on package server to get access to any url.
-  ITEM        type what to list; one of phases|projects|files
-              To list projects a phase must be set.
-              To list files a phase and a project must be set.
-
-DEFAULTS:
-
-  You don't need to set all values by command line. Use a config to set defaults
-  ./getfile.sh.cfg
-
-EXAMPLES:
-
-  If url, secret, project and phase are set in the config you can operate by
-  setting the filename to request.
-
-  getfile.sh -f FILE 
-    downloads FILE to the current dir.
-
-  getfile.sh -f FILE -o my-own-filename.tgz 
-    downloads FILE as my-own-filename.tgz
-
-  getfile.sh -f ALL 
-    there is a special file ALL; it fetches all filenames by executing a directory 
-    listing and then downloads all remote files with their original name
-
-  getfile.sh -e preview -l projects
-    list existing projects in phase preview
-
-  getfile.sh -l files
-    list existing files of current project
-
-  Remark: The directory listing can be turned off on the package server and
-  results in a 403 status.
-```
+* Secure download from CI package server using shared secret
+* detect a changed download to run an upgrade or just apply rules for configs
+* Extraction to target dir
+* Cleanup: delete all files in target dir that are not included in the software package
+* Create (config) files by inserting strings into placeholders
+* Several hooks to allow custom actions
+* Handle multiple applications on a machine (aka profiles)
+* Writes a logfile per execution and application (and to stdout)

docs/config.json

@@ -12,8 +12,12 @@
         "date_modified": false,
         "jump_buttons": true,
         "edit_on_github_": "iml-it/appmonitor/tree/master/docs",
+        "edit_on": {
+            "name": "Gitlab",
+            "basepath": "https://git-repo.iml.unibe.ch/iml-open-source/imldeployment-client/-/tree/master/docs"
+        },
         "links": {
-            "Git Repo": "__URL__"
+            "Git Repo": "https://git-repo.iml.unibe.ch/iml-open-source/imldeployment-client/"
         },
         "theme": "daux-blue",
         "search": true