Merge branch 'eliminate-scheduler' into 'master'

Update docs

See merge request !30

docs/30_Configuration/10_Database.md

-
-# Basic settings #
-
-There are 2 defaults:
-
-```
-dir-localdumps = /var/iml-backup
-keep-days = 7
-```
-
-## dir-localdumps ##
-
-{string}
-
-The target directory for local dumps. It is used by 
-
-* the database dump scripts
-* the transfer script to store the client backups
-* the restore script
-
-Below that one a directory for the service will be generated; inside that one the database dumbs with scheme name and timestamp, i.e.
-
-    /var/iml-backup/mysql/mydatabase__20190827-2300.sql.gz
-
-## keep-days ##
-
-{integer}
-
-The number of days how long to keep dumps locally.
-
-Remark:
-To make a database restore its dump must be located at this directory.
-
-To restore an older database you need to restore the dump from the filke backup first.
-
-If you have local Mysql daemon or Pgsql you can test it by starting
-
-```
-# dump all databases
-sudo ./localdump.sh
-```
-
-```
-# show written files
-find /var/iml-backup
-```

docs/30_Configuration/20_Database.md

+# Basic settings for database backups #
+
+There are 2 required values in the jobs/backup.job
+
+```text
+dir-localdumps = /var/iml-backup
+keep-days = 7
+```
+
+This defines the backup target for sql dumps and how long they will be kept locally.
+
+There is an optional value to define the target directory for archived dumps. This value is used for couchdb2 only.
+
+`dir-dbarchive = /var/localdumps/archive`
+
+see [backup.job](50_File_backup.job.md)
+
+## Backup sqlite ##
+
+Sqlite files can be located anywhere in the filesystem. That's why the
+cannot be located with an auto detection. You need to define them in
+the file jobs/backup-dbfiles.job first.
+
+Per database file set a line with the `sqlite = ` prefix
+
+```text
+sqlite = /var/lib/whatever/sqlite-database_01.db
+sqlite = /var/lib/somewhere/else/db.sqlite
+```
+
+see [backup-dbfiles.job](50_File_backup-dbfiles.job.md)

docs/30_Configuration/20_Filetransfer.md → docs/30_Configuration/30_Filetransfer.md

 The file transfer describes how local directories will be backed up
 
-* choose backup tool
+* choose backup tool; incremental or full backup
 * where to write data (backup target)
-* incremental or full backup
-* how to delete backup data
+* set password for local encryption
+* more options
+* how to cleanup old backup data
 
 Details to the the given config entries you find in the description for
 [transfer.job](50_File_transfer.job.md).
@@ -82,3 +83,95 @@ When restoring data ... this is the path for restored files:
 
 `restore-path = /restore`
 
+### Duplicity settings ###
+
+When writing backup data duplicity creates the target file and a index.
+By default the index is written to ~/.cache/duplicity/ (and to the backup target).
+If your /root partition has a limited size and/ or you have a lot of files
+(i.e. on a file server) then you can put the index files somewhere else:
+
+`duplicity_cachedir = [full path]`
+
+Duplicity with rsync needed an additional ssh-backend *pexpect* on Debian.
+It can be set with enabling:
+
+`# duplicity_ssh-backend = pexpect`
+
+On the backup target you can define the size of the chunks to write.
+This would allow to write the backup data to an additional medium with
+limited size (i.e. CD or DVD).
+
+Set a size in MB
+
+`duplicity_volsize = 350`
+
+### Restic settings ###
+
+When writing backup data restic creates a local index.
+By default the index is written to ~/.cache/restic/.
+If your /root partition has a limited size and/ or you have a lot of files
+(i.e. on a file server) then you can put the index files somewhere else:
+
+`restic_cachedir = [full path]`
+
+The default verbose of IML backup for restic is 2. During file transfer
+this shows unchanged/ new/ deleted files. Change it to a less value
+to generate less log data.
+
+`restic_verbose = 2`
+
+A file transfer can be tagged. Each restic backup gets a defined tag
+
+`restic_tag = imlbackup`
+
+For restore you can copy files to the restore location. With restic
+you can mount the backup volume into the local filesystem using fuse.
+Then you can browse in the snapshots of the backup data and copy
+the files and dirs with linux commands.
+
+The mountpoint for the backup storage is a full path (it will be created
+when selcting to mount in the restore).
+
+`restic_mountpoint = /mnt/restore`
+
+## Cleanup options ##
+
+### Duplicity ###
+
+Duplicity can set a value how long to keep backup data on the backup storage.
+It will keep all files of incremental and full backup to be able to restore 
+data of the given min. availability.
+
+Example if you defined a monthly full backup and keep backups for 6 monthes ... 
+then it keeps all files up to 6 monthes plus all older files up to the next
+full backup. You will find backup files up to 7 monthes.
+
+```text
+  1 month    (for monthly full backup)
++ 6 monthes  (min. availability for restore)
+---------------------------------------------
+  7 monthes  
+```
+
+The value duplicity_keep is a number and a letter for the time format.
+
+s, m, h, D, W, M, or Y (indicating seconds, minutes, hours, days, weeks, months, or years respectively)
+
+`duplicity_keep = 6M`
+
+### Restic ###
+
+Restic works like a repository and uses deduplication. The first run of a file backup
+is a full backup and all following runs make an incremental backup.
+
+You have the possibility to keep all younger backups and have the possibility to delete
+older backups but to keep N weekly, monthly and yearly snapshots.
+
+```text
+# prune 
+restic_keep-hourly = 100
+restic_keep-daily = 90
+restic_keep-weekly = 12
+restic_keep-monthly = 12
+restic_keep-yearly = 10
+```

docs/30_Configuration/50_File_backup-dbfiles.job.md

+This is a config file for file based databases that can be anywhere in the filesystem.
+
+At the moment it is used by sqlite backup only.
+
+# dist file #
+
+```text
+# ----------------------------------------------------------------------
+#
+# jobfile for backup of filebased databases
+# list of files with servicename as key ... and its files with full path
+#
+# ----------------------------------------------------------------------
+#
+# SYNTAX: 
+# [variable] = [value]
+#
+# - variable must start in first column
+# - char "=" must be surrounded by space
+# - value - any string; no " needed
+#
+# EXAMPLE:
+# sqlite = /var/lib/whatever/mysqlite-database.db
+#
+# ----------------------------------------------------------------------
+
+
+
+# ----------------------------------------------------------------------
+```

docs/30_Configuration/50_File_backup.job.md

@@ -33,12 +33,18 @@ dir-dbarchive = /var/localdumps/archive
 
 ## dir-dbarchive ##
 
-Target dir for database archives (to keep deleted schemes or databases that are not backupped anymore; couchdb2 feature)
+{string}
+
+Define the target directory for archived dumps. Do not add a trailing `/`.
+
+It is used to keep deleted schemes or databases that are not backupped anymore. It is used for couchdb2 only.
 
 `dir-dbarchive = /var/localdumps/archive`
 
 ## dir-localdumps ##
 
+{string}
+
 Target dir for database backups of all types. 
 Below that directory a subdir for the database type will be created (i.e. mysql or pgsql).
 
@@ -48,6 +54,8 @@ see also **keep-days**
 
 ## keep-days ##
 
+{integer}
+
 How many days to keep database dumps locally in `dir-localdumps = ...`
 
 `keep-days = 7`
@@ -56,4 +64,6 @@ see also **dir-localdumps**
 
 ## lang ##
 
+{integer}
+
 Language of the shell. Do not change to keep documented keywords for date and time.

docs/40_Usage/10_Database.md

+## Make database dumps ##
+
+To create backup dumps we use `./localdump.sh`
+
+```text
+SYNTAX:
+localdump.sh [[operation]] [Name_of_service] [[more services]]
+localdump.sh restore  [Name_of_service] [file-to-restore]
+
+  operation       - one of backup|restore; optional parameter; default is backup
+  Name_of_service - name of database service
+                    You get a list of all available services without parameter
+                    Use ALL for bulk command
+  file            - filename of db dump to restore
+
+Known services (see ./plugins/localdump):
+couchdb
+couchdb2
+ldap
+mysql
+pgsql
+sqlite
+```text
+
+If you have local Mysql daemon or Pgsql you can test it by starting
+
+```text
+# dump all databases
+sudo ./localdump.sh ALL
+```
+
+```text
+# show written files
+find /var/iml-backup
+```
+
+The keyword **ALL** loops over all databases. If ever possible all schemes will be detected and dumped.
+
+To dump schemes of a specific database type add the name of a known service.
+
+```text
+# dump all Mysql databases
+sudo ./localdump.sh mysql
+```
+
+## Structure in the backup folder ##
+
+In the database dump folder is a subdir per service `/var/iml-backup/[service]`.
+
+Below the service folder are files named like the database scheme + `__` + timestamp.
+
+All dumps are gzip compressed.
+
+At the end of a backup task with localdump.sh older files older than *keep-days* 
+will be deleted from `/var/iml-backup/[service]`.
+
+### Backup sqlite ###
+
+Keep in mind that you need to define sqlite databases in jobs/backup-dbfiles.job first.
+
+```text
+# dump all Sqlite databases
+sudo ./localdump.sh sqlite
+```
+
+This greps "^sqlite = " in jobs/backup-dbfiles.job and squentially dumps each sqlite file.
+
+In the folder /var/iml-backup/sqlite/ it creates 2 files per database 
+
+* the gzip compressed dump (filename is full path with replacing `/` by `_`)
+* a .META file that contains the original full path for restore
+
+## Restore database dumps ##
+
+Remark:
+To make a database restore its dump must be located at this directory.
+
+The value keepdays contains number of days how long to keep dumps locally.
+If your dump to restore is older than this given range then you need
+to restore files of /var/iml-backup first.
+
+To start a restore you need the backup type. You come into an
+interactive mode where you get a file selection and can enter 
+da (new) database where to import these data.
+
+```text
+# restore with selection
+sudo ./localdump.sh restore mysql
+```
+
+If you additionally add a filename of an existing database file the database
+will be restored directly into the original database.
+
+```text
+/opt/imlbackup/client/localdump.sh restore mysql /var/iml-backup/mysql/mydatabase__20220127-2303.sql.gz
+```
+
+With that mechanism you could build a script for a bulk import.