diff --git a/docs/30_Configuration/10_Database.md b/docs/30_Configuration/10_Database.md deleted file mode 100644 index ef1a17aab4e944b1f2bc1293c1163fe4fa3317d3..0000000000000000000000000000000000000000 --- a/docs/30_Configuration/10_Database.md +++ /dev/null @@ -1,46 +0,0 @@ - -# Basic settings # - -There are 2 defaults: - -``` -dir-localdumps = /var/iml-backup -keep-days = 7 -``` - -## dir-localdumps ## - -{string} - -The target directory for local dumps. It is used by - -* the database dump scripts -* the transfer script to store the client backups -* the restore script - -Below that one a directory for the service will be generated; inside that one the database dumbs with scheme name and timestamp, i.e. - - /var/iml-backup/mysql/mydatabase__20190827-2300.sql.gz - -## keep-days ## - -{integer} - -The number of days how long to keep dumps locally. - -Remark: -To make a database restore its dump must be located at this directory. - -To restore an older database you need to restore the dump from the filke backup first. - -If you have local Mysql daemon or Pgsql you can test it by starting - -``` -# dump all databases -sudo ./localdump.sh -``` - -``` -# show written files -find /var/iml-backup -``` diff --git a/docs/30_Configuration/_index.md b/docs/30_Configuration/10_Initialisation.md similarity index 100% rename from docs/30_Configuration/_index.md rename to docs/30_Configuration/10_Initialisation.md diff --git a/docs/30_Configuration/20_Database.md b/docs/30_Configuration/20_Database.md new file mode 100644 index 0000000000000000000000000000000000000000..971f0168d32cffa3b6db93e81ef5f2bb103034fb --- /dev/null +++ b/docs/30_Configuration/20_Database.md @@ -0,0 +1,31 @@ +# Basic settings for database backups # + +There are 2 required values in the jobs/backup.job + +```text +dir-localdumps = /var/iml-backup +keep-days = 7 +``` + +This defines the backup target for sql dumps and how long they will be kept locally. + +There is an optional value to define the target directory for archived dumps. This value is used for couchdb2 only. + +`dir-dbarchive = /var/localdumps/archive` + +see [backup.job](50_File_backup.job.md) + +## Backup sqlite ## + +Sqlite files can be located anywhere in the filesystem. That's why the +cannot be located with an auto detection. You need to define them in +the file jobs/backup-dbfiles.job first. + +Per database file set a line with the `sqlite = ` prefix + +```text +sqlite = /var/lib/whatever/sqlite-database_01.db +sqlite = /var/lib/somewhere/else/db.sqlite +``` + +see [backup-dbfiles.job](50_File_backup-dbfiles.job.md) diff --git a/docs/30_Configuration/20_Filetransfer.md b/docs/30_Configuration/20_Filetransfer.md deleted file mode 100644 index e8b4b98f2694868b0111acbbb9cc6fb0415e9995..0000000000000000000000000000000000000000 --- a/docs/30_Configuration/20_Filetransfer.md +++ /dev/null @@ -1,84 +0,0 @@ -The file transfer describes how local directories will be backed up - -* choose backup tool -* where to write data (backup target) -* incremental or full backup -* how to delete backup data - -Details to the the given config entries you find in the description for -[transfer.job](50_File_transfer.job.md). - -## Backup tool ## - -You can decide between Restic (Default) and Duplicity. - -`bin = ...` - -```text -bin = restic -type = auto -``` - -If you use Duplicity for servers it is suggested to run (daily) incremental -backups and full backups on a specific day. - -```text -bin = duplicity -type = inc -full = WDM:3rd Fri -start-time-full = 23:0. -``` - -If you use Duplicity on a desktop client you maybe want to run incremental backups -and a full backup if the last one is older than a wanted time. - -```text -bin = duplicity -type = auto -auto = 1M -``` - -## Backup target ## - -The definition of a backup target depends on the used backup tool (restic|duplicity). -You need a bit knowledge about these tools. - -We have tested the following targets - -* local directory/ mounted filesystem i.e. Samba share - * Restic: `/mnt/mounted_directory` - * Duplicity: `file:///mnt/mounted_directory` -* ssh target (scp or rsync) - * Restic: `sftp:imlbackup@backup-target.example.com:/mnt/backupdata` - * Duplicity: `rsync://imlbackup@backup-target.example.com//mnt/backupdata` - * Duplicity: `scp://imlbackup@backup-target.example.com//mnt/backupdata` -* https - * Restic: `rest:https://user:pass@backup-target.example.com:8000/user/` - -Set your value in `storage = ...` - -Example for Restic with SFTP transfer - -`storage = sftp:imlbackup@backup-target.example.com:/mnt/backupdata` - -## Local encryption ## - -Restic an Duplicity encrypt local data with a password before transferring them. - -Deploy a host specific password with an orchestration tool (Ansible, Puppet, ...) -or on a manual installation keep a copy of it on a safe place. Without the password -you cannot decrypt backup data after a filesystem crash. - -`passphrase = EnterYourSecretHere` - -## More options ## - -Automatic backup of samba shares - it works only if samba shares were detected. -If no samba config is available it has no effect and shows no error. - -`sambashares = 1` - -When restoring data ... this is the path for restored files: - -`restore-path = /restore` - diff --git a/docs/30_Configuration/30_Filetransfer.md b/docs/30_Configuration/30_Filetransfer.md new file mode 100644 index 0000000000000000000000000000000000000000..69022067ae806ca187d2868c4033e908358fdceb --- /dev/null +++ b/docs/30_Configuration/30_Filetransfer.md @@ -0,0 +1,177 @@ +The file transfer describes how local directories will be backed up + +* choose backup tool; incremental or full backup +* where to write data (backup target) +* set password for local encryption +* more options +* how to cleanup old backup data + +Details to the the given config entries you find in the description for +[transfer.job](50_File_transfer.job.md). + +## Backup tool ## + +You can decide between Restic (Default) and Duplicity. + +`bin = ...` + +```text +bin = restic +type = auto +``` + +If you use Duplicity for servers it is suggested to run (daily) incremental +backups and full backups on a specific day. + +```text +bin = duplicity +type = inc +full = WDM:3rd Fri +start-time-full = 23:0. +``` + +If you use Duplicity on a desktop client you maybe want to run incremental backups +and a full backup if the last one is older than a wanted time. + +```text +bin = duplicity +type = auto +auto = 1M +``` + +## Backup target ## + +The definition of a backup target depends on the used backup tool (restic|duplicity). +You need a bit knowledge about these tools. + +We have tested the following targets + +* local directory/ mounted filesystem i.e. Samba share + * Restic: `/mnt/mounted_directory` + * Duplicity: `file:///mnt/mounted_directory` +* ssh target (scp or rsync) + * Restic: `sftp:imlbackup@backup-target.example.com:/mnt/backupdata` + * Duplicity: `rsync://imlbackup@backup-target.example.com//mnt/backupdata` + * Duplicity: `scp://imlbackup@backup-target.example.com//mnt/backupdata` +* https + * Restic: `rest:https://user:pass@backup-target.example.com:8000/user/` + +Set your value in `storage = ...` + +Example for Restic with SFTP transfer + +`storage = sftp:imlbackup@backup-target.example.com:/mnt/backupdata` + +## Local encryption ## + +Restic an Duplicity encrypt local data with a password before transferring them. + +Deploy a host specific password with an orchestration tool (Ansible, Puppet, ...) +or on a manual installation keep a copy of it on a safe place. Without the password +you cannot decrypt backup data after a filesystem crash. + +`passphrase = EnterYourSecretHere` + +## More options ## + +Automatic backup of samba shares - it works only if samba shares were detected. +If no samba config is available it has no effect and shows no error. + +`sambashares = 1` + +When restoring data ... this is the path for restored files: + +`restore-path = /restore` + +### Duplicity settings ### + +When writing backup data duplicity creates the target file and a index. +By default the index is written to ~/.cache/duplicity/ (and to the backup target). +If your /root partition has a limited size and/ or you have a lot of files +(i.e. on a file server) then you can put the index files somewhere else: + +`duplicity_cachedir = [full path]` + +Duplicity with rsync needed an additional ssh-backend *pexpect* on Debian. +It can be set with enabling: + +`# duplicity_ssh-backend = pexpect` + +On the backup target you can define the size of the chunks to write. +This would allow to write the backup data to an additional medium with +limited size (i.e. CD or DVD). + +Set a size in MB + +`duplicity_volsize = 350` + +### Restic settings ### + +When writing backup data restic creates a local index. +By default the index is written to ~/.cache/restic/. +If your /root partition has a limited size and/ or you have a lot of files +(i.e. on a file server) then you can put the index files somewhere else: + +`restic_cachedir = [full path]` + +The default verbose of IML backup for restic is 2. During file transfer +this shows unchanged/ new/ deleted files. Change it to a less value +to generate less log data. + +`restic_verbose = 2` + +A file transfer can be tagged. Each restic backup gets a defined tag + +`restic_tag = imlbackup` + +For restore you can copy files to the restore location. With restic +you can mount the backup volume into the local filesystem using fuse. +Then you can browse in the snapshots of the backup data and copy +the files and dirs with linux commands. + +The mountpoint for the backup storage is a full path (it will be created +when selcting to mount in the restore). + +`restic_mountpoint = /mnt/restore` + +## Cleanup options ## + +### Duplicity ### + +Duplicity can set a value how long to keep backup data on the backup storage. +It will keep all files of incremental and full backup to be able to restore +data of the given min. availability. + +Example if you defined a monthly full backup and keep backups for 6 monthes ... +then it keeps all files up to 6 monthes plus all older files up to the next +full backup. You will find backup files up to 7 monthes. + +```text + 1 month (for monthly full backup) ++ 6 monthes (min. availability for restore) +--------------------------------------------- + 7 monthes +``` + +The value duplicity_keep is a number and a letter for the time format. + +s, m, h, D, W, M, or Y (indicating seconds, minutes, hours, days, weeks, months, or years respectively) + +`duplicity_keep = 6M` + +### Restic ### + +Restic works like a repository and uses deduplication. The first run of a file backup +is a full backup and all following runs make an incremental backup. + +You have the possibility to keep all younger backups and have the possibility to delete +older backups but to keep N weekly, monthly and yearly snapshots. + +```text +# prune +restic_keep-hourly = 100 +restic_keep-daily = 90 +restic_keep-weekly = 12 +restic_keep-monthly = 12 +restic_keep-yearly = 10 +``` diff --git a/docs/30_Configuration/50_File_backup-dbfiles.job.md b/docs/30_Configuration/50_File_backup-dbfiles.job.md new file mode 100644 index 0000000000000000000000000000000000000000..497c9de1889924cad10ddb68a07ecb01d581fa3c --- /dev/null +++ b/docs/30_Configuration/50_File_backup-dbfiles.job.md @@ -0,0 +1,30 @@ +This is a config file for file based databases that can be anywhere in the filesystem. + +At the moment it is used by sqlite backup only. + +# dist file # + +```text +# ---------------------------------------------------------------------- +# +# jobfile for backup of filebased databases +# list of files with servicename as key ... and its files with full path +# +# ---------------------------------------------------------------------- +# +# SYNTAX: +# [variable] = [value] +# +# - variable must start in first column +# - char "=" must be surrounded by space +# - value - any string; no " needed +# +# EXAMPLE: +# sqlite = /var/lib/whatever/mysqlite-database.db +# +# ---------------------------------------------------------------------- + + + +# ---------------------------------------------------------------------- +``` diff --git a/docs/30_Configuration/50_File_backup.job.md b/docs/30_Configuration/50_File_backup.job.md index 8a0aa2e0c89f893b9a3e400af762e00e1de474d1..79d84f40ef7e51cf077bce6a75fe1331e7a2206d 100644 --- a/docs/30_Configuration/50_File_backup.job.md +++ b/docs/30_Configuration/50_File_backup.job.md @@ -33,12 +33,18 @@ dir-dbarchive = /var/localdumps/archive ## dir-dbarchive ## -Target dir for database archives (to keep deleted schemes or databases that are not backupped anymore; couchdb2 feature) +{string} + +Define the target directory for archived dumps. Do not add a trailing `/`. + +It is used to keep deleted schemes or databases that are not backupped anymore. It is used for couchdb2 only. `dir-dbarchive = /var/localdumps/archive` ## dir-localdumps ## +{string} + Target dir for database backups of all types. Below that directory a subdir for the database type will be created (i.e. mysql or pgsql). @@ -48,6 +54,8 @@ see also **keep-days** ## keep-days ## +{integer} + How many days to keep database dumps locally in `dir-localdumps = ...` `keep-days = 7` @@ -56,4 +64,6 @@ see also **dir-localdumps** ## lang ## +{integer} + Language of the shell. Do not change to keep documented keywords for date and time. diff --git a/docs/40_Usage/10_Database.md b/docs/40_Usage/10_Database.md new file mode 100644 index 0000000000000000000000000000000000000000..35ced633e166d039d1b44930780291f8228f8dca --- /dev/null +++ b/docs/40_Usage/10_Database.md @@ -0,0 +1,98 @@ +## Make database dumps ## + +To create backup dumps we use `./localdump.sh` + +```text +SYNTAX: +localdump.sh [[operation]] [Name_of_service] [[more services]] +localdump.sh restore [Name_of_service] [file-to-restore] + + operation - one of backup|restore; optional parameter; default is backup + Name_of_service - name of database service + You get a list of all available services without parameter + Use ALL for bulk command + file - filename of db dump to restore + +Known services (see ./plugins/localdump): +couchdb +couchdb2 +ldap +mysql +pgsql +sqlite +```text + +If you have local Mysql daemon or Pgsql you can test it by starting + +```text +# dump all databases +sudo ./localdump.sh ALL +``` + +```text +# show written files +find /var/iml-backup +``` + +The keyword **ALL** loops over all databases. If ever possible all schemes will be detected and dumped. + +To dump schemes of a specific database type add the name of a known service. + +```text +# dump all Mysql databases +sudo ./localdump.sh mysql +``` + +## Structure in the backup folder ## + +In the database dump folder is a subdir per service `/var/iml-backup/[service]`. + +Below the service folder are files named like the database scheme + `__` + timestamp. + +All dumps are gzip compressed. + +At the end of a backup task with localdump.sh older files older than *keep-days* +will be deleted from `/var/iml-backup/[service]`. + +### Backup sqlite ### + +Keep in mind that you need to define sqlite databases in jobs/backup-dbfiles.job first. + +```text +# dump all Sqlite databases +sudo ./localdump.sh sqlite +``` + +This greps "^sqlite = " in jobs/backup-dbfiles.job and squentially dumps each sqlite file. + +In the folder /var/iml-backup/sqlite/ it creates 2 files per database + +* the gzip compressed dump (filename is full path with replacing `/` by `_`) +* a .META file that contains the original full path for restore + +## Restore database dumps ## + +Remark: +To make a database restore its dump must be located at this directory. + +The value keepdays contains number of days how long to keep dumps locally. +If your dump to restore is older than this given range then you need +to restore files of /var/iml-backup first. + +To start a restore you need the backup type. You come into an +interactive mode where you get a file selection and can enter +da (new) database where to import these data. + +```text +# restore with selection +sudo ./localdump.sh restore mysql +``` + +If you additionally add a filename of an existing database file the database +will be restored directly into the original database. + +```text +/opt/imlbackup/client/localdump.sh restore mysql /var/iml-backup/mysql/mydatabase__20220127-2303.sql.gz +``` + +With that mechanism you could build a script for a bulk import.