# IML BACKUP #

Backup scripts using duplicity.
Runs on Linux.

Free software. GNU GPL 3.0.

Source: <https://git-repo.iml.unibe.ch/iml-open-source/iml-backup/>
Duplicity: <http://duplicity.nongnu.org/>

## Why ##

We don't want to configure a backup set on a "central backup server" for each new node. Each new node pushed its own backup data to a backup target.

We want to push data from a private network to target; a central backup server would ot reach some clients.

A set of database backup scripts detects exsiting locally running database servers and puts a compressed dump file per database scheme to a local backup directory.

Then a transfer script uses a tool to encrypt and transfer local backups and other local folders to a backup target.

## Features ##

### Database dumps ###

Supported Databases for backup and restore

* Mysql/ Mariadb (mysql_dump)
* postgres (pg_dump)
* sqlite (by naming files with full path in a config)

Limited support:

* couchdb (using a config with naming convention)
* ldap (without restore so far)

### backup tools ###

DUPLICITY

* Incremental and full backups
* encrypted backups using GPG
* set size of backup volumes
* delete old backups by a given time limit
* several backup targets (we currently use scp:// rsync:// and file://)

RESTIC

* creates an initial full backup - and then never again.
* encrypts data
* deduplicates files
* delete backups by rules to keep a count of hourly, daily, weekly, mothly, yearly backups
* several backup targets (we currently use sftp:// https:// and file://)

### control simoultanous backups ###

As an optional feature you can control the count simultanous written backups.
This requires additional effort next to the client installation.

## Installation ##

- Uncompress / clone the client to a local directory
- go to jobs directory to copy the *.job.dist files to *.job
- configure *.job files
- manual test run
- create a cronjob

### Uncompress client ###

To put all files into a directory i.e.

    /opt/imlbackup/client

then use the **root** user and follow these steps:

    # Create the directory level above
    mdir -p /opt/imlbackup/

    # download
    cd /opt/imlbackup/
    wget https://git-repo.iml.unibe.ch/iml-open-source/iml-backup/-/archive/master/iml-backup-master.tar.gz

    # extract
    tar -xzf iml-backup-master.tar.gz
    mv iml-backup-master client

    # remove downloaded file
    rm -f iml-backup-master.tar.gz

    # to set pwd to /opt/imlbackup/client:
    cd client

### database backup: set local backup target ###

Create a **jobs/dirs.job** (copy the delivered *.dist file)

    cd jobs
    cp backup.job.dist backup.job

There are 2 defaults:

    dir-localdumps = /var/iml-backup
    keep-days = 7

**dir-localdumps**

{string}

The target directory for local dumps. It is used by 

* the database dump scripts
* the transfer script to store the client backups
* the restore script

Below that one a directory for the service will be generated; inside that one the database dumbs with scheme name and timestamp, i.e.

    /var/iml-backup/mysql/mydatabase__20190827-2300.sql.gz

**keep-days**

{integer}

The number of days how long to keep dumps locally.

Remark:
To make a database restore its dump must be located at this directory. To restore an older database you need to restore the dump from duplicity first.

If you have local Mysql daemon or Pgsql you can test it by starting 

    # dump all databases
    sudo ./localdump.sh

    # show written files
    find /var/iml-backup

### Define local directories to backup ###

Edit **jobs/dirs.job** again.

There are a few include definitions:

    # ----------------------------------------------------------------------
    # directory list to transfer
    # without ending "/"
    # missing directories on a system will be ignored
    # ----------------------------------------------------------------------
    include = /etc
    include = /var/log
    include = /home

... and excludes

    # ----------------------------------------------------------------------
    # excludes
    # see duplicity ... added as -exclude-regex parameter
    # ----------------------------------------------------------------------

    # exclude = .*\.(swp|tmp)

    # mac file
    # exclude = \.DS_Store

    # all subdirs containing "cache/", i.e. any/path/zend-cache/[file]
    # exclude = cache/.*

**include**

{string}

Multiple entries are allowed. Each defines a starting directory that is backed up recursive.

Do not use a trailing slash "/".

Each include line creates ist own backup volume on the backup target: One duplicity backup command will be started per include.

An include for the database dumps is not needed - it will be added automatically.

Missing directories on a system will be ignored and does NOT throw an error. So you can write a "general" single config and deploy all servers.

**exclude**

{string}

Multiple entries are allowed. Each defines a regex that is applied to all include items. This could have an negative impact. I suggest not to define an exclude - only if it is needed because of huge wasted space.

TODO: advanced stuff ... There is a possibility for directory based include and exclude rules.

### Setup the target ###

Edit **jobs/transfer.job**. This file handles the transfer of local directories
to a backup target. You find comments in the config.

By default the backp tool "restic" is activated (and recommended). You can switch to duplicity
if you feel familiar with it.

`bin = restic`

Create a repository base directory with the wanted protocol. This step has to be done
once for all systems you want to backup. The IML Backup will create a subdirectory
with the hostname for its backups. Set your target in storage:

`storage = sftp://backup@storage.example.com//netshare/backup`

## Production usage ##

Edit **jobs/transfer.job**.

Set a password to encrypt local data with it. Each system should have its own password.
Use a long password - i.e. 128 characters.
Save your password list - if you loose it you cannot restore data anymore.

`passphrase = EnterYourSecretHere`

Change the restore path if needed. A restore does not overwrite the current files.

`restore-path = /restore`

Then have a look to the section with variables that have the prefix of bin = ... (restic_... or duplicity_...).

The index files are stored in $HOME. Because /root partition could be to small for
systems with many files (fileserver) you can put the index somewhere else:

`*_cachedir = ...`

With this loglevel you get a list of new or changed files in the log:

`restic_verbose = 2`

You can set a tag that is set for backups by script.

`restic_tag = imlbackup`

The mount of backup sets is a restic feature. After mounting a backup there
you can browse via filesystem through backups and timestamps.

`restic_mountpoint = /mnt/restore`

Define how many backups you wanto to keep. After the backup of a directory
the cleanup strategy will be applied.

```
# prune 
restic_keep-hourly = 100
restic_keep-daily = 90
restic_keep-weekly = 12
restic_keep-monthly = 12
restic_keep-yearly = 10
```

### setup backup times ###

### Create a cronjob ###

### Monitoring ###

### Restore files ###

### Restore databases ###