Backup (Old Notes)
******************

.. note:: The process below can easily be adapted for use with any linux based
          storage system because we use Duplicity, ssh keys and gpg for
          encryption.

Links
=====

- `How To Use Duplicity with GPG to Securely Automate Backups on Ubuntu`_
- `Setting up Duplicity with GnuPG`_

Strategy (draft)
================

http://duplicity.nongnu.org/duplicity.1.html

Using the following commands::

  remove-all-but-n-full
  remove-all-inc-of-but-n-full

We will run one backup which does weekly full, daily incremental, deleting full
backups over 4 weeks, and incremental over 7 days.

We will run another (separate) monthly full backup which is deleted after 3
months.

For database backups::

  # put the database backups into a 'backup' folder on the cloud server e.g.
  /home/web/repo/backup/pkimber.net/20141024_1700.sql
  /home/web/repo/backup/hatherleigh.info/20141024_1704.sql

  # cron task will remove the previous days backups after making todays e.g.
  rm /home/web/repo/backup/pkimber.net/20141023_1600.sql
  rm /home/web/repo/backup/hatherleigh.info/20141023_1604.sql

  # duplicity will back this up to rsync.net
  duplicity full --encrypt-key="ABCD0001" \
    scp://123@tv-s009.rsync.net/pkimber.net/backup \
    /home/web/repo/backup/pkimber.net
  duplicity full --encrypt-key="ABCD0001" \
    scp://123@tv-s009.rsync.net/hatherleigh.info/backup \
    /home/web/repo/backup/hatherleigh.info

  # duplicity will verify the backup
  duplicity verify --encrypt-key="ABCD0001" \
    scp://123@tv-s009.rsync.net/pkimber.net/backup \
    /home/web/repo/backup/pkimber.net
  duplicity verify --encrypt-key="ABCD0001" \
    scp://123@tv-s009.rsync.net/hatherleigh.info/backup \
    /home/web/repo/backup/hatherleigh.info

What can we do?
---------------

Delete *all* backups older than *x* days (weeks or months)

Remove *all* backups older than *count* full backups

Remove all incrementals older than *count* full backups

::

  F   1st Sept
  I
  I
  F   15th Sept
  I
  I
  F   1st Oct
  I

Getting Started
===============

`Install Duplicity 0.6`_

.. note:: We are using Duplicity 0.6 because the latest version of Duplicity on
          Ubuntu 16.04 doesn't work nicely with the Duplicity on our 14.04
          servers.  When our servers are using 16.04, then we can probably
          go back to installing the standard distro version i.e.
          ``sudo apt-get install duplicity python-paramiko``

Restore
-------

:doc:`restore`

.. _duplicity_command_examples:

Duplicity
---------

.. tip:: To install an earlier version of Duplicity, see
         `Install Duplicity 0.6`_

To list the files on ``rsync.net``::

  # database backup (and any files in the backup folder)
  ssh 123@usw-s001.rsync.net ls -la hatherleigh.info/backup
  # files backup
  ssh 123@usw-s001.rsync.net ls -la hatherleigh.info/files

To list backup dates::

  duplicity collection-status ssh://123@usw-s001.rsync.net/hatherleigh.info/backup

To list the backups::

  duplicity list-current-files ssh://123@usw-s001.rsync.net/hatherleigh.info/backup
  duplicity list-current-files ssh://123@usw-s001.rsync.net/hatherleigh.info/files

Duplicity makes restoring easy. You can restore by simply reversing the remote
and local parameters.

.. note:: You will probably see ``Operation not permitted`` errors.  This is
          Duplicity attempting to restore owner and group permissions on the
          files.

To restore a folder::

  PASSPHRASE="gpg-password" \
    duplicity \
    --file-to-restore \
    "path/to/folder/" \
    ssh://123@usw-s001.rsync.net/hatherleigh.info/files \
    /path/to/restore/folder/

.. note:: When restoring a folder, ``/path/to/restore/folder/`` must not exist.
          It will be created by Duplicity.

To restore a single file (in this example we are restoring from a ``Dropbox``
backup)::

  PASSPHRASE="gpg-password" \
    duplicity \
    --file-to-restore \
    "Dropbox/Contact/Cycle Policy.docx" \
    ssh://123@usw-s001.rsync.net/dropbox/web.hatherleigh.info/files \
    "/path/to/restore/Cycle Policy.docx"

.. note:: When restoring a single file, ``/path/to/restore/policy.docx`` is the
          file name **NOT** the folder name.

To restore by date or time:

To restore a full set of files from 2 days ago (note you can omit the
``restore``)::

  duplicity restore -t 2D ssh://123@usw-s001.rsync.net/hatherleigh.info/backup
    /path/to/restore/file

To restore a full set of files from a specific time (note you can omit the
``restore``):

The ``--time`` format is ``YYYYMMDDTHHMMSSZ`` and Duplicity will pull the
restore from the next backup older than the time entered e.g.
``collection-status`` shows a backup dated ``20141125T112710Z`` and one dated
``20141125T122710Z`` enter ``--time`` as ``20141125T113000Z`` and retrieve data
from ``20141125T112710Z``::

  duplicity restore --time 20141125T113000Z \
    ssh://123@usw-s001.rsync.net/hatherleigh.info/backup \
    /path/to/restore/file

.. warning::

   Restoring from rsync.net back to any location other than the original
   location will result in an ``Error '[Errno 1] Operation not permitted:``
   prefix to each restored file although the files will restore and be
   available... this is a known bug with Duplicity to do with permissions. See:
   `Why do I get an 'Operation not permitted'`_

Usage
=====

.. _install_duplicity_0_6:

Install Duplicity 0.6
=====================

Here are some instructions for installing 0.6 version of duplicity.  I've
installed it in ``/opt/duplicity`` so that it is not overwritten by a new
version when we upgrade ubuntu.

Remove the installed version of ``duplicity``::

  apt remove duplicity

Download the latest version of the 0.6 tarball from:
https://code.launchpad.net/duplicity/0.6-series/0.6.26/+download/duplicity-0.6.26.tar.gz

Create a directory for Duplicity (you will need to ``sudo`` for this)::

  # sudo
  mkdir /opt/duplicity

Extract the archive to a temporary folder.

Change into the folder and run::

  # sudo
  python setup.py install --prefix=/opt/duplicity

You might need to::

  sudo apt install librsync-dev python-dev python-lockfile python-paramiko

Create a script (in your home folder) to run duplicity as follows::

  # vim ~/bin/duplicity

  #!/bin/bash
  # exit immediately if a command exits with a nonzero exit status.
  set -e
  # treat unset variables as an error when substituting.
  set -u
  export PYTHONPATH='/opt/duplicity/lib/python2.7/site-packages/'
  /opt/duplicity/bin/duplicity $@

Make it executable::

  chmod a+x ~/bin/duplicity

Test it using::

  ~/bin/duplicity --version

Then uninstall the distro version::

  sudo apt-get remove duplicity


.. _`How To Use Duplicity with GPG to Securely Automate Backups on Ubuntu`: https://www.digitalocean.com/community/tutorials/how-to-use-duplicity-with-gpg-to-securely-automate-backups-on-ubuntu
.. _`Setting up Duplicity with GnuPG`: http://codegouge.blogspot.co.uk/2012/01/setting-up-duplicity-with-gnupg.html
.. _`Why do I get an 'Operation not permitted'`: http://askubuntu.com/questions/266877/why-do-i-get-an-operation-not-permitted-error-when-running-duplicity-as-sudo