I have worked a lot with backups over the years and one thing I have learned is that managing a traditional backup solution often takes too much time, consumes too much resources and costs too much. But what about backups in a cloud environment? I still have important data and a lot of servers that need to be up and running. True, but if you do it right from the beginning you can take a fundamentally different approach to data protection and like many other things in the cloud it is a huge improvement. These are my general rules that I will describe more in details.
1. Do not use a traditional backup system
You do not need it, It will save you time and money not to use it.
2. Make servers stateless
Store no data that you cannot recreate on the server if you do not have to, especially not the root disk. A web application server should be stateless but a database server is stateful. When you have a stateful server make the root drive stateless and store you stateful data on an Openstack Volume. If your logs are important to you, do it right from the beginning and ship them to a log server instead of backing them up.
3. Automate server provisioning
If you automate the server provisioning:
a. you do not need to backup the OS, application and configuration files.
b. you get the best documentation you can have of a server and you should store the recipe in source code repo.
c. it will take just minutes to provision new servers for recovery, horizontal scaling and test.
4. Use native application backup tools
My experience is that you will get the most reliable and efficient backups by using the applications native backup feature. Another advantage is also that the application owner will have control over both the backup and recovery procedures. Schedule the backups locally in the application or in the OS with unix/linux cron or windows task scheduler. Don’t forget to copy the data to a storage that is in a different availability zone or region. The best solution is normally to use an object storage solution like Swift or similar. Modern applications often have plugins to support object storage natively.
5. Monitor your backups
The challenge with locally scheduled backups is that you do not have a central location where you can monitor that all backups are successfully running. But there is an easy solution for that as well. Use an event monitoring solution that you provide your self or you can use a SaaS like Pushmon, Opsgenie Heartbeat or similar.
6. Automated recovery tests
When you automate the provisioning it is very easy to test recovery as well. And if you can use it in your daily work, that's even better. One example is to build test environments using the production backup data. If you automate and schedule it to get it done, and then use event monitoring to make sure it works.
Provisioning
There are many ways of provisioning a server but I will just show you how you can use the native OpenStack tools to do it. If it is a single server you can just use Nova and if there are multiple servers and/or other infrastructure services required you can use Heat.
We will use a single server and nova in this example to keep it simple.
We create a simple nginx web server on CentOS. Below is the nova command used and the user_data input file.
nova boot webserver01 --flavor m1.small --image centos-6.5-20140117.0 --key-name my-keypair --user-data ./WebServer.user_data
Server (Instance) Backup and recovery
Yes there are occasions where you cannot or its not practical to automate server provisioning or to do native application backups. No worries, there is a very easy way to do server backups in OpenStack. In this example I am using the cli client but you can do it with the API also.
If you have a stateless server and you need to recover the node you can just do a rebuild and the server will be up and running again with the same IP, name, config etc.
With the nova rebuild command you will just enter the server name and image name. If you have verified that the user_data is compatible with another OS you can even use this to change or upgrade the OS version:
nova rebuild webserver01 centos-6.5-20140117.0
If you want to create a backup you will use the nova backup command:
usage: nova backup <server> <name> <backup-type> <rotation>
If we want to make daily backups with a 1 week retention and a weekly backup with a 4 weeks retentions this is what should schedule in cron for example.
`0 2 * * 6 nova backup webserver01 webserver01-daily-date +%F daily 6`
`0 2 * * 1,2,3,4,5,7 nova backup webserver01 webserver01-weekly-date +%F weekly 4`
When running a backup a snapshot is made of the server and that snapshot is then stored as an image in Openstack. The backup image can then be used to boot new servers or to recover the current one.
Here is how to recover the server from a backup image:
nova rebuild webserver01 webserver01-daily-2014-12-12
Volume backup and recovery
There are volume snapshot and backup functions available as well. If you have stateful data on a Volume and there is no advantage in using the native application backup then you should look at the OpenStack Volume backup functionality.
Create a volume
In this example we will create a 10GB volume and attach it to our database server.
usage: cinder create [--display-name <display-name>] <size>
cinder create --display-name dbserver-vol01 10
usage: nova volume-attach <server> <volume> [<device>]
nova volume-attach dbserver01 c0e9e951-e33f-488e-a605-95d0ecc728e1 /dev/vdb
Create a filesystem, mount it and add some data.
Snapshot a volume
A snapshot is great for creating fast point in time copies of volumes. It will help you to protect your data from logical errors but it will probably not help you if the physical disk system will break.
To create a snapshot on a volume that is attached to a server you need to add the “--force True” option.
usage: cinder snapshot-create [--force <True|False>] [--display-name <display-name>] <volume-id>
cinder snapshot-create --force True --display-name dbserver-vol01-snapshot dbserver-vol01
To attach a snapshot to a server you need first create a volume from the snapshot.
cinder create --snapshot-id 4e87739b-4369-430d-bcdb-034b0f0ff861 --display-name vol-from-snapshot 10
And then you can attach it as a new device or replace the current device.
nova volume-attach dbserver01 de75c58b-f311-4e7b-acc7-ae9c2cfeb18b /dev/vdc
Backup a volume
The backup feature is a lot like the server (instance) Root disk backup. It will make a copy of the volume and store it as file in but now directly on Swift object storage.
The problem with the current backup feature is that you cannot make a backup of an attached volume. But if you first do a snapshot, then you can make a backup from that snapshot but in the process you need to convert the snapshot to a volume first. Not very straight forward but it works.
Here is the procedure:
Create a snapshot
cinder snapshot-create dbserver-vol01 --force True --display-name dbserver-vol01-backup
Create a volume from the snapshot
cinder create --snapshot-id 0ac15385-6fdb-42df-b779-2869c4518ed5 --display-name dbserver-vol01-backup 10
Make a volume backup
cinder backup-create --container volume-backup --display-name dbserver-vol01-2015-03-30 dbserver-vol01-backup
Here is how you can list your backups
cinder backup-list
To recover a volume you need to detach the volume from the server, recover the data and then reattach the volume again.
nova volume-detach dbserver c0e9e951-e33f-488e-a605-95d0ecc728e1
cinder backup-restore --volume-id dbserver-vol01 e8ec1eee-a9cb-4135-b8a9-21173bd965db
nova volume-attach dbserver c0e9e951-e33f-488e-a605-95d0ecc728e1
From nova you need to specify the volume id and your need the backup id in the cinder backup-restore command.
To sum it up
Do it right from the beginning and you can retire your old backup solutions, you won't even need to use the built in backup features in OpenStack.