Swift cloud object storage is ideal for storing your application backups. Here are 3 ways to move your backups to the cloud.
Swift cloud object storage is ideal for storing your application backups. Here are 3 ways to move your backups to the cloud.
Swift is part of the OpenStack software platform. Objects are written to multiple disk drives spread throughout storage nodes in the cluster. All of this logic is managed by OpenStack so you as a user don’t have to think about how data is stored physically. Our (Elastx) Swift cluster span over more than one datacenter, which means all data stored is georedundant.
In the examples we will upload a PostgreSQL database dump to Swift, but it can be any type of file you want to back up. You can use Swift for other purposes other than backup, you can even serve static websites directly from Swift, but that is out of the scope for this post.
Swift can automatically delete an object after a specified amount of seconds passed. This means we don’t even need to manage objects due to expire in our code. Examples are shown with both permanent storage (default behaviour) and automatic expiration.
If you haven’t already, start by sourcing your openrc.sh to set up the environment in your shell. You can download your own openrc file by navigating to the “API Access” tab located in “Compute/Access & Security” when logged on to the OpenStack dashboard.
source ~/openrc.sh
python-swiftclient
python-swiftclient can be installed through most package managers and runs on any OS with Python >=2.6 support. Consult https://www.swiftstack.com/docs/integration/python-swiftclient.html for more information how to install.
First we create a container in Swift to hold all our backups:
swift post pg_backup
Then we upload our database dump to the newly created container:
swift upload pg_backup mydb.sql
If we want Swift to automatically expire a file we pass the header ‘X-Delete-After’ during the upload:
swift upload --header ‘X-Delete-After: 3600’ pg_backup mydb.sql
This will delete the object after 3600 seconds (1h).
We can list all objects in the container using `swift list pg_backup`. To read object metadata (size, modified, expiration, etc) we use:
swift stat pg_backup mydb.sql
cURL
Swift API is RESTful which means you can use cURL to upload and download objects in storage using HTTP protocol. cURL is available in all popular package managers. We also use jq to process the return json, so make sure you install that as well.
We need to authenticate first and save the X-Auth-Token header data and Swift URL we receive in order to use all subsequent commands. Syntax is different between API versions. Example is made for v3.
OS_AUTH_URL=https://ops.elastx.cloud:5000/v3/auth/tokens
tmpheader=$(mktemp)
tmpjson=$(mktemp)
curl -sf -D $tmpheader -o $tmpjson -H "Content-Type: application/json" -d '{ "auth": {
"identity": {
"methods": ["password"],
"password": {
"user": {
"name": "'"${OS_USERNAME}"'",
"domain": { "id": "default" },
"password": "'"${OS_PASSWORD}"'"
}
}
},
"scope": {
"project": {
"name": "'"${OS_TENANT_NAME}"'",
"domain": { "id": "default" }
}
}
}
}' $OS_AUTH_URL
OS_TOKEN=$(grep 'X-Subject-Token' $tmpheader | awk '{printf $2}' | tr -d "\r")
SWIFT_URL=$(cat $tmpjson | jq '.token.catalog[] | select(.name == "swift") | .endpoints[] | select(.interface == "public" ) | .url' | tr -d '"' )
rm $tmpheader
rm $tmpjson
You can add code above to the bottom of your openrc file so you auth directly when sourcing it.
Create container in Swift to hold backups:
curl -sf -H "X-Auth-Token: $OS_TOKEN" -X PUT $SWIFT_URL/pg_backup
Upload database dump to a Swift container:
curl -sf -H "X-Auth-Token: $OS_TOKEN" -H "Content-Type: application/x-sql" --data-binary "@mydb.sql" -X PUT $SWIFT_URL/pg_backup/mydb.sql
Upload database dump with 3600 seconds (1h) automatic expiration to a Swift container:
curl -sf -H "X-Auth-Token: $OS_TOKEN" -H "X-Delete-After: 3600" -H "Content-Type: application/x-sql" --data-binary "@mydb.sql" -X PUT $SWIFT_URL/pg_backup/mydb.sql
When done communicating we want to revoke the token as we don’t need it any longer:
curl -H "X-Auth-Token: $OS_TOKEN" -H "X-Subject-Token: $OS_TOKEN" -X DELETE $OS_AUTH_URL
Rclone
Rclone is built in Go and comes as a single binary file. Syntax is very much alike rsync. It supports many different types of cloud storage which makes data migration simple. You can download it at http://rclone.org/downloads/.
Create a file called rclone.conf with following content:
[swift-remote]
type = swift
user = $OS_USERNAME
key = $OS_PASSWORD
auth = $OS_AUTH_URL
domain = default
tenant = $OS_TENANT_NAME
When we want to refer to Swift we will now use 'swift-remote:' and rclone will handle auth for us (assuming openrc has been sourced first).
Create a container in Swift:
rclone --config rclone.conf mkdir swift-remote:pg_backup
Upload db dump to pg_backup container:
rclone --config rclone.conf copy mydb.sql swift-remote:pg_backup
Default behaviour is to copy entire directory contents unless source is a file.
At the time of writing (2017-04-07) the current version of rclone (v1.36) does not support expiration header for Swift. It's scheduled for v1.37 (Github issue #59) or later. Meanwhile you can delete old files with:
rclone --config rclone.config --min-age 1h delete swift-remote:pg_backup/
Update 2020-11-16: Since v1.52 rclone support sending additonal headers, so you can now set expiration date on file uploads:
rclone --config rclone.conf --header-upload "X-Delete-After: 3600" copy mydb.sql swift-remote:pg_backup
Summary
That's it! My personal recommendation is to use python-swiftclient or OpenStack CLI client (not shown in examples) where available. Both are maintained by the OpenStack community and are the most supported software out there.
Rclone can be useful if you use more than one cloud storage platform, or at least want the possibility to be able to. Rclone is the cloud equivalent to rsync, a software known to be excellent when it comes to copy large amounts of data and files.
cURL can be run almost anywhere (it even works on AmigaOS, DOS and OS/2!) but require more knowledge of Swift REST API.
If you have any questions or want to know more about how OpenStack can help your company contact me or a colleague and we'll be in touch!
Andreas Holmsten