You start doing some research about this topic when one of the following events occurs:
- You lose your data
- You start hosting third party data
We usually start making backups cause we don’t want to lose important bits. ( or at least we want to minimize the amount of lost data )
This question could be easily under-estimated. This is, imho, one of the most important question when talking about backups.
In my case I wanted to backup:
- My config files ( apache, mysql, nginx etc )
- The www directory
- mysql data
The backup location differs proportionally with the “security level” you want to get.
In some case scenarios it could be enough to backup your data using the same Hard Drive but just in a different folder.
So here i’ll propose a quick Q&A. list to help you pick the correct “Where” answer:
- Covered from unwanted file deletions => backup on a different folder
- Covered from hard drive failure => backup on a different hard drive
- Covered from robbering/flames/water => backup on a different machine on another location
- Covered from large scale issues ( Earthquakes, Tsunamis, Aliens? ) =>backup on different machine(s) on another country/continent
There’s no definitive answer to this question! There are only best-practices you can put in place!
The first rule: Running several servers I learned that the server containing the data we want to backup should not be ( as much as possible ) involved in the backup process. I used to backup my data with some scripts inside the same server, I discovered this approach is not scalable and somehow not reliable as I wanted it to be.
The second rule (which is also an effect of the previous one ) is: do not send your backup to the backup servers, let the B.S. fetch the data it needs. Why? Suppouse your server get hacked and the attacker decides to look around and find your scripts, this could lead to 2 problems:
- If you didn’t configure the backup server ACLs correctly you’re going to give the attacker free access also to your backup servers.
- They’ll be able to delete your data also from your backup servers.
The third rule: do not re-invent the wheel. There are a lot of free tools in the internet you can use. In my scenario i picked these ones:
- rsync: very popular sync tool for windows and linux
- rsnapshot: a very handy tool that works close with rsync to make incremental backups in a very efficient way
- mysqldump: the console based backup tool.
This is also a crucial question and you’ll need to think about the answer very carefully. It all depends on you and there are no strict rules on this.
I probably already mentioned I’ve some web servers. Since some days ago each of them were taking care about its own backup. I manually created a bunch of highly customizable scripts that I used for more than a year now.
Even if I don’t like to change things that simply works I love to improve & optimize.
The old setup was:
- Each web server had a script that was responsible of backing up files and databases,
- Each web server had another script which connected to a remote ftp and send those backups there
- The backup granularity was daily and I had at least 1 week of data.
The new setup i wanted was:
- A new backup server that takes care of everything.
- Only one place where i could change my backup parameters for all the servers.
Obviously the new setup is more complex and expensive than the previous one, but I didn’t want to live with the old setup anymore.
I searched for a backup server online and I eventually found a low-end dedicated server which had with very poor cpu performance but enough space for my needs.
First of all create a user
On the webserver you’d like to backup i suggest to add a brand new user that will handle all the stuff we need to backup our data. The whole “tutorial” is based on this step.
For you lazy sysadmins here is the shell command you should launch to create the user.
sudo adduser backupper
Note: I’ll refer to the “backupper” user in along the tutorial. If you decided to rename it then you should take care to replace the username below.
Setting up regular mysql backups with cronjobs
Data stored in mysql databases are usually, for a web server, a top priority that needs to be backed up regularly.
Before proceding setting up rsnapshot we will need to set up a mysql backup script that will be launched daily with a cronjob. Before creating the script we need to create a user that, for security purposes, is only allowed to issue commands that are needed to mysqldump.
In order to do that you’ve to open the mysql console using your mysql admin/root password:
mysql -u root -p
Then, choose a password for your new user and issue the following 2 sqls:
Note: replace the passwords with your own
Once created the user create a file named .my.cnf inside the home directory of backupper user. ( /home/backupper ) with the following content
In order to add a bit of security issue the following shell command
chmod 700 .my.cnf
This will prevent other users to read the mysql configuration file.
To test the whole thing issue the following shell command:
mysql -Bse "show databases"
If the result of this command is the list of your databases then everything was setted up correctly. If you get an error ( probably “Access denied” ) then you should double check all the previous steps.
Now that our mysql configuration is up and running we need a shell script that will take care to backup our databases for us. You can use the following simple script. Name it “backup_databases.sh” and place it inside /home/backupper.
If you’re familiar with bash scripts then you’ll easily understand how the code works. Btw, for the sake of clarity, every time you launch this script a complete backup of all your databases will be placed inside ~/mysqlbackups
The last step is to set up a cronjob that will launch the above mentioned shell script. It’s a good idea to let the cronjob start during the night ( expecially if your server is very busy during the day ). In my crontab i decided to fire the mysql backups at 01:10 each day.
sudo su backupper crontab -e
while editing the crontab add something like this
10 1 * * * /bin/bash /home/backupper/backup_databases.sh 2>&1 > /dev/null
Now everything related to mysql should be setted up!
Setting up SSH with Public/Private keys
Before going any further we need to make ssh authentication with public/private keys between the 2 servers work properly. There are a lot of tutorials online you can use but for clarity I’ll write the commands i launched:
On the backup server:
cd ~/.ssh ssh-keygen -f rsynckey -C 'My Rsync Key' -N '' -t rsa -q
Once the key is created you’ll need to copy the public key on the web server appending/creating something like this to the file /home/backupper/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC.......NbPjH [email protected]
In order to add some security to the connection i suggest to prepend the following to the previously added line ( Everything must go in the same line )
The entire line should look like the following after the mod:
from="ip_of_backup_server",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-rsa AAAAB3NzaC.......NbPjH [email protected]
This will restrict the incoming connections only from the specified ip ( You can also specify the hostname there ) and it will disable some unnecessary stuff.
Since from my backup server I want to backup files owned by root I need the “backupper” user to be in sudoers file. Then open up your /etc/sudoers file in the backup server and add the following:
backupper ALL= NOPASSWD: /usr/bin/rsync
Using this trick we’ll let the backupper user to run sudo only if he needs to execute the rsync binary.
In order to make things easier lets create ~/.ssh/config file in the Backup Server.
Host client1 HostName client1.tobackup.com Port 22 IdentityFile ~/.ssh/rsynckey
This will simplify us the life afterwards while configuring rsnapshot. If you want to test that everything is setted up correctly just fire in the backup server
[email protected]:~/$ ssh [email protected] PTY allocation request failed on channel 0
It could sound like an error message but it’s not! Infact if you get that response that will mean:
- The public/private key is setted up correctly
- The config file is setted up correctly
- You restricted the key to not have the pty ( Correct since we won’t use it )
If you get a Password prompt that means that something went wrong. Double check the previous steps!
Setting up rsnapshot
Why rsnapshot? Well, cause it’s perfect. It does perfectly handle my situation and it also save some disk space on your backup server using hard links instead of doing each time a full copy ( You’ll also save bandwidth ).
In order to set up rsnapshot correctly you’ll only have to install some goodies.
On both backup server and the web server install rsync:
sudo apt-get install rsync
(Backup server only) install rsnapshot
sudo apt-get install rsnapshot
From now on all the stuff will need to be done on the backup server only.
Like every good tool rsnapshot has a config file. We need to create a config file that will define a couple of things. Remember: TABS must separate all “columns”
In the rsnapshot.conf:
- line 2: defines the folder where rsnapshot will write the backups
- from line 3 to 8: we define absolute path to some binaries
- from line 9 to 11: we define the retain values. This will specify how many backups ( for the specified “timeframes” ) we want rsnapshot to “maintain”
- from line 13 to 15: we define what to backup.
- line 13: we want to backup the folder /home/backupper/mysqlbackups (look at the mysql section above) and we will log in to a host named “client1” (look at the ssh section above) using the “backupper” user.
- line 14: we want to backup the /etc folder. We also want to run rsync with “sudo”
- line 15: we want to backup the /var/www folder. We will also exclude some folders from the backup ( stats, logs, phptmp … ) since we don’t want to backup temporary files or logs.
Now that we’ve a config file we need to know if everything works. In order to try it just issue the following shell command:
rsnapshot -c /path/to/rsnapshot.conf -t daily
rsnapshot will read your config file and will echo all the commands that he will fire. “-t” specifies “run in test mode”. If no errors appear ( like tab errors ) it’s time to remove the “-t” parameter and see if everything works for real.
If everything works great ( could take some time depending on the size of your data ) It’s time to setup the needed cronjobs. All depends on which “timeframes” ( line 9 to 11 ) you use on your config file. If , for example, you use only the daily timeframe you’ll only need to setup a daily cronjob.
00 02 * * * rsnapshot -c /path/to/rsnapshot.conf daily
That will be fired at 2:00 am each day ( remember that if you mysqlbackup your data at 01:00 am you’ll need to be sure you download the fresh dumps after that cronjob ended that task ).
Now, the only thing you’ve to do is to sit and let rsnapshot do the hard job for you!
Conclusions + Security thoughts
Rsnapshot + rsync + ssh seems to be a good choice to make incremental and efficient backups. they seems to be robust and well known tools. About security you can even improve the level just by using rssh or tweaking the “command” parameter inside the ssh key. In my case the security thoughts adopted and implemented in this howto are enough.
I hope I actually helped someone to setup something cool and fully functional.