DRBD

DRBD (Distributed Replicated Block Device) is one way to implement warm standby on Linux systems by replicating an entire disk partition over the network. As changes are made to the master server, the secondary servers are kept in sync. If there’s a failure, you can bring up one of the secondaries.

I’ve been playing with DRBD at work today and found it pretty simple to configure using the instructions in The Ubuntu Server Giude

The most hassles I had were that I put data on the partition before I set up DRBD so I had to resize the filesystem to make room for the metadata block that DRBD wants to put at the end of the partition. This was not a big deal either in the end and there is actually an option to put the metadata elsewhere.

We have some scripts here at work that manage bringing up any services that need the drbd partition to be mounted. Note that the mounted device is the /dev/drbdN device, not /dev/sdaN. Unfortunately I can’t publish them here but it is not too difficult to write one. The basic concept is to put the sys V rc style links in a different directory and loop through them once DRBD is online. In our case we have the database on the DRBD partition so we don’t have to muck around with a separate database replication service.

DRBD might even be a decent solution for offsite backups if you have admin rights on both hosts. Just keep everything you need to keep backed up on a special partition and it will sync in the background (It looks like there’s a setting to throttle the bandwidth).

One limitation is that you can’t mount the partition on the secondary drbd nodes. You have to make them primary first and then you can mount them. I see that’s it’s possible to have dual primary nodes though.

Another gotcha seems to be that you must have drbd running to read from the partition so you can’t just grab the backup disk and boot from it.