Migrating Data Between Redis Servers Without Downtime

Image for post
Image for post
Photo by Ray Hennessy on Unsplash

Recently we had to migrate all the data from one Redis server to another. There are a few ways this can be done. However, we needed to migrate the data without downtime or any interruptions to the service.

We decided the best course of action was a three step process:

  1. Update the service to send all write commands (SET, DEL, etc) to both servers. Read commands (GET, etc) would still continue to goto the first server.

Now, step 1 and 3 were actually quite easy because we already have our own Redis client interface where it was easy to hide the sending of reads and writes to separate servers without any change in application logic. Thanks Go interfaces!

Step 2 is the bulk of the work and has a few gotchas:

  1. Redis provides a serialisation format from the DUMP command so that you can load any key into the same or a new server. However, this is a binary format and care needs to be taken when handling it correctly in Bash. More details about this later.
export OLD=”redis-cli -h foo.0001.usw2.cache.amazonaws.com”
export NEW=”redis-cli -h bar.clustercfg.usw2.cache.amazonaws.com”

I have obfuscated the original Redis hostnames. The important part is that you provide the CLI with any arguments you need to interact with the server.

Before we continue we should check that both servers are accessible and responding:

$OLD PING
# PONG
$NEW PING
# PONG

Great! Now let’s push the big red button. Or, at least look at it:

for KEY in $($OLD --scan); do
$OLD --raw DUMP "$KEY" | head -c-1 > /tmp/dump
TTL=$($OLD --raw TTL "$KEY")
case $TTL in
-2)
$NEW DEL "$KEY"
;;
-1)
$NEW DEL "$KEY"
cat /tmp/dump | $NEW -x RESTORE "$KEY" 0
;;
*)
$NEW DEL "$KEY"
cat /tmp/dump | $NEW -x RESTORE "$KEY" "$TTL"
;;
esac
echo "$KEY (TTL = $TTL)"
done

I should also mention that we were only copying around 320k keys. Bash seemed to have no problem with this (after all the keys were loaded) but if you are loading a huge amount of keys this may not work.

It might be helpful to read my other blog post: Deleting a Huge Number of Keys in Redis.

If you see an error similar to “(error) MOVED 660 10.1.174.29:6379” it’s because you are running against a cluster and will need to add the “-c” option to your redis-clicommand.

Originally published at http://elliot.land on November 27, 2018.

Written by

I’m a data nerd and TDD enthusiast originally from Sydney. Currently working for Uber in New York. My thoughts here are my own. 🤓 elliotchance@gmail.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store