Deleting A Huge Number of Keys In Redis

We needed to cleanup (delete) some archived keys from Redis. A lot of them.

Prepare All The Keys To Delete

The first step is to create a file with all of the keys to be deleted. We exported this from our database, and it’s just a text file with one key per line, it looks like this:

The result file, which I will call all_redis_keys.txt contains 92,402,859 keys (3.1 GB).

Dry Run

Trying to execute one Redis request per key would be far too slow. Luckily DEL allows multiple keys to be passed to in and returns the number of keys deleted.

Before we unleash this on Redis we want to make sure that it’s going to work. Fortunately we can substitute the DEL command with EXISTS command which returns the number of keys that exist (ie. the number of keys that would have been deleted).

Let’s run through the first 100,000 keys in batch sizes of 1000 and output the number to another file:

The tee command will output as well as writing to count.txt. If this is too noisy for you then you could just write directly to the file with > count.txt.

The count.txt (or stdout) file looks like this:

We can add up all of these numbers:

This returns that 589,742 keys would have been deleted.

The Real Run

real    104m33.7s
user 3m34.005s
sys 0m48.68s

Now count the number of keys that were deleted:

Over 90 million keys deleted in less than 2 hours (14,526 per second).

Originally published at http://elliot.land on July 6, 2018.

Written by

I’m a data nerd and TDD enthusiast originally from Sydney. Currently working for Uber in New York. My thoughts here are my own. 🤓 elliotchance@gmail.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store