Robust integration with Redis on Azure and Polly

A client of mine requested an integration with OpenWeatherMap, so like so many times before it was a chance to think about how to make such an integration robust and performant. Its as common a task as they come, but also something that tends to end up feeling more complex than I would like. Having heard a lot of good things and played a bit with Redis I felt that it would be a good choice for providing super fast caching, while also allowing for more than basic key/value storage.

Getting off the ground

The project is already running on Azure, so it was an obvious choice to give Azures new Redis based caching service a go. As of now the service is still in preview, but the the level of caching I need I feel quite comfortable with it. Getting started was as easy as most things on Azure - click add, fill in a name and press go. As every day as this has become, I am still blown away by how easy and fast it is every time I need to provision a new VM or service – and a Redis cache is no different.

On the downside I am not quite convinced by the new Azure portal, because to me the UX feels more shiny than useful. As of this writing the caching service is only available through the portal, but inspite of my reservations it was easy to get going, and it provides a nice overview of requests, storage space used as well as cache hits and misses.

On redis.io I saw a reference to a Redis client written by the good people at StackExchange. Already being a fan of some of their other libraries, like Dapper and Mini-Profiler, that Sam Saffron wrote, I felt comfortable leaning on their library. Of course there was a Nuget package for it, so minutes alter I was ready to get cracking. The API is very straight forward, uses familiar patterns, and works as you would expect as a .NET developer. As you can see below, you simply store by key and optionally pass in a time span until the value should be deleted from the cache.

using (var redis = ConnectionMultiplexer.Connect("xxx-xxx-xxx"))
{
    IDatabase db = redis.GetDatabase();

    db.StringSet(key, _binarySerializer.Serialize(myObject), new TimeSpan(2, 0, 0));

    var myObjectBackFromCache = _binarySerializer.Deserialize<T>(db.StringGet(key));
}

The only work to be done for my task was the serialization. So all in all it didn't take much code to get the basic caching in place.

Redis as more than a key/value store

If you are only interested in the integration story and know Redis, you can skip this section. So far what we looked at could be implemented with any key/value store, so the only upside so far is that Redis is a damned fast one. As mentioned before Redis is more than that - it is most popularly described as a data structure server. Conceptually this means that it can handle storing values, lists, maps, sets and sorted sets. I won't go into too much detail about it, because http://redis.io and other sites already do this very well. So just to make the point let's look at one of the more complex structures, just to see that it really is still quite simple. A sorted set is, as the name suggests, a list of unique values, that are sorted. In Redis each value is assigned a score, which is used for sorting. So one could emagine using this to implement any kind of system that needs a fast method of keeping score and ordering users, pages, products or what ever is in your domain.

using (var redis = ConnectionMultiplexer.Connect(_connectionStringProvider.RedisConnection))
{
    IDatabase db = redis.GetDatabase();
    db.SortedSetAdd("ProgrammingLanguages", "Powershell", 1);
    db.SortedSetAdd("ProgrammingLanguages", "SQL", 1);
    db.SortedSetAdd("ProgrammingLanguages", "Ruby", 2);
    db.SortedSetAdd("ProgrammingLanguages", "Python", 5);
    db.SortedSetAdd("ProgrammingLanguages", "JavaScript", 6);
    db.SortedSetAdd("ProgrammingLanguages", "C#", 7);
    db.SortedSetAdd("ProgrammingLanguages", "F#", 8);

    Console.WriteLine(db.SortedSetScore("ProgrammingLanguages", "Python").ToString());
    //Prints: 5

    db.SortedSetIncrement("ProgrammingLanguages", "F#", 1);

    var sortedSetRange = db.SortedSetRangeByRankWithScores("ProgrammingLanguages", 0, 5, Order.Descending); 

    Console.WriteLine(string.Join(",", sortedSetRange.Select(x => x.Score)));
    // Prints: 9,7,6,5,2,1

    Console.WriteLine(string.Join(",", sortedSetRange.Select(x => x.Element)));
    // Prints: F#,C#,JavaScript,Python,Ruby,SQL
}

Hopefully this shows that the concepts are quite simple and easy to work with. To me the challenge is to think about storage in this way, because it is different from other databases I have worked with – but it is surely something I will keep in mind for future projects.

Retrying with Polly, and why async is still tricky

After I wrote the OpenWeatherMap integration, the next step was to make use of the caching in a robust way. For this I turned to a small library named Polly. This fitted in nicely with the principle of using small, focused libraries that do a single thing well.

Polly lets you write two kinds of policies where you specify which exceptions to handle and either if you want to retry for a certain number of times, or if you want it to act as a circuit breaker if the exception is thrown a certain number of times. One thing that puzzled me, was that the two could not directly be combined to allow you to try a number of times and then break the circuit. To get this to work you will need to use nested policies. In the end I chose just tokeep it simple and go with one retry, because if hitting the cache ends up taking too much time it kinds of defeats the purpose of having it in the first place. So I ended up with something like the following.

var policy = Policy.Handle<RedisException>().Retry();

var result = policy.Execute(async () => await GetWeatherData());

Having written this I thought everything was looking good, but of course I wanted to try it out, so I wrote a small integration test. It turned out I had walked face first into one of the C# async gotchas. Nothing stopped me from returning a Task<WeatherData> from the function executed by my policy, but it actually breaks the logic in two distinct ways. In the above code, if GetWeatherData throws an exception it will be thrown after exiting the scope of the policy and it will be wrapped as an AggregateException, which the policy would not handle even if it had been in scope. Bad news indeed.

The big issue here is that although I see why it works that way, it is not directly obvious. I might be an idiot, but efter reading Thomas Petriceks post, that I linked to above, at least I know I am not the only idiot. After getting a better understanding of the subject I rewrote my logic so that I wait for the Task to finish, and then wrapped the entire policy execution as a Task instead.

Wrapping it up into one nice package

Now I was down to one more non-functional requirement. I wanted to bypass the cache rather than have the exception throw if Redis for some reason was unreachable. This was just a matter of running my function to get the weather data directly and returning the data from there when Redis throws an exception. With this in place a wrote a generalized CacheService, so i can do something similar for other integrations. In the end it looks something like this.

public Task<T> GetOrRun<T>(string key, Func<Task<T>> func)
{
    return Task.Factory.StartNew(() =>
    {
        try
        {
            return Policy.Handle<RedisException>().Retry().Execute(() =>
            {
                using (var redis = ConnectionMultiplexer.Connect(_connectionStringProvider.RedisConnection))
                {
                    IDatabase db = redis.GetDatabase();
                    var objectFromCache = _binarySerializer.Deserialize<T>(db.StringGet(key));

                    if (objectFromCache != null) { return objectFromCache; }
                    var task = func();
                    task.Wait();
                    T result = task.Result;

                    db.StringSet(key, _binarySerializer.Serialize(result), new TimeSpan(2, 0, 0));
                    return result;
                }
            });
        }
        catch (RedisException)
        {
            var task = func();
            task.Wait();
            return task.Result;
        }
    });
}

With this relatively small amount of code we have a reusable caching service, with retry logic that is bypassed if the service is down. Extending it with more configuration options would be fairly easy, but for now YAGNI – You meaning me.

With this exercise I have certainly gained two new friends in Redis and Polly – but I also got a taste of why our new pal async/await is no silver bullet.