How to retry generically in C#

Introduction

The elmcity service makes API calls to many other services, including the Azure table service, Delicious, Eventful, Upcoming, Eventbrite, Facebook, and a wide variety of iCalendar sources. In this week's companion article on the Radar blog, I discuss why it's hard for users to master the principle that enables this kind of loose coupling: indirection. Here I'll explore one of the practical problems for developers. When your service relies on partner APIs, how do you handle transient failures when calling those APIs?

One part of the answer is, of course, to cache responses, and I'll explore the elmcity service's use of Azure's cache system another time. Another part of the answer is to retry API calls, and that's what I'll focus on here.

A generic retry capability

I suppose there are programmers who can immediately see the need for generic solutions to problems, and then create those solutions proactively. I'm not one of them. I need to repeat myself a few times, in a few different contexts, in order to find out how to apply the DRY (Don't Repeat Yourself) principle. In this case, I wound up writing a few different wait/retry loops before I was ready to generalize.

The basic idea is straightforward: retry a method, wait between retries, end on an evaluated condition. But what's The Right Way To Do It? On Stack Overflow I read many opinions (1, 2, 3). In the end I concluded that There's More Than One Way To Do It, and tried to accommodate a few different styles. To illustrate them, let's run through the test cases:

Succeed on the first try

private int retries;

private bool CompletedIfIntIsTwo(int i, object o)
{
  return i == 2;
}

private int Twice(int i)
{
  retries++;
  return i * 2;
}

[Test]
public void RetrySucceedsOnFirstTry()
{
  retries = 0;
  var completed_delegate =
    new GenUtils.Actions.CompletedDelegate<int, object>(CompletedIfIntIsTwo);
  var r = GenUtils.Actions.Retry<int>(
    delegate() { return Twice(1); },
    completed_delegate,
    completed_delegate_object: null,
    wait_secs: 0,
    max_tries: 1,
    timeout_secs: TimeSpan.FromSeconds(10000));
  Assert.AreEqual(2, r);
  Assert.AreEqual(1, retries);
}

This test illustrates the pattern. GenUtils.Actions.Retry is the generic retry method, parameterized by some type. In this example, Twice is the retried method. It returns an integer; in production the return type is typically an HttpResponse or some type that encapsulates an HttpResponse. CompletedIfIntIsTwo is the method that tests for completion. The call to GenUtils.Actions.Retry takes six arguments:

1. A delegate that wraps the method to be retried.

2. A delegate that wraps the method to test for completion.

3. An object that can optionally transmit extra data to the completion delegate.

4. How many seconds to wait before retrying.

5. How many tries are allowed before giving up.

6. How many seconds can elapse before giving up.

In this test, since Twice(1) returns 2, the test should succeed on the first try.

Succeed on subsequent try

private bool CompletedIfIntEndsWithZero(int i, object o)
{
  retries++;
  if (retries == 1)
    return false;
  String s = Convert.ToString(i);
  return s.EndsWith("0");
}

private int RandomEvenNumber()
{
  retries++;
  var ticks_as_str = Convert.ToString(System.DateTime.Now.Ticks);
  var seed_string = ticks_as_str.Substring(ticks_as_str.Length - 4, 4);
  var random = new Random(Convert.ToInt32(seed_string));
  var i = random.Next();
  while (i % 2 != 0)
    i = random.Next();
  return i;
}

[Test]
public void RetrySucceedsOnSubsequentTry()
{
  retries = 0;
  var completed_delegate =
    new GenUtils.Actions.CompletedDelegate<int, object>(CompletedIfIntEndsWithZero);
  var r = GenUtils.Actions.Retry<int>(
    delegate() { return RandomEvenNumber(); },
    completed_delegate,
    completed_delegate_object: null,
    wait_secs: 0,
    max_tries: 10000,
    timeout_secs: TimeSpan.FromSeconds(10000));
  Assert.That(Convert.ToString®.EndsWith("0"));
  Assert.That(retries > 1);
}

This test retries RandomEvenNumber. Sooner or later it will return a number that ends with zero. It excludes the case where that happens on the first try, sets the bar high for max_tries and timeout_secs, and waits for the expected result: a number that ends with zero.

End when too many tries

private bool CompletedNever(int i, object o)
{
  return false;
}

[Test]
public void RetryEndsAfterMaxTriesExceeded()
{
  var completed_delegate =
    new GenUtils.Actions.CompletedDelegate<int, object>(CompletedNever);
  try
  {
    var r = GenUtils.Actions.Retry<int>(
      delegate() { return RandomEvenNumber(); },
      completed_delegate,
      completed_delegate_object: null,
      wait_secs: 1,
      max_tries: 5,
      timeout_secs: TimeSpan.FromSeconds(100));
  }
  catch (Exception e)
  {
    Assert.AreEqual(GenUtils.Actions.RetryExceededMaxTries, e);
  }
}

This test uses a completion test that will never succeed. It waits 1 second per try, sets max_tries low and timeout_secs high, and expects to see a RetryExceededMaxTries exception.

End when timeout expires

[Test]
public void RetryEndsAfterTimeout()
{
  var completed_delegate =
    new GenUtils.Actions.CompletedDelegate<int, object>(CompletedNever);
  try
  {
    var r = GenUtils.Actions.Retry<int>(
      delegate() { return RandomEvenNumber(); },
      completed_delegate,
      completed_delegate_object: null,
      wait_secs: 1,
      max_tries: 100,
      timeout_secs: TimeSpan.FromSeconds(5));
  }
  catch (Exception e)
  {
    Assert.AreEqual(GenUtils.Actions.RetryTimedOut, e);
  }
}

This test also uses CompletedNever. It waits 1 second per try, sets max_tries high and timeout_secs low, and expects to see a RetryTimedOut exception.

Retry transmits exception

private int ExceptionIfOdd(int i)
{
  if (i % 2 != 0)
    throw new Exception("OddNumberException");
  return i;
}

[Test]
public void RetryTransmitsException()
{
  var completed_delegate =
    new GenUtils.Actions.CompletedDelegate<int, object>(CompletedNever);
  try
  {
    var r = GenUtils.Actions.Retry<int>(
      delegate() { return ExceptionIfOdd(1); },
      completed_delegate,
      completed_delegate_object: null,
      wait_secs: 0,
      max_tries: 100,
      timeout_secs: TimeSpan.FromSeconds(5));
  }
  catch (Exception e)
  {
    Assert.AreEqual("OddNumberException", e.Message);
  }
}

This test retries a method that throws an exception, and expects the retry logic to propagate that exception.

An implementation of generic retry

Now that we've seen how to use this implementation of generic retry, here's the code:

public class Actions
{
  public static Exception RetryExceededMaxTries = new Exception("RetryExceededMaxTries");
  public static Exception RetryTimedOut = new Exception("RetryTimedOut");
 
  public delegate T RetryDelegate();
 
  public delegate bool CompletedDelegate(T result, Object o);
 
  public static T Retry(RetryDelegate Action, CompletedDelegateObject> Completed, 
    Object completed_delegate_object, int wait_secs, int max_tries, TimeSpan timeout_secs)
  {
    T result = default(T);
    DateTime start = DateTime.UtcNow;
    int tries = 0;
    var method_name = Action.Method.Name;
    while (TimeToGiveUp(start, timeout_secs, tries, max_tries) == false)
    {
      try
      {
        tries++;
        result = Action.Invoke();
        if (Completed(result, completed_delegate_object) == true)
          return result;
      }
      catch (Exception e)
      {
        LogMsg("exception", "RetryDelegate: " + method_name, 
          e.Message + e.StackTrace);
        throw e;
      }
      HttpUtils.Wait(wait_secs);
    }
 
    if (TimedOut(start, timeout_secs))
      throw RetryTimedOut;
 
    if (ExceededTries(tries, max_tries))
      throw RetryExceededMaxTries;
 
    return result;
  }
 
  private static bool TimeToGiveUp(DateTime start, TimeSpan timeout_secs, 
    int tries, int max_tries)
  {
    var timed_out = TimedOut(start, timeout_secs);
    var exceeded_tries = ExceededTries(tries, max_tries);
    var result = (timed_out || exceeded_tries);
    return result;
  }
 
  private static bool ExceededTries(int tries, int max_tries)
  {
    var exceeded_tries = tries > max_tries;
    return exceeded_tries;
  }
 
  private static bool TimedOut(DateTime start, TimeSpan timeout_seconds)
  {
    var timed_out = DateTime.UtcNow > start + timeout_seconds;
    return timed_out;
  }
}

I have a confession to make. This code stretches the limits of my still-limited understanding of C# generics and delegates. But writing the tests helped me to think about the problem I'm trying to solve. And running the tests in the debugger has enabled me visualize what's going on. I think this is a reasonable solution, and it seems to work well in practice. But there's always more to learn, and I welcome feedback!