Unit testing multi-threaded code

Attempts to unit test multi-threaded code most often lead to tests like:

[Test]
public void FragileAndSlowTest()
{
    // Setup
    SystemUnderTest sut = new SystemUnderTest();
     
    // Start new thread
    sut.DoAsync(); 
 
     // Make sure the thread has finished execution
    Thread.Sleep(2000);
 
    // Assert
    ...
}

This testing approach has two disadvantages:

1. From time to time this test fails when the thread is not started and executed within two seconds. This can happen for example when the PC is busy with other activities. We can call these kinds of tests fragile. 
2. This test takes alway at least two seconds to run. This is far to slow for a unit test.

Solution 1 - Call functionality without multi-threading code 

We can execute the method which is executed in the new thread directly excluding the concurrency aspect. Either we make the method 'public' or we extract the functionality of the method into a separate class and then call that method directly from the test:

[Test]
public void TestWithoutTouchingThreadingCode()
{
    // Setup
    SystemUnderTest sut = new SystemUnderTest();
     
    // Directly call synchronous method 
    // which is internally called by DoAsync()
    sut.Do(); 
 
    // Assert
    ...
}

Advantages:

• We are forced to separate concerns in the code (functionality and concurrency) leading to better maintainability. 

Disadvantages:

• We have to make the private method Do() public for the purpose of testing. In production we would never call the method from outside so we are violating the information hiding principle. 
• The DoAsync() method is not executed in any test. If we have a more complex concurrency behaviour than in the example like starting an additional thread when the first thread is finished, we will also not test this behaviour. 

Solution 2 - Test multi-threading code in one thread

We can replace the thread starting object with a test stub, which executes the Do() method in the same thread as the unit test. The example shows C# code using tasks from the .NET Framework Class Library. The default TaskScheduler can be replaced with our own implementation called DeterministicTaskScheduler. The test executes all code in the current thread:

[Test]
public void TestMultiThreadingCodeSynchronously()
{
    // Arrange
    DeterministicTaskScheduler taskScheduler =
        new DeterministicTaskScheduler();
    SystemUnderTest sut = 
        new SystemUnderTest(taskScheduler);
     
    // Execute multi-threaded code and 
    // return immediately.
    sut.DoAsync();

    // Now execute the new task on
    // the current thread.
    taskScheduler.RunTasksUntilIdle();
 
    // Assert
    ...
}

See this blog post for a detailed description of the DoAsync() implementation and the DeterministicTaskScheduler.
The concept also works very well for code using timers:

[Test]
public void ShouldRetrieveDataEverySecond()
{
    // Arrange
    timerStub = new DeterministicTimer();
    dataProviderStub = MockRepository.GenerateStub<IDataProvider>();
    systemUnderTest = new PollCommand(timerStub, dataProviderStub);

    // Act
    systemUnderTest.StartPoll();
    // Now execute the timer actions on
    // the current thread.
    timerStub.TickSeconds(3);

    // Assert
    dataProviderStub.AssertWasCalled(x=>x.Retrieve(),y=>y.Repeat.Times(3));
}

Advantages:

• We are executing the DoAsync() method in our test.
• Our multi-threading code can still use directly the concurrency methods of the .NET Framework Class Library like Task.Factory.StartNew()  

Disadvantages:

• We need to prepare our code to inject the thread starting object. 

See also my MSDN Magazine article for general remarks about unit testing multithreaded code.

Unit testing c# code which starts new threads via tasks

Unit testing multithreaded code seems sometimes impossible. We expect Unit Tests to run fast and to deliver predictable results. Tests which are executing code that spawns new threads are very often slow and fragile. These tests are not Unit Tests. Unit Tests are tests running our code in isolation from other parts of the system. Another part of the system are the classes providing access to the operating system multithreading capabilities. We use these classes to start new threads.
To test the functional part of our code with unit tests we have to exclude the concurrency and synchronization aspects. These aspects are system-wide concerns and have to be tested later separately with integration tests. Integration tests can for example put the system under load to uncover synchronisation problems between multiple threads.
But first we have to assure that the functional part of our code works as expected. We have to equip our code to be able to isolate the concurrency and synchronization aspects. Then we can write Unit Tests which are executing our functionality on the same thread as the unit test.
So for Unit Tests in conjunction with multithreaded code we can summarize:

• Multithreaded code has to be equipped for isolating concurrency and synchronization aspects from functional aspects. 
• Unit Tests can only test the functional aspect of multithreaded code. They can not test concurrency and synchronization aspects of the code, because these are concerns outreaching a single unit.

The jMock Team has described an brilliant approach for Java in the 'jMock Cookbook: Test Multithreaded Code'. This concept can also be applied to C# code using Tasks of the .NET Framework Class Libarary. The default TaskScheduler can be replaced with our own implementation called DeterministicTaskScheduler to test our code synchronously in a deterministic way. Unit tests are then able to run the tasks on their own thread:

[Test]
public void ShouldUpdateTheStatusWhenDoAsync()
{
    // Arrange 
    DeterministicTaskScheduler taskScheduler = new DeterministicTaskScheduler();
    SystemUnderTest sut = new SystemUnderTest(taskScheduler);
    Assert.AreEqual("Nothing done", sut.Status);
    
    // Act
    sut.DoAsync(); // starts a new task and returns immediately
    Assert.AreEqual("Starting task", sut.Status);
    // Now execute the new task
    taskScheduler.RunTasksUntilIdle();

    // Assert
    Assert.AreEqual("Task done", sut.Status);
}

Our code which starts a new task has to provide dependency injection capability for the TaskScheduler. We start a new task in DoAsync() via Task.Factory.StartNew():

public class SystemUnderTest
{
 private string status;
 private TaskScheduler taskScheduler;

 public SystemUnderTest(TaskScheduler taskScheduler)
 {
  status = "Nothing done";
  this.taskScheduler = taskScheduler;
 }

 public string Status {get { return status;}}

 public void DoAsync()
 {
  status = "Starting task";
  Task.Factory.StartNew(
      DoWork1, 
      new CancellationToken(), 
      TaskCreationOptions.None, 
      taskScheduler);
 }

 private void DoWork1()
 {
  // Do some work here
  status = "Task done";
 }
}

The DeterministicTaskScheduler to be used in unit tests:

    /// 

    /// TaskScheduker for executing tasks on the same thread that calls RunTasksUntilIdle() or RunPendingTasks() 
    /// 
    public class DeterministicTaskScheduler : TaskScheduler
    {
        private List<Task> scheduledTasks = new List<Task>(); 

        #region TaskScheduler methods

        protected override void QueueTask(Task task)
        {
            scheduledTasks.Add(task);
        }

        protected override bool TryExecuteTaskInline(Task task, bool taskWasPreviouslyQueued)
        { 
            scheduledTasks.Add(task);
            return false;
        }

        protected override IEnumerable<Task> GetScheduledTasks()
        {
            return scheduledTasks;
        }

        public override int MaximumConcurrencyLevel { get { return 1; } }

        #endregion

        /// 

        /// Executes the scheduled Tasks synchronously on the current thread.
        /// If those tasks schedule new tasks they will also be executed
        /// until no pending tasks are left.
        /// 
        public void RunTasksUntilIdle()
        {
            while (scheduledTasks.Any())
            {
                this.RunPendingTasks();
            }
        }

        /// 

        /// Executes the scheduled Tasks synchronously on the current thread.
        /// If those tasks schedule new tasks they will only be executed 
        /// with the next call to RunTasksUntilIdle() or RunPendingTasks(). 
        /// 
        public void RunPendingTasks()
        {
            foreach (var task in scheduledTasks.ToArray())
            {
                this.TryExecuteTask(task);
                scheduledTasks.Remove(task);
            }
        }
    }

In the client production code we can inject a real TaskScheduler from the current context to our class:

SystemUnderTest sut = new SystemUnderTest(
                      TaskScheduler.FromCurrentSynchronizationContext());
sut.DoAsync();

See also my MSDN Magazine article for general remarks about unit testing multithreaded code.