Attempts to unit test code which uses a timer most often lead to tests like:
[Test]
public void SlowAndFragileTest()
{
// Setup
dataProviderMock = MockRepository.
GenerateMock<IDataProvider>();
systemUnderTest = new PollCommand(dataProviderMock);
// Act
systemUnderTest.StartPoll();
// Make sure the timer has been executed at least
// once. The action is started after 10 seconds.
Thread.Sleep(11000);
// Assert
dataProviderMock .AssertWasCalled(
x=>x.Retrieve());
}
This testing approach is problematic. The test always takes 11 seconds, which is far to slow for a unit test. Depending on the timer used it is not guaranteed at what exact point in time the action is triggered. The action itself also takes time, so we can't be 100% sure that the action is finished when we test reaches the assert statement.
Solution - Test code with timers in one thread
We can separate the timer mechanics from our code and use a special timer which executes the action on the current thread. We can also tell this timer how much time has passed by:
[Test]
public void ShouldRetrieveDataEvery10Seconds()
{
// Arrange
timerStub = new DeterministicTimer();
dataProviderMock = MockRepository.
GenerateMock<IDataProvider>();
systemUnderTest = new PollCommand(
timerStub,
dataProviderMock);
// Act
systemUnderTest.StartPoll();
// Now execute the timer actions on
// the current thread.
timerStub.TickSeconds(30);
// Assert
dataProviderMock.AssertWasCalled(
x=>x.Retrieve(),
y=>y.Repeat.Times(3));
}
This test is very fast and takes fractions of a second. This test is also robust because every code is executed on the current thread.
Our PollComand class we are testing now looks very simple:
Attempts to unit test multi-threaded code most often lead to tests like:
[Test]
public void FragileAndSlowTest()
{
// Setup
SystemUnderTest sut = new SystemUnderTest();
// Start new thread
sut.DoAsync();
// Make sure the thread has finished execution
Thread.Sleep(2000);
// Assert
...
}
This testing approach has two disadvantages:
From time to time this test fails when the thread is not started and executed within two seconds. This can happen for example when the PC is busy with other activities. We can call these kinds of tests fragile.
This test takes alway at least two seconds to run. This is far to slow for a unit test.
Solution 1 - Call functionality without multi-threading code
We can execute the method which is executed in the new thread directly excluding the concurrency aspect. Either we make the method 'public' or we extract the functionality of the method into a separate class and then call that method directly from the test:
[Test]
public void TestWithoutTouchingThreadingCode()
{
// Setup
SystemUnderTest sut = new SystemUnderTest();
// Directly call synchronous method
// which is internally called by DoAsync()
sut.Do();
// Assert
...
}
Advantages:
We are forced to separate concerns in the code (functionality and concurrency) leading to better maintainability.
Disadvantages:
We have to make the private method Do() public for the purpose of testing. In production we would never call the method from outside so we are violating the information hiding principle.
The DoAsync() method is not executed in any test. If we have a more complex concurrency behaviour than in the example like starting an additional thread when the first thread is finished, we will also not test this behaviour.
Solution 2 - Test multi-threading code in one thread
We can replace the thread starting object with a test stub, which executes the Do() method in the same thread as the unit test. The example shows C# code using tasks from the .NET Framework Class Library. The default TaskScheduler can be replaced with our own implementation called DeterministicTaskScheduler. The test executes all code in the current thread:
[Test]
public void TestMultiThreadingCodeSynchronously()
{
// Arrange
DeterministicTaskScheduler taskScheduler =
new DeterministicTaskScheduler();
SystemUnderTest sut =
new SystemUnderTest(taskScheduler);
// Execute multi-threaded code and
// return immediately.
sut.DoAsync();
// Now execute the new task on
// the current thread.
taskScheduler.RunTasksUntilIdle();
// Assert
...
}
See this blog post for a detailed description of the DoAsync() implementation and the DeterministicTaskScheduler.
The concept also works very well for code using timers:
[Test]
public void ShouldRetrieveDataEverySecond()
{
// Arrange
timerStub = new DeterministicTimer();
dataProviderStub = MockRepository.GenerateStub<IDataProvider>();
systemUnderTest = new PollCommand(timerStub, dataProviderStub);
// Act
systemUnderTest.StartPoll();
// Now execute the timer actions on
// the current thread.
timerStub.TickSeconds(3);
// Assert
dataProviderStub.AssertWasCalled(x=>x.Retrieve(),y=>y.Repeat.Times(3));
}
Advantages:
We are executing the DoAsync() method in our test.
Our multi-threading code can still use directly the concurrency methods of the .NET Framework Class Library like Task.Factory.StartNew()
Disadvantages:
We need to prepare our code to inject the thread starting object.
See also my MSDN Magazine article for general remarks about unit testing multithreaded code.
Unit testing multithreaded code seems sometimes impossible. We expect Unit Tests to run fast and to deliver predictable results. Tests which are executing code that spawns new threads are very often slow and fragile. These tests are not Unit Tests. Unit Tests are tests running our code in isolation from other parts of the system. Another part of the system are the classes providing access to the operating system multithreading capabilities. We use these classes to start new threads.
To test the functional part of our code with unit tests we have to exclude the concurrency and synchronization aspects. These aspects are system-wide concerns and have to be tested later separately with integration tests. Integration tests can for example put the system under load to uncover synchronisation problems between multiple threads.
But first we have to assure that the functional part of our code works as expected. We have to equip our code to be able to isolate the concurrency and synchronization aspects. Then we can write Unit Tests which are executing our functionality on the same thread as the unit test.
So for Unit Tests in conjunction with multithreaded code we can summarize:
Multithreaded code has to be equipped for isolating concurrency and synchronization aspects from functional aspects.
Unit Tests can only test the functional aspect of multithreaded code. They can not test concurrency and synchronization aspects of the code, because these are concerns outreaching a single unit.
The jMock Team has described an brilliant approach for Java in the 'jMock Cookbook: Test Multithreaded Code'. This concept can also be applied to C# code using Tasks of the .NET Framework Class Libarary. The default TaskScheduler can be replaced with our own implementation called DeterministicTaskScheduler to test our code synchronously in a deterministic way. Unit tests are then able to run the tasks on their own thread:
[Test]
public void ShouldUpdateTheStatusWhenDoAsync()
{
// Arrange
DeterministicTaskScheduler taskScheduler = new DeterministicTaskScheduler();
SystemUnderTest sut = new SystemUnderTest(taskScheduler);
Assert.AreEqual("Nothing done", sut.Status);
// Act
sut.DoAsync(); // starts a new task and returns immediately
Assert.AreEqual("Starting task", sut.Status);
// Now execute the new task
taskScheduler.RunTasksUntilIdle();
// Assert
Assert.AreEqual("Task done", sut.Status);
}
Our code which starts a new task has to provide dependency injection capability for the TaskScheduler. We start a new task in DoAsync() via Task.Factory.StartNew():
public class SystemUnderTest
{
private string status;
private TaskScheduler taskScheduler;
public SystemUnderTest(TaskScheduler taskScheduler)
{
status = "Nothing done";
this.taskScheduler = taskScheduler;
}
public string Status {get { return status;}}
public void DoAsync()
{
status = "Starting task";
Task.Factory.StartNew(
DoWork1,
new CancellationToken(),
TaskCreationOptions.None,
taskScheduler);
}
private void DoWork1()
{
// Do some work here
status = "Task done";
}
}
///
/// TaskScheduker for executing tasks on the same thread that calls RunTasksUntilIdle() or RunPendingTasks()
///
public class DeterministicTaskScheduler : TaskScheduler
{
private List<Task> scheduledTasks = new List<Task>();
#region TaskScheduler methods
protected override void QueueTask(Task task)
{
scheduledTasks.Add(task);
}
protected override bool TryExecuteTaskInline(Task task, bool taskWasPreviouslyQueued)
{
scheduledTasks.Add(task);
return false;
}
protected override IEnumerable<Task> GetScheduledTasks()
{
return scheduledTasks;
}
public override int MaximumConcurrencyLevel { get { return 1; } }
#endregion
///
/// Executes the scheduled Tasks synchronously on the current thread.
/// If those tasks schedule new tasks they will also be executed
/// until no pending tasks are left.
///
public void RunTasksUntilIdle()
{
while (scheduledTasks.Any())
{
this.RunPendingTasks();
}
}
///
/// Executes the scheduled Tasks synchronously on the current thread.
/// If those tasks schedule new tasks they will only be executed
/// with the next call to RunTasksUntilIdle() or RunPendingTasks().
///
public void RunPendingTasks()
{
foreach (var task in scheduledTasks.ToArray())
{
this.TryExecuteTask(task);
scheduledTasks.Remove(task);
}
}
}
In the client production code we can inject a real TaskScheduler from the current context to our class:
SystemUnderTest sut = new SystemUnderTest(
TaskScheduler.FromCurrentSynchronizationContext());
sut.DoAsync();
See also my MSDN Magazine article for general remarks about unit testing multithreaded code.
There are mainly two reasons for using asynchronous programming:
Utilize your CPU cores by parallelizing CPU intensive work.
Keep your application responsive while long running activities are in progress. These activities may be CPU intensive work or are waiting for a response from the web, lan or database (IO bound work)
Responsiveness is not only required for applications with an user interface but also for background services and applications. Background services should answer additional request even when they are currently busy with something else. On a user interface a user wants to at least be able to cancel a long running activity.
The following simple example demonstrate asynchronous programming with .NET 4.5 and async/await with an console application. The two long running activities (DoAction1() and DoAction2()) with CPU bound work are executed in parallel. The user can exit the application even before the computation and printing of the result has finished:
class Program
{
static void Main()
{
var program = new Program();
program.PrintResultAsync();
Console.WriteLine("Press <return> to exit ...");
Console.ReadLine();
}
public int DoAction1()
{
int result = 0;
for (int i = 0; i < 5; i++)
{
Console.WriteLine("Action1 "+ result);
Thread.Sleep(250);
result++;
}
return result;
}
public int DoAction2()
{
int result = 0;
for (int i = 0; i < 5; i++)
{
Console.WriteLine("Action2 " + result);
Thread.Sleep(250);
result--;
}
return result;
}
public async void PrintResultAsync()
{
Tuple<int, int> r = await ComputeResultAsync();
Console.WriteLine("Result " + r.Item1 + " + " + r.Item2 + " = " + (r.Item1 + r.Item2));
}
private async Task<Tuple<int, int>> ComputeResultAsync()
{
var t1 = Task<int>.Factory.StartNew(this.DoAction1);
var t2 = Task<int>.Factory.StartNew(this.DoAction2);
return new Tuple<int, int>(await t1, await t2);
}
}
The application has the output on the console below:
We can compare the previous async/await implementation with an equal implementation using plain threads. The thread based implementation is much longer and requires more overhead like additional member variables:
class Program
{
static void Main()
{
var program = new Program();
program.PrintResultWithThreads();
Console.WriteLine("Press <return> to exit ...");
Console.ReadLine();
}
public int DoAction1()
{
int result = 0;
for (int i = 0; i < 5; i++)
{
Console.WriteLine("Action1 "+ result);
Thread.Sleep(250);
result++;
}
return result;
}
public int DoAction2()
{
int result = 0;
for (int i = 0; i < 5; i++)
{
Console.WriteLine("Action2 " + result);
Thread.Sleep(250);
result--;
}
return result;
}
private int r1;
private int r2;
private ManualResetEvent e1 = new ManualResetEvent(false);
private ManualResetEvent e2 = new ManualResetEvent(false);
private WaitHandle[] manualEvents;
public void PrintResultWithThreads()
{
var printThread = new Thread(
() =>
{
e1 = new ManualResetEvent(false);
e2 = new ManualResetEvent(false);
manualEvents = new WaitHandle[] { e1, e2 };
ComputeResultsWithThreads();
WaitHandle.WaitAll(manualEvents);
Console.WriteLine("Result: " + r1 + " + " + r2 + " = " + (r1 + r2));
});
printThread.IsBackground = true;
printThread.Start();
}
private void ComputeResultsWithThreads()
{
var t1 = new Thread(
() =>
{
this.r1 = this.DoAction1();
this.e1.Set();
});
t1.IsBackground = true;
t1.Start();
var t2 = new Thread(
() =>
{
this.r2 = this.DoAction2();
this.e2.Set();
});
t2.IsBackground = true;
t2.Start();
}
}
In his book “The Clean Coder” Robert C. Martin tells several anecdotes from his professional live as a software developer to explain the attitude of a professional developer. That reminds me of a story when I started my current job.
It was around 2009 when I started to work in an organization with seven software development teams each with the size of approximately ten people. The teams delivered every two month new software versions which are then manually integrated to a running system on a medical device hardware. The integration took around additional four to six weeks. I couldn't believe that software was created with this “stone age” process. In the worst case the developers got feedback for their features from the fully integrated system only after three month. Oh my good, why do I started to work there? I talked to my boss about my concerns and he admitted that we have to change something. So I was appointed as a project manager. The core project team had two members: Matt, a very experienced software consultant and me.
We called the project “DailyBuild”. So our goal was to speed up the integration cycle time from ninety days to one. Yes, there where a lot of obstacles on our way. For example the teams used different source code control systems, we perceived the number of used build scripting tools as nearly infinite and the total software build time was about eight hours. Many problems we had to solve were related to the organizational structure of the development department. And of course if you want to change the way of working of people, a lot of them say: “Why should we change, we have always worked like this”. The only good thing was that I knew all the time that the project could not fail. I knew that the benefits would be overwhelming. The agile movement was already around several years and thousands of software teams had already adopted Continuous Integration.
After 4 month we had the first successful automated build including a simple smoke test of the integrated system on a virtual machine. It took some more weeks to finish things. Today, no one in the development department can imagine to work any more without the “DailyBuild” process in place. Yes we made it!
As an absolute beginner to the language I needed to write my first Perl script. As a big fan of Test-Driven Development (TDD) I thought it would be a good idea to start with a test when doing the first Perl program. And it worked really nice. This post should be a simple step-by-step tutorial for Perl beginners who want to write simple Unit Tests for Perl. I will use my first Perl script as an example.
The Example: A ClearCase trigger
To customize the behaviour of ClearCase you have to write Perl scripts
which can be associated with any ClearCase command as a so called
ClearCase trigger (see IBM Rational ClearCase: The ten best triggers). For my example, I needed a trigger that updates a FitNesse Wiki page (the file name is always "content.txt") when it is checked-in to ClearCase. If the file contains a string like "$Revision: \main\MAINLINE_SQE\3 $" the Perl script should update the version information. That's it.
Step-by-Step Tutorial
Install Perl.
Create a folder "PerlScripts" for the new Perl scripts. We will have two files in this folder: "CiVersionFitnesseTrigger.pl" is the Perl script for the trigger. "CiVersionFitnesseTriggerTests.pl" is the Perl script for the corresponding Unit Tests.
Download the Test::Simple Perl module. Unpack the gz archive. We will only need the file "Simple.pm" from the folder "lib/Test". Create a folder "Test" as sub folder of our "PerlScripts" folder. Copy the file "Simple.pm" to this "Test" folder.
We start writing our first test in "CiVersionFitnesseTriggerTests.pl":
use Test::Simple tests => 1;
# System under test
require 'CiVersionFitnesseTrigger.pl';
# Testing is_fitnesse_wiki_page() method
ok(FitTrigger::is_fitnesse_wiki_page('content.txt'), 'content.txt is page');
We start defining an empty sub routine and an empty main routine in "CiVersionFitnesseTrigger.pl":
package FitTrigger;
sub is_fitnesse_wiki_page {
return 0;
}
#
# Main method
#
1;
We can now run the first unit test and see it failing:
Now we have the infrastructure to start implementation. We fix the first failing test:
package FitTrigger;
sub is_fitnesse_wiki_page {
my ($file_name) = @_;
return $file_name =~ m/^(.*\\)?content\.txt$/
}
#
# Main method
#
1;
Now run the unit test again and it succeeds:
We continue the cycle of writing new unit tests and implementing the script step by step. In the end we have 12 unit tests and 1 integration test:
use Test::Simple tests => 13;
# System under test
require 'CiVersionFitnesseTrigger.pl';
# Testing is_fitnesse_wiki_page() method
ok(FitTrigger::is_fitnesse_wiki_page('content.txt'), 'content.txt is page');
ok(FitTrigger::is_fitnesse_wiki_page('c:\content.txt'), 'c:\content.txt is page');
ok(FitTrigger::is_fitnesse_wiki_page('..\content.txt') , '..\content.txt is page');
ok(FitTrigger::is_fitnesse_wiki_page('c:\temp\content.txt'), 'c:\temp\content.txt is page');
ok(!FitTrigger::is_fitnesse_wiki_page('content.txt.old') , 'content.txt.old is not a page');
ok(!FitTrigger::is_fitnesse_wiki_page('somecontent.txt') , 'somecontent.txt is not a page');
ok(!FitTrigger::is_fitnesse_wiki_page('content.txt\something.txt') , 'content.txt\something.txt is not a page');
# Testing getTempFolder() method
my $tmpFolder = FitTrigger::get_temp_folder();
ok(defined($tmpFolder) && $tmpFolder ne '' && length($tmpFolder) > 1 , 'temporary folder not empty');
# Testing getTempFile() method
my $tmpFile = FitTrigger::get_temp_file();
ok(defined($tmpFile) && $tmpFile ne '' && length($tmpFile) > 1 , 'temporary file not empty');
# Testing update_revision_in_target() method
my $testFile = "$tmpFolder\\test.txt";
my $targetFile = "$tmpFolder\\target.txt";
open("TESTFILE", ">$testFile") ||
&error("Could not open test File $testFile for writing");
print TESTFILE "hallo1\nhallo2\n\$Revision: VERSION_ZZZ \$\n";
close TESTFILE;
my $newVersion = 'VERSION_111';
FitTrigger::update_revision_in_target($testFile,$targetFile,$newVersion);
open(F,"$targetFile");
my @list = ;
my $content=join('',@list);
close F;
my $expectedContent = "hallo1\nhallo2\n\$Revision: VERSION_111 \$\n";
ok($content eq $expectedContent, 'version was updated in target file');
# Testing overwrite_file() method
FitTrigger::overwrite_file($targetFile,$testFile);
open(F2,"$testFile");
@list=;
my $newContent =join('',@list);
close F2;
ok($newContent eq $expectedContent, 'file was overwritten with a modified file');
ok(! -e $targetFile, 'modified file is deleted');
# Testing main() method
$testFile = "$tmpFolder\\content.txt";
open("TESTFILE", ">$testFile") ||
&error("Could not open test File $testFile for writing");
print TESTFILE "hallo1\nhallo2\n\$Revision: VERSION_ZZZ \$\n";
close TESTFILE;
$ENV{CLEARCASE_PN}=$testFile;
$ENV{CLEARCASE_ID_STR}='VERSION_888';
system ("perl CiVersionFitnesseTrigger.pl");
my $expectedContentMain = "hallo1\nhallo2\n\$Revision: VERSION_888 \$\n";
open(F3,"$testFile");
@list=;
my $newContentMain =join('',@list);
close F3;
ok($newContentMain eq $expectedContentMain, 'perl script has updated content.txt');
The complete implementation in "CiVersionFitnesseTrigger.pl" looks like:
package FitTrigger;
sub is_fitnesse_wiki_page {
my ($file_name) = @_;
return $file_name =~ m/^(.*\\)?content\.txt$/
}
sub get_temp_folder {
my $tmp_folder = $ENV{TMP};
$tmp_folder = $ENV{TEMP} unless ($tmp_folder);
$tmp_folder = "/tmp" unless ($tmp_folder);
return $tmp_folder;
}
sub get_temp_file {
my $tmp_folder = get_temp_folder();
return "$tmpFolder\\ccTriggerTmp.$$";
}
sub update_revision_in_target {
my $source = @_[0];
my $target = @_[1];
my $revision = @_[2];
open("SOURCE", "$source") ||
&error("Could not open source file $source for reading");
open("TARGET", ">$target") ||
&error("Could not open target file $target for reading");
while ()
{
if (/\$Revision:?.*\$/) {
s/\$Revision:?.*\$/\$Revision: $revision \$/;
}
print TARGET;
}
close SOURCE;
close TARGET;
}
sub overwrite_file {
my $source = @_[0];
my $target = @_[1];
open (SOURCE, "$source") ||
&error ("Could not open source file $source for reading");
open (TARGET, ">$target") ||
&error ("Could not open target file $target for writing");
while() {
print TARGET;
}
close(SOURCE);
close(TARGET);
unlink($source);
}
sub error {
my ($message) = @_;
die ($message."\nUnable to continue checkin ...\n");
}
#
# Main method
#
# Summary:
# If the name of the checkin file is ‘content.txt’ then search in the content of the file for a string like
# „$Revision: \main\MAINLINE_22_WIPID\4 $“. This string will then be replaced
# with e.g. „$Revision: \main\MAINLINE_22_WIPID\5 $“.
my $check_in_file = $ENV{'CLEARCASE_PN'};
my $revision = $ENV{'CLEARCASE_ID_STR'};
if(is_fitnesse_wiki_page($check_in_file)) {
my $targetFile = get_temp_file();
update_revision_in_target($check_in_file,$targetFile,$revision);
overwrite_file($targetFile,$check_in_file);
}
1;
Currently I'm part of a team that tries to introduce test automation to our organization. We are developing products for the healthcare sector with relatively long release cycles due to high regulatory requirements. These long release cycles are resulting mainly because of high manual test efforts and missing test automation.
There were some discussions which tools to use for test automation. Two main possibilities are available:
Traditional commercial, heavyweight, GUI-based, record and replay tools like HP QuickTest Professional, IBM Rational Robot, QF-Test or Borland SilkTest.
In a pilot project we found some advantages of Fitnesse over traditional commercial testing tools:
No Licence costs for Fitnesse
Ok, in a big company it's not a big issues to spend some money for commercial tools, but even then you will not buy licences for every machine and every employee. Fitnesse we use on every developer machine, on every tester laptop, on every machine in the test lab, on laptops for presentations in meetings. You can use it on every machine you want without filling order forms and waiting weeks for completion of the order process. So the use of Fitnesse is not limited to the test specialist but instead we can use it cross-functional for application specialists, testers, developers and software architects.
Simple Installation of Fitnesse
Fitnesse can be brought to a machine simply by copying a folder with its subfolders. Or you can run it from an USB stick, which is quite practical for tests on systems wich are not connected to the corporate network.
Test-First approach with Fitnesse
It is a natural approach to write down the test specification before or during the development of the software, because developers need the input to provide test fixtures for connecting the Fitnesse tests with the production code.
Please refer to Elisabeth Hendrickson's blog for similar and more advantages:
Agile-Friendly Test Automation Tools/Frameworks
