Unit tests - Our Story
Are there any software developers nowadays who have not heard of unit tests and all their alleged benefits? Well, most of us have at least some understanding of what it is all about. And some fraction of us even use and take advantage of them.
But in case you have not got the slightest clue about what unit tests are, here are some excerpts from Wikipedia:
“In computer programming, unit testing is a software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested to determine whether they are fit for use.”
“Unit testing finds problems early in the development cycle”
“Unit testing allows the programmer to refactor code or upgrade system libraries at a later date, and make sure the module still works correctly (e.g., in regression testing). The procedure is to write test cases for all functions and methods so that whenever a change causes a fault, it can be quickly identified. Unit tests detect changes which may break a design contract. Unit testing may reduce uncertainty in the units themselves and can be used in a bottom-up testing style approach. By testing the parts of a program first and then testing the sum of its parts, integration testing becomes much easier.”
Also worth pointing out is that unit tests are mostly used for testing business logic or low-level functions. These parts form the foundation of your software and must be robust and reliable. Otherwise not much else will work in the higher levels of your application. Thus it is extremely important to avoid bugs in this area.
So we can probably all agree that unit tests are valuable and should be adapted in your software development process.
But how many developers actually use unit tests? And what is their experience?
We will tell you our story, with the hope that you will gain some insight and get you started with your own unit tests. Even if we almost exclusively use Delphi for our development work, our implementation also applies for other tools, should you happen to use those.
We have been actively developing and maintaining our Delphi products (Pascal Analyzer, Pascal Browser, and Pascal Expert) since 1999. Yet, it was just a few years ago that we started implementing unit tests. With hindsight, we should of course have begun much earlier!
Our products are quite complex. The source code base consists of hundreds of thousands lines of code, and this is without counting third-party code. The code consists of different parts like lexer, parser, symbol finder, analyzer etc. Output is in form of text or HTML reports that list various problems and aspects of the code. Even very small changes in the code base for Pascal Analyzer can significantly affect results in these reports.
For example: A user reports a problem in one of the report sections in our main product Pascal Analyzer's Warnings Report and the section: “WARN7-Local variables that are referenced before they are set”. This is just one of hundreds of report sections in the product, but among the most important and of most value to our users, when searching for code errors.
The problem reported by our user in this case, is that there is a false positive report in a special situation for this particular report section. We quickly check the relevant code in RAD Studio and indeed understand that this is an edge case which needs to be handled in a special way.
Nemas problemas! Code fix is applied; the false positive is gone! Release an update and everything is cool. Take a break, or do whatever makes you feel good and relax.
That is, until reports from users start pouring in: There are now false positives in other reports! And these false positives did not exist before the latest update. So, a regression bug rears its ugly head! Bad! Back into the Delphi IDE to fix the problem and release a new patch! This time everything works as it should.
To avoid these embarrassing moments, unit tests then came to the rescue for us. Starting with unit tests a few years ago, there has been a steady growth in numbers, and we now keep a collection of more than 600 tests. These are run while developing and correcting code, as well as automatically each night. Sometimes they are run after each code change, just to make sure that nothing is messed up. We have got to the point where we can be fairly sure our software works, just if the unit tests pass. So one could really say that unit tests have helped us maintain and increase the quality of our products!
A particular lesson learnt in the process is:
Unit tests are not primarily for finding errors. It is more about assuring that new errors or so called regression bugs do not slip into a release. Which is very important! Nobody wants to ship something faulty, and particularly not to take care of the mess it potentially creates.
So how did we implement unit tests in our Pascal Analyzer product? First, we built our own unit testing engine. Which sounds a bit overambitious, but actually it was just a couple of hours of work. A natural and good choice is otherwise to use an existing solution available for Delphi, like DUnitX which is open source. But as said, we decided to roll our own. The advantage with keeping our own infrastructure is that we are not dependent on a third-party product. But your mileage may vary. You should definitely consider DUnitX or a similar framework before settling on a home-brewn solution.
To start with, we created a clone of our Delphi project for the command-line based analyzer, PalCmd.dproj. The new project was called PalUnittests.dproj and changed to run unit tests. So this is just a project for our own use, and is not released externally.
To run unit tests, we just execute the program from the command-line, and it will write the results to standard output just like other command-line programs.
We then created a handler class with some simple methods to run the tests and report the results. The code snippets shown in the following are just for illustration purposes, and is nothing you need to understand or learn. View them as samples and inspiration for your own efforts.
..
type
TUnitTester = class
private
FChecker : TObjectChecker;
FCodeHandler : TCodeHandler;
public
constructor Create(CodeHandler : TCodeHandler);
destructor Destroy; override;
procedure Test1_GenericsSet;
procedure Test2_Tokenizer;
procedure Test3_As;
procedure Test4_Assigned;
// and so on for all tests..
..
end;
The handler class contains methods to run the unit tests, either all of them or just a single one. Running just one specific unit test comes handy when working with a specific problem.
Each particular unit test is in its own Delphi unit, which helps to keep the overview of available tests. In our case the unit contains code that is parsed and analyzed. Like in this example:
unit UnitTestsCode19;
interface
implementation
type
THelpedClass = class
end;
THelperClass = class helper for THelpedClass
public
procedure DoSomething;
end;
TByteHelper = record helper for Byte
public
function ToStr : string; overload; inline;
end;
procedure THelperClass.DoSomething;
begin
end;
function TByteHelper.ToStr: string;
begin
end;
procedure __Helpers;
var
Obj : THelpedClass;
S : string;
B : Byte;
begin
Obj := THelpedClass.Create;
Obj.DoSomething;
B := 555;
S := B.ToStr;
end;
end.
Of course, the code is just for unit testing, so it does not make much sense otherwise, which you can probably see from the code above.
The handler class has a method that does the actual testing and determines if the test passes or if it fails:
procedure TUnitTester.Test19_Helpers;
var
Obj : TAbsBaseElement;
begin
Set1 := [kfRead];
Set2 := [kfRead, kfAssignSource];
FChecker.VerifyObject('__Helpers', FCodeHandler.VarsList, 'Obj', '', [Set1], 'UnitTestsCode19');
FChecker.VerifyProc('', FCodeHandler.StrucProcsList, 'DoSomething', 'THelperClass', [Set1]);
Set1 := [kfRead, kfAssignSource];
FChecker.VerifyProc('', FCodeHandler.StrucFuncsList, 'ToStr', 'TByteHelper', [Set1]);
UTWriteln('OK Test19_Helpers');
end;
FChecker in the code above is an object of a class TObjectChecker which contains utility methods for evaluating objects. UTWriteln is just a wrapper function that writes output text to the command-line.
The methods VerifyObject and VerifyProc will raise an exception if anything is wrong, and cause the unit test to fail. In this unit test, it is verified that references are set correctly for the involved identifiers.
Here is another example of one of our unit tests:
unit UnitTestsCode31;
interface
implementation
type
TAnimal = class
public
procedure M31; virtual;
end;
TMammal = class(TAnimal)
public
procedure M31; override;
end;
procedure TAnimal.M31;
begin
end;
procedure TMammal.M31;
begin
inherited M31;
end;
end.
As for many other unit tests, the code is artificial and not intended to be run. In the example code above, this is apparent, since no identifiers are declared in the interface section.
And the corresponding code for this unit test:
procedure TUnitTester.Test31_Inherited;
begin
FChecker.VerifyProc('', FCodeHandler.StrucProcsList, 'M31', 'TAnimal', [kfRead]);
FChecker.VerifyProc('', FCodeHandler.StrucProcsList, 'M31', 'TMammal', []);
UTWriteln('OK Test31_Inherited');
end;
This particular unit tests verifies that references to the M31 methods are set correctly.
To run a particular unit test, for example #567, just give the command “palunittests /U567” on the command-line:
Or, to run all unittests, the command is “palunittests /U”. The time needed to run over six hundred unit tests is about two minuters.
With our infrastructure, adding a new test just involves creating a new Delphi unit and writing the code that will be tested and the test itself in a method. Plus a few other minor code changes that are not described here. So to apply the plumbing for a new unit test is quick and easy. To write the actual test however can of course be trickier.
One benefit of implementing unit tests is that it encourages TDD, which stands for test driven development. In our case, now when we receive an error report, we normally create a unit test for it. We then run the new unit test. If it fails, the reported error is confirmed and should be fixed.
Then work continues in the code until the unit test passes. This means that also the original bug reported by the user has been fixed. But we in addition have to run all unit tests to make sure that nothing else has broken.
Some more considerations:
- If one or more unit tests are broken because of code changes, fix them as soon as possible or at least so you can test before releasing a new version.
- Try to run unit tests automatically, perhaps as part of your build process.
- Never ship a release if unit tests fail.
- Combine unit testing with code analysis, for example with our Pascal Analyzer or Pascal Expert tools, to catch problems and bugs.
To summarize, here is our recommendation:
Start using unit tests if you have not done so yet! The benefits outweigh the work needed to implement them. You will also get this wonderful feeling of knowing that your application behaves as expected.
Well, almost knowing... Your software will still have bugs, but hopefully a lot less of them!
If you want to read more about different types of testing, we recommend the Gurock Quality Hub.