Thursday, August 09, 2012

Unit tests are not all black and white

And they are not all red or green either. From a management perspective it might make sense to divide unit tests into the ones that pass, and mark everything else as an error. After all, this is what is confirmed to work and what is not.

From a development's perspective this approach makes less sense. The main goal is still to get all unit tests in the green zone. It's just that some of the test that do not pass are not errors that need immediate attention. Some of the error may already be planned to be fixed in the future, configuration problems or dependencies on resources that are not available at the moment of the test run.

Take for example a test that uses a database (i known that technical it is not a unit test). You want to test that some updates are done correctly. If for some reason the database is down and you can't get an open connection to the database, would you fail the test? My suggestion is that you don't. The test is not about getting a connection to the database but about doing the update.

They way I like to use unit tests is to utilize the ignore or inconclusive state of tests. If it is some configuration,  some 3. party component missing on the test machine or some error that is scheduled in the future I like to take them out of the errors in need of immediate attention.

I have seen panicking because hundreds of tests fail just because the password of the database user had changed. This should maybe fail in a single test that verifies that the database is accessible, but not the whole test suite.

It's a good idea to prioritize your tasks, and by using levels of severity in your unit tests you can optimize your bug fixing efforts a bit and focus on the errors that are real.

No comments:

Post a Comment