No Silver Bullet - Fail Proof Systems

Written by | 01 June 2014 | Published in 2014 June
In software development process one can not guarantee the correctness of a program. Though, through effective software testing, higher quality softwares could be delivered with more satisfied users, lower maintenance cost, more accurate and reliable results.

If you are a toothpaste manufacturer, you wouldn’t need to worry whether it will work or not, but a software developer can’t feel confident about the working of the software (s)he developed. Like other developers, I can just hope that on the next day, it will not break, which barely happens.

Unless we start testing, there will always a probability of failure after deployment of the software. Where large softwares are developed & maintained simultaneously, automated testing and continuous integration comes to rescue us. In testing automation, unit testing is a technique which may reduce bugs in a system to zero, proving a bullet, silver or not.

What is Unit testing?

At a broader level, I can say that unit testing is process of testing certain functions, areas or units. This gives us the ability to make sure that our code/unit does what its coded to do. Units are tested using a test program to provide inputs to each component and check the output.

Testing is many things but most importantly it is not the process of demonstrating that errors are not present. It just only add a value to the software product. Adding value means raising the quality or reliability of the program.

To summarize, unit testing is the process of isolating a piece of code from rest of the system and finding errors in it, note it is not same as proving that there are no errors.

Is it worth doing?

Certainly writing tests to test code is an overhead, but it has various advantages that can’t be ignored easily.

  • Unit testing done early in the development phases, saves us from a lot of trouble. It provide early sanity checks.

  • Unit Tests allows you to make big changes to code quickly. You know it works now because you've run the tests, when you make the changes you need to make, you need to get the tests working again. This regression testing saves from future bugs.

  • Good unit tests can help document and define what something is supposed to do. One can quickly refer to a well written unit test to get a quick idea about how that piece of code works.

  • Finally, writing tests before writing code is another way to implement the functionality. When the requirements are clear but procedure to implement is not then Test Driven Development (TDD) is a great way to use unit tests.

So, to answer the question, Unit Testing is usually worth the effort, but the amount of effort we need to put it in, is a matter of choosing sides in a tradeoff.

How to do it?

Now specifically talking about testing php code, we're going to need a third-party library, like PHPUnit, that will actually execute the tests we write using its API. PHPUnit is a great tool at hand to write test. It can be integrated with your choice of Integrated Development Environment (IDE) like PhpStorm, Eclipse, NetBeans, etc. Though it is not necessary to use PHPUnit with an IDE always, you can use it in a Command Line Interface (CLI), but if you want to trace execution of your tests then you need an IDE.

Further text is all about PhpUnit in a CLI environment. Github also run full test suite on each commit or pull request using Travis-CI which is what Joomla is currently using.

Creating a test case

In Unit testing we can use both black box testing and white box testing techniques to create a test case. A test case contains specifically 4 parts, -preconditions, inputs, expected output, postconditions. After formulating a test case now all it needs to write in PHPUnit testcase.

The basic wireframe of a testcase for a method bar() of class Foo is

class FooTest extends PHPUnit_Framework_TestCase
    public function setUp(){
 //Setting up precondition common to all test cases
   public function testBar()
    //Setting up precondition specific to this test case
    // Assert statements goes here for example
    $foo = new Foo;
    $this->assertEquals($$inputs), $expectedOutput);
    //Setting up postcondition specific to this test case
   public function tearDown(){
 //Setting up post condition common to all test cases

Running a testcase

If PHPUnit is setted up in CLI env. then to run a test, you just only need to run command phpunit <path/to/testcase>

For example: If FooTest is present in current working directory then `phpunit Footest` will do the job.

Case of Joomla Framework

Currently in the framework we have in total 39 packages including Test Helper package. Each package have a test suite implemented using PHPUnit with them, with only exception being Compat package and Test Helper package.

To run tests present in a package, it is required to have a pre-setup environment for phpunit. For example, to run tests for Filesystem package, follow the following steps

  • Obtain a copy of git repository hosted on GitHub at either by cloning it or downloading and extracting a zip archive.
  • Change present directory to the directory where you just cloned/extracted package repo.
  • Initialize composer once to resolve all dependencies and install autoloader.
  • Execute all tests at once by using command `phpunit` in CLI

There are two packages, Form package & Crypt package whose tests are failing because of autoloading issue, which is going to be solved in next major version i.e. 2.0. Few others have low test coverage but others have more than 70% coverage.

Overall Joomla Framework is in good condition but we can make it great by adding some effort in improving test suite of some packages and fixing failing build packages.

In the GSoC project “Framework Unit Testing”, you can expect a genuine effort that I will put, in improving test suite of each package. I will just not only work on the packages with low code coverage but also review existing tests that we have now.

Yes, code coverage is an important parameter in weighing a test suite but it is not only the scale that quality or necessity of tests will be measured upon. My workflow will be like to take a package, study its existing tests, improve them if possible and add new tests for the public interface (i.e. testable code).

Read 8157 times Tagged under Google Summer of Code, English