The Problem: Testing software has been a huge challenge as long as software has been around. A big part of this is the simple mathematics of if / then statements.
To over simplify a bit, every if statement creates two possible execution paths through a program. The number of possible paths is roughly 2 raised to the nth power (2n), where n is the number of if statements. Joomla has over 12,000 if statements. This means that there is a ridiculous number of possible combinations that we could theoretically test for Joomla – more than anyone would have time for in 100 lifetimes. And this ignores the fact that Joomla runs in a wide variety of operating environments (Linux, Windows, Apache, IIS, MySQL, SQL Server, and so on). Suffice it to say that testing Joomla is a big job.
Before you give up in despair, things are not quite as bleak as they might seem. For example, by writing Joomla using object-oriented design principals, we reduce the interdependence between different parts of the package. From a testing perspective, this means that we can test many parts of the program independently of other parts. This reduces the number of possible test cases dramatically. Even so, we still have far more theoretical test cases then we can ever hope to test.
Given the impossibility of testing every possible execution path and environment, how do we test Joomla to make it as bug free as possible? The answer is that we use a number of different testing strategies. Taken together, these provide a high degree of confidence that the Joomla releases we ship will work as advertised and a mechanism for continuous improvement.
Automated Unit Testing
Unit testing is the most scientific form of testing. Here we write test programs that test specific lines of code in the package. Unit testing in Joomla is used mostly for the platform, where each program does a very specific low-level job. Automated unit tests are run frequently and report immediately if a programming change causes a test to fail. When this happens, the change is undone until the problem is resolved. This greatly reduces the chance that a change will cause new bugs. Also, when new programs are added to the platform, they generally require that automated tests for these programs are added at the same time.
With unit testing, we can actually measure the exact number of lines of code that are being tested. As of this writing, there are automated unit tests that test about 40% of all of the code in the platform. Volunteer coders are busy writing more tests all of the time, and this number is expected to continue to move up. Unit testing is an important component of the Joomla testing strategy, and the more code we can test this way, the fewer bugs we will have.
If it ever reached 100%, would that mean we would have no more bugs? As you might guess, the answer is no. Even if we test all of the code in the platform, there is always room for human error in the tests. Also, even if the platform programs work exactly as designed, programs sometimes have design flaws that cause bugs. Plus, the platform programs might be used incorrectly somewhere else in Joomla. Nevertheless, unit testing dramatically reduces the likelihood of bugs.
Automated System Testing
The other type of testing we use in Joomla is automated system testing. This is very easy to understand. Imaging writing a detailed test instruction book like this:
- Go to Article Manager and click New.
- Enter in a new article called "Test" and assign it to the Featured (home) page.
- Go to the front end home page and make sure the article is there.
System tests are software programs that follow a test script exactly like this. Instead of a person typing and clicking, system tests send simulated key strokes and mouse clicks to the browser to work through different usage scenarios. Then we can automatically check the page to make sure the results are what we expect.
At the present time, we have a set of 57 system tests that take about 40 minutes to run and check about 3400 conditions. This is the equivalent of a person testing for several hours. System tests are run automatically every night and tell us right away if a recent change has caused a problem. Since we started using system tests, they have proven to be very helpful in catching numerous bugs before they were released.
As currently used, system tests in Joomla do not try to test every usage scenario or every core extension. They are more of a "reality check" to make sure that the basic system functions work as expected. We would love to have more system tests to test more of the core functionality and in more detail. This is an area where we could use help, and you don't have to be a super-experienced programmer to write system tests.
Automated unit and system testing help us prevent bugs before they get released and help us make sure bugs stay fixed. However, as we all know, the most important test of any software package is when people actually use it. This is especially true of a program like Joomla that is used in literally millions of websites, often in ways that the program designers never imagined. Bugs reported by users are absolutely essential to maintaining and improving the quality of Joomla.
When someone reports a bug in Joomla, we create an issue in the Issue Tracker. Before a proposed bug fix is applied to the actual code base, we normally have two people (other than the person who coded the bug fix) test that the proposed fix works correctly and doesn't cause any new bugs. The exact way this is done depends on the nature of the bug. In some cases, where the problem is complicated and could have more potential to cause problems somewhere else, we might require more testing be done before making the change.
It is important to understand that this process only works if people in the community take the time to report bugs. Reporting bugs is one of the simplest yet most important things that Joomla users can do to help the project. If a bug doesn't get reported, it almost certainly won't get fixed.
When we are getting ready to release a new Joomla version, we do some special release testing. We test that the download packages and automatic updates work. We also run the system tests to make sure that something didn't go wrong in building the packages. This testing is normally done by Bug Squad volunteers.
In a perfect world, updates to Joomla would never cause problems with third-party extensions. However, any time we make a change to the Joomla core there is always a possibility that the change will cause a problem with an extension. Ideally, extension developers should periodically check that their extensions run correctly on the current code base. That way, if a change causes a problem, it could be reported and fixed prior to a release. If this is not feasible, a second option is to test extensions using the proposed release packages that are created a few days prior to each release.
This is an area where we are actively working to improve our testing. A new group is being formed specifically to test proposed releases with third-party extensions.
How Can You Help?
Joomla is a community based project. All coding and testing is done by volunteers. There are many ways you can help the Joomla get even better. Here are a few:
- If you find a bug, make sure to report it. If you aren't sure, post a question in the support forum.
- If you want to help testing bug fixes or releases in the Bug Squad, send me an email. Remember, you do not need to know programming to help in the Bug Squad.
As we have seen, testing Joomla is a large job and we use a number of different test strategies. The many volunteers who help with testing make it possible to maintain and improve the quality of Joomla. I encourage you to join in the effort.