Testing code is an important part of software development, and automated testing should make sure that ongoing development doesn’t break existing functionality. Therefore, automated testing is very important in the quality management of our product Joomla!

The current testing situation for Joomla! 4 was not so good, because we had a lot open ends and some parts were not working together. So we met with the automated testing team in Cologne. Thanks to djumla GmbH, who offered us space in their office, we could keep the cost at a low level. We only worked on getting the testing for Joomla 4 ontrack, and especially worked on these specific areas:

• Testing Setup
• Unit Test
• Integration Tests
• System Tests
• Testing the new media manager
• Javascript Tests
• Static Code Analysis for finding security issues
• Documentation

Testing Setup

We have a mixed setup. One part is running on TravisCI and another part is using a self-hosted Drone server as Continuous Integration system. The goal here was to simplify the system. Now Drone is doing all the important testing, and so passing tests on Drone is required to merge a Pull Request.

We are still using Travis, but only for testing Joomla on the next PHP version. We do this because we are a bit lazy and Travis gives us the newest PHP without interaction on our side. This test is not required to pass for merging a Pull Request. Beside Drone and Travis, we also use AppVeyor for running tests on Windows.

Unit Test

Last year we started with splitting our tests. Our so-called "unit tests" were a mix of real unit tests and integration tests. The difference is that unit tests do not involve other parts of the application, like the database, in order to work. These tests run very fast and test just one single part of Joomla in isolation from the rest.

We created separate repositories, moved all tests out of the main repository and cleaned them up depending on the type. This work is finished for the unit test and we let the tests run on Drone and AppVeyor for PHP 7.0, 7.1 and 7.2.

Integration Tests

This part isn’t re-integrated yet. One of the main problems here is that we have to check every test – and the rather low test quality is challenging this task. In the past we had a requirement that people contributing code and new features had to write tests for it, and most of the time this task got finished at the very end and people hadn’t much experience with testing, resulting in a low quality.

System Tests / Testing the new media manager

We have been working for quite a long time on system testing for Joomla. We had multiple GSOC projects in the past years creating these tests, but unfortunately these tests never made it into the core.

At this sprint we were able to finally merge a couple of tests into the core. We are testing the installation, “com_content”, “com_category”, “com_users” and the new Media Manager. We have some more work to do here, but this is a good starting point.

Javascript Tests

Because of the very extensive changes in this area, we restarted with a minimum setup to have a starting point. Within the old tests a lot of jQuery was used, and in Joomla! 4 the team removed the dependency to jQuery, so this needs a complete rewrite.  

Static Code Analysis for finding security issues

The german company RIPS has developed a comprehensive tool that is able to discover all sorts of security issues by doing in-depth static code analysis through the code base. RIPS has already discovered multiple issues in the past using that tool and reported those to the security in a professional way, so the benefit of using their technology is quite obvious.

Therefore, we did the first steps to integrate the analysis tool into our continuous testing process. RIPS offers a command-line tool that streamlines that process, so getting the initial setup (push current code base to the tool, get result) running was very straightforward. Tobias and David from JSST reviewed the first set of results during the sprint to review the findings and mark false-positives.

The next step to take after the sprint is to further integrate RIPS into our CI workflow and especially adjust it to the fact that we are running various branches with different codebases (3.x, 4.x) that will require different false-positives lists. Once this has been sorted out, we are ready to go.

We would like to thank RIPS for their awesome support! Their help for our opensource project is highly appreciated.

Documentation

The least favourite part for every developer - but an important one.

Sadly we haven’t had the time to improve so much on this. But with the main problem “testing is failing constantly on Joomla 4” out of the way, we will do this over the next days. So when you read this article it should be done :-)

Final words

Our goal was to get the testing setup running again, and support people in their coding work instead of confusing them. All tests are now green, so people can rely on testing results. We also added stuff we tried to add for years. The atmosphere at the sprint was also very good and working has made fun.