Once, there was a time when testers had to manually test everything in Joomla, clicking through all the views and features each and every time a change was made. It was exhausting, cumbersome and prone to errors.

Along came Selenium! A tool to automatically control browsers, defining long click paths through all of Joomla, repeatable, reliable, not exhausting and requiring less people-power. And the testers saw all the tests that they made, and it was very good.

Testing Selenium

Except it wasn't. It turns out that human interaction with a browser contains hundreds of tiny actions which you don't notice normally. You scroll a bit, wait for a second here, and do other stuff like logging in there. Lots of unnoticed side effects which influence what the browser is actually doing. "Click on the save button and see that the entry shows up in the list view" sounds easy, but it requires you to wait for the reload of the page, wait for everything to be rendered properly, and then maybe scroll down a bit to find the entry you saved, set or reset filters, etc.

Why am I writing this? Because tests turned out to be pretty unstable, and unreliable. We run these tests in our CI system for each pull request and for each merge and most of the time that would result in false negatives occurring because of things like the installation of Joomla failing because the script didn't wait long enough at a certain point.

Fixing Selenium?

So we used a wrapper for Selenium in the form of Codeception. Did that make everything perfect? Actually, no. It didn't alleviate the issues much, but every little thing helps, right? Then why isn't every feature of Joomla completely covered by these automated tests by now? To those asking: Have you ever tried getting Selenium set up on your local machine?

Selenium might be a cool tool in theory, but from my perspective its principles, its basic design hasn't changed and adapted in the last 15+ years. I find it rather difficult to get it going... This is probably the main reason why our test coverage is rather poor. Even if people were willing to write tests and would be comfortable with the programming style necessary, most already fail in getting Selenium set up locally to be able to run their own tests.

Other software has similar issues

So everything is doom and gloom and we'll be testing by hand for all eternity?

Actually, no. You might have heard about a new web technology which came up recently, called Javascript. I know, who needs yet another programming language, right? Anyway, so this Javascript seems to be quite a big thing, right? And they seem to have had similar problems with testing as we have and so they didn't write just one replacement for Selenium, but several. One of them is Cypress.

What does Cypress do? Cypress allows you to write tests similar to Selenium and to run them on several different browsers. However, while Selenium is rather difficult to set up, Cypress is pretty much the opposite. Specifically, if you want to run these tests for the Joomla CMS and you have checked out the 4.3-dev branch already, you are basically ready to go. Cypress is automatically installed when running "npm install" and the only thing you need for your local environment is to modify the cypress.config.js file in the root folder.

How to use Cypress

But how do we actually run the tests now? Just call "npm run cypress:open" in a CLI in your Joomla folder and a short time later, the app will open. Here you can select to run the e2e tests and in there decide which browser to use. It then lists all the test suites available and you can click on the one you want to run right now.

"That sounds pretty much like how you would expect Selenium to work. What's so special about this?"

Have you seen a GUI for Selenium? Me neither. Selenium doesn't provide such a GUI out of the box. I can guarantee already, that this GUI is a huge simplification for us doing the testing. But that isn't the only thing that makes Cypress far superior to Selenium.

When selecting a test suite, the tests are then executed and you can watch the tests in the browser in real-time. During that execution, you see the script that is executed on the one side and the result in the browser on the other side, but more importantly: You can go back into the execution of the tests and take a look at snapshots during each execution step. These also aren't just screenshots, but real snapshots of the browser at that moment, allowing you to inspect the actual HTML at that moment in time. Screenshots and even videos from the tests are also possible.

Why Cypress is so good

Cypress also runs on Javascript/NodeJS natively, which allows for pretty good integration with all the events in a browser. And while Joomla is mainly a PHP project, the fact that tests are written in Javascript does make writing tests easier.

The handling of loading a page is generally also easier.

Where you have to do nasty hacks in Selenium to wait for onReady, Cypress handles this automatically for you. You basically feel that these people looked at Selenium and solved all the big issues you had with it in their product. And all of this is just the features out of the box. The plugin market is vast and has lots of other cool features, like partial screenshots and comparisons against known good snapshots.

What's left to do

Now that we have got this really cool tool, the rest will basically write itself, right? We hope so, even though of course it is going to be quite a bit of work. Most of the existing system tests have been converted to Cypress already and we hope to be able to gain enough experience with it to soon push out lots of new, good tests. Right now we split up the code similar to how we did it in Codeception with a block of tests and helper code in our tests folder and a support project called joomla-cypress, which provides some common behaviour. Hopefully, by the end of the year, we will have a sizable test suite, allowing us to go one step further.

Additional features of Cypress

Cypress has rather good browser add-ons to record click-throughs in a browser. That would allow us to ask people to provide us with such a recording to recreate issues. It will take the guesswork out when getting enough information to recreate the problem manually. At some point, this could even mean the requirement for automated tests for a new feature.

How it can help the third-party extension developers

Why am I writing all of this? Automated tests aren't just a fancy tool for software development in their ivory tower. Especially for smaller third-party extensions, automated tests are the solution for developers to stay on top of issues. It helps to keep their extensions working easily in newer Joomla versions. Joomla is always asking third-party developers to test their extensions against our current codebase, but that of course requires quite a bit of work if you were to do all of this manually. Recorded tests would allow them to easily repeat the work for every release catching bugs before their clients.

In the long term, we want to provide howtos and tutorials on how to get a full setup with continuous integration, code style checks, and automated testing. All against the latest Joomla codebase for extension developers. This would mean they can create a setup that regularly tests their extensions against the current Joomla codebase and if these tests fail, they get early warnings, allowing everyone to give feedback to the project or to adapt their own code to work with the new Joomla versions. In the long term, this hopefully means fewer issues for end users and more time for developers to focus on new or improved features instead of having to do bug fixes.

Until we reach that goal, there is still a lot of work to do, but I hope that you will all join us on the journey of automated tests. One day thanks to volunteers' hard work and Cypress, when all the tests are written we can enjoy a cocktail relaxing next to the cypresses on Cyprus. One can dream!