Report from the Engine Room - What happens behind the scenes
Today I'd like to share with you what has happened in the last few months, actually years, behind the scenes of the Joomla project. I'm not talking about the politics or big decisions on the direction of the Content Management System (CMS), but about some of the work done to our infrastructure which is tucked away and nobody gets to see apart from special tours like today's article.
Several years ago Joomla started using Continuous Integration (CI) systems. These systems try to raise the quality of Joomla by running tests, checking the code style, and making sure that the code used works on all supported platforms. This happens with every change we make to the CMS, but the CMS isn't the only system we have.
While the repository of the CMS is our biggest asset, we actually have over 200 repositories, which make up the whole Joomla project. These repositories contain the different websites under the joomla.org domain, the different framework packages, our issue tracker, schemas for the XML and JSON we are using and lots more. Quite a lot of these repositories use a CI system, but as you can see from the sheer number, these weren't all set up by the same person and so we had at some point at least 6 different CI systems in use.
Joomla is driven by volunteers and sometimes volunteers leave the project. That is life, but that simple fact becomes problematic when the knowledge about a (CI) system leaves with them.
Three years ago we encountered a situation where this happened to the majority of the CI systems we had in use. In addition, one of the most widely used CI systems in our project, Travis, stopped providing support to open-source projects like ours. These 2 issues forced us to do the right thing and unify our CI systems to as few systems as possible.
Today we have managed to reduce this to just 2 systems, mainly Drone CI and in a few areas where we use appveyor to test our code on Windows machines.
For all those asking why we used Drone CI instead of GitHub Actions: We simply standardized on a system we already had in use and when we started that conversion, GitHub Actions was pretty new.
But I don't want to bore you with the sometimes repetitive work involved to migrate from Travis to Drone. Instead, I want to bore you with the last system we migrated from, Jenkins. Jenkins had a special situation, since it did some rather complex tasks, mainly deploying directly to our live websites, handling stuff like API documentation, schemas, Joomla nightly builds, etc. At the same time, the situation mentioned earlier about lost knowledge of the system also applied to this tool and so we had issues keeping it updated, fixing issues and supporting it.
Most importantly, however, we want to reduce the entry barrier for any new volunteer to help with the CI system(s) and the fewer systems you have to learn, the better. Jenkins had to go.
Step by Step
In the end, you have to do this one step at a time and so I started migrating task by task. Jenkins was mainly used to deploy changes to our websites, not so much in running our CI tests for the CMS or the Joomla framework. This meant that instead of just running some tests and waiting for success or failure at the end, we would be doing deployments and that means the code shouldn't accidentally delete everything when doing a deployment.
Some of these deployments were easy, like the one for our issue tracker or the website of the Joomla framework. They only require you to ssh into a server and kick off a built-in script which does most of the updating for you. So converting the Jenkins job to a Drone configuration was actually pretty easy. Some were slightly more complex, like the update.joomla.org site.
One would think that it should be easier because we are just copying static files, but the Jenkins setup has one rather big difference compared to Drone: Drone works from Docker containers and thus starts on a green field each time.
Jenkins is basically just another program running on the server on its permanent filesystem. So changes you did in an earlier run persisted to the next one and have to be cleaned up or at least accounted for.
Issues, issues, issues
While testing all these deployments, it showed that Jenkins wasn't properly maintained for quite some time already and with it quite a few of the repositories and assets under its rule. Many dependencies were updated and quite a few jobs and installations were faulty right now and either ran on an old version of its repository or didn't work at all. This brings me to the probably most interesting configuration of the whole transformation.
api.joomla.org is a website which displays the automatically generated API documentation. The tool phpdocumentor reads all docblocks of the source code of both Joomla and the Joomla framework and that is the first problem we encounter. phpdocumentor is stored as a separate executable on the Jenkins server and is 2 major versions behind. There is not really a version we can run on our Drone server, at least not with good conscience. The next problem is, that phpdocumentor in its newest version panics when reading the Joomla CMS docblocks because of a bug in its code and some invalid @since tags in the Joomla CMS codebase. If that weren't enough, even with those things fixed, phpdocumentor still didn't generate a complete rendering, since it failed on the class diagrams.
But while the issue with the @since tag could be solved by fixing our Joomla code and the class diagram was generated when using the docker image instead of the .phar release, this brought up the next issue: Our template was still designed for phpdocumentor 1.x and we would be using version 3.x. Yet another thing to fix.
Now we have phpdocumentor running properly the output looks okay, but it would have been easy if that had been the hard part of all of this. The next problem is, that we have not just one run of phpdocumentor, which we have to deploy, but several versions of the CMS and the framework, which documentation we have to merge into one website. Which also contains some static files. In the end, this resulted in three jobs in Drone, one to deploy the static files, one for deploying the CMS documentation and one for the framework documentation.
Jenkins has been retired since the end of May 2022. All those jobs and deployments have been migrated to Drone and each configuration is stored in the respective repository. From the moment we decided to retire Jenkins until this was actually done, it took several years and the actual work still required several man-weeks to get everything finished. The new setup with Drone means a contributor who wants to work on our CI system only needs to learn that one program and it also ties a lot better into our general permission handling. The end result looks rather promising and is a very good and solid foundation to now extend our automated tests and deployment from there. But that is the topic of the next article. I hope this little insight into the Joomla project engine room was interesting to you. I'd like to thank Harald Leithner, who did a lot of this work together with me.