Open Source Testing is one of the ways I give back to my current favorite open source CMS project. 
It’s not glorious, sexy or anything but it can definitely feel rewarding, like you’re helping make things move forward.

It’s also often frustrating and that’s OK to admit too.

The first thing that comes to mind with testing is that it has a pretty low barrier to entry: it’s open to anyone, even beginners - on the surface, as long as you can read and follow instructions you’re in, and is a great way to learn about a software project and how it works.

In my case, that’s a good thing; I’m definitely not what an open source project would want as a contributing developer. My knowledge of PHP is rudimentary and while I know that today’s AI coding assistants are powerful (very powerful even), without truly understanding the code that is generated, it creates more danger than not, and creates more work for the reviewers, the maintainers and the release managers than it helps, so I stay away from creating code for the project ;).

I could be wrong, but I think Open Source testing is often misunderstood.

 To some, it means following some precise testing instructions and saying “it works for me.”
To others, it’s a vague, thankless task that sits somewhere between development and support.

 In reality, I’ve learned, thanks to interactions I’ve had with quite a few fellow community members, that it is necessary, appreciated and impactful. I might even go as far as to say that it is one of the most impactful contributions anyone can make to a complex ecosystem like a CMS.

But what is a good Open Source Tester?

At first (i.e. and to be honest, for almost a year), I thought testing was about following testing instructions, confirming the BEFORE condition was true, and matching the AFTER condition(s) that are outlined in the Pull Request (PR). But over time, I learned that testing isn’t just about functionality, it’s also importantly about intent and opinion.

 Every pull request has the goal to solve a problem/bug, add, alter or improve a feature that already exists, but if you don’t understand the goal(s), you risk validating the wrong thing. As I have seen with my own eyes, a feature can “work” and still fail its purpose.

Testers in most projects are not limited to developers, they are often regular users who want to give back to open source software. Users who can have varying degrees of expertise - and that’s crucial. Being deep in the code of a project can create blinders or biases, and we need diversity of mindsets, diversity of testing environments, and very importantly diversity of testing datasets to have quality results.

(Parentheses open)

Let’s talk about these last 3 points for a moment:

1. diversity of mindsets,
2. diversity of testing environments, and
3. diversity of testing datasets

Diversity of Mindsets

Everyone will have a different journey into and throughout an open source project, This will shape what you know or how you interact with it. When reading a PR and testing it, you will (and should) have your own way to interpret it - sure the author of the PR will give you some context, and the issue if there is one will also give you some valuable information but how you go about proving and testing that PR also matters.

For my part, I try to always test what is asked but also turn on some options, features around what I am testing that may be directly or indirectly interact with (we can even say break) what the PR is having me test, and what I do will be different than the next person and that’s the idea…

Diversity of Testing Environments

In theory, software written for PHP 8.5 should behave the same way on all hosting environments right? 😂 … yes it should be so many factors affect how it will run that diversity of testing environments is crucial which is why most open source projects require at least 2 tests per PR beyond which additional tests will be done by maintainers and/or release managers before it becomes finally available in a package downstream. Remember, an open source CMS has tens of or hundreds of thousands of lines and how that code interacts with your environment can be quite different. And there are hundreds of thousands of slight variations in the way hosting companies run their platforms… The more varied our testing environments, the better the tests.

Diversity of Datasets

One thing I’ve seen many testers do in the beginning (myself included) is to have as clean an environment closest to the core product as possible and while that does have its purpose, it’s important to remember that a PR lives out there in websites that will have framework, templates, and extensions and a variety of datasets. Some websites will be very small and some will contain hundreds or thousands of pages, so great testing includes a variety of datasets.

 For myself, I always create my test environment off of the hosting root, with a multi-lingual setup, with 2-3 solid extensions and I change about a dozen settings in my CMS’s Global Configuration so that it’s not plain vanilla. Imagine how much more effective the testing becomes if every tester did their ‘own’ combinations!

Testing in ‘complex’ conditions reveals far more than a fresh setup ever will.

(Parentheses closed)

Good to great testing isn’t just about matching the BEFORE and AFTER condition, it is about making your opinion known. “YES I think this PR would be good for the community” or “I really don’t think this would be good for the project long term because… “

Especially in the beginning as a tester or even in time as being just a user (a non-coder), you don’t feel very legitimate sharing your opinion on a pull request - whether you think it’s good to be included or not - and yet, I have discovered that is a very important aspect of testing PRs. Users need to express their voice in an open source project as they are the ones living with the code that is added to the project.

As I’ve said before, testing open source software can be frustrating, remember: you’re volunteering your free time… and even with the best of intentions on everyone’s part, some of the following conditions happen:

1. Vague or incomplete testing instructions, or no instructions at all.

2. Tests that depend on very specific data or configurations that are not provided.

3. Lack of feedback after you’ve tested (your effort disappears into the ether).

4. PRs going stale after you’ve spent quite some time testing them.

5. Feeling unsure whether your opinion is “valid” or useful.

6. Re-testing the same PR many times after updates or requested changes.

7. Re-testing the same PR many times and finding out the PR does not get accepted by the maintainers or release manager.

8. The occasional blunt or unclear communication in the PRs discussion thread.

BUT, the light at the end of the tunnel is that the positives outweigh the frustrations.

Testing is about people and community. Open Source testing thrives on collaboration, and feedback is part of that. The most effective testers are not just precise - they’re constructive. They explain issues they encounter clearly, suggest improvements when they think they can help, and respect the effort behind every contribution.

And they aren’t afraid to express their opinion because in the community and in the real world is where the code, and the bugs live.

One last thought: with the advent of automation and AI, it might be easy to think that human testing is less needed than before - but that would be a grave assumption to make. While you now have easy access to automated testing and that’s great, it’s important to remember that both have their place. Automated tests are excellent for consistency and regression checks, they rarely capture the nuance of real user behavior. Manual human exploration remains essential. The human mind is incredibly clever and capable and will interpret, see and react to things in ways that automated testing will not come up with.

When testers share how they tested something, they enable others to validate, reproduce, and build on their work. This turns one tester’s effort into collective progress. Accessibility and usability also come into play. A feature that works technically but confuses users or excludes certain groups isn’t truly complete. Testing should reflect the diversity of real users of your community. 

In the end, Open Source testing is not just about finding bugs or matching a condition. It’s about improving quality, protecting users and the community, and strengthening the project as a whole. It may not always be visible work, but it sure is essential work.

And for those who choose to embrace its challenges, the experience is incredibly rewarding: you become part of a vibrant, living project shaped by the contributions of thousands, with your own work woven into its story for forever. Not a bad way to spend your time, wouldn’t you say? ;)