Bugs & Fun

Good bug testing for software projects is about destruction not confirmation. You can easily follow the test instructions for a Pull Request (PR) and show that it fixes the problem it’s designed to fix. But what else might the new code be doing?

Do you know how to test code?

For many readers, it might not be something you’ve considered doing but it’s a great skill to pick up. Joomla! has a number of events at the moment ready to help you get onboard.

If you’re interested in learning how to get involved with testing Joomla, join the Joomla Bugs & Fun @Home event each weekend where the team can help you with learning how to use the Joomla Patch Tester and other tools Joomla uses to create documentation and create translations. 

On October 17, Pizza, Bugs and Fun 2020 is going to be running as a virtual event, where you’ll be able to get involved with users all over the world for a full day of testing, translating and documentation.

Appetite for destruction

The basics of testing a proposed PR for Joomla would be to find an open PR in GitHub, load/apply it via the patchtester, follow the developer’s description of what you need to do to test the patch, and confirm the PR passed testing when everything worked.😕 You’re partially there...

When you test this way, you basically only confirm what the developer was anticipating their PR would do when applied is what they expected it would do. It’s very likely that your test will work out and no error occurs, because the developer coded his/her improvement exactly the way that his test case works to fix the issue.

However, taking your patch testing skills up to beastmode requires an appetite for destruction. To make the best code, the testing should be strenuous.

Breaking things is OK

So what else should you do? The goal of each test should be to run scenarios to stress test and potentially break the new code, find errors, unwanted loopholes, and check whether there are any unwanted side effects.

The task is very easy to say but hard to execute: think outside the box. How can you break what’s on the screen to make sure it’s the best code it can be?

Firstly ask yourself, does this PR make sense at all? Does it improve Joomla’s code? Does it make the user experience better? Is the issue the PR tries to fix an issue at all?

If your questions result in the answer being no, don’t just blindly follow the developer’s instructions. Question them. Discuss the issue in the Pull Request. Interact with the developer and the Production Department via channels on RingCentral in the Bugs & Fun @Home team to create some discussion on the matter.

Next look at the code for the pull request. You’ll learn along the way what’s good and bad code, and whether changes have been made that might be done better with some strengthening.

As part of your testing, look at whether the code would fail by testing to break the process. i.e. try to make the test fail: 

• if numbers are expected, insert letters
• if integers are expected insert digits
• if an email is expected, insert an URL
• if you should set up a user with certain permissions, create a user with fewer permissions or more

Use your creativity to find ways to find potential bugs in what’s being put forward.

If it’s a user interface element think how you could interact with this element. The author tells you to click it? Then press [TAB] + [ENTER] or double click, triple click, touch or drag it. Check for accessibility properties: contrast, size, aria attributes, resize.

• Scale it up (if it’s a module, create two or more of this module at the same time)
• Provoke a conflict if it’s something that affects another component.
• Change the language constant, if there is any, via a language override.
• Overflowing values is also a good way to test: if an input field is limited to a certain size, put in longer words or disable the size restriction via the developer console and test then (in both cases the system should return an error message and not save the truncated value due to security reasons).

Sometimes the developer doesn't know the original core functionality, so, for example, there are sometimes different values saved (integer IDs vs. language code string (en-GB)) when the site is single lingual or multilingual. Check if all old values are still working.

Enable the debug mode and measure the speed of new database queries, what happens if you add thousands of users into your system or articles.

There are countless ways to try and break things. The CMS Release Teams and Bug Squad perform the above range of processes repeatedly throughout the year and are always looking for new people to join them to both help accelerate the process as well as distribute the load. 

Documentation, Documentation, Documentation

If you pass or fail your test of the PR, don’t forget to document what you did to break it. Write down all test cases you’ve run through because then the developer who’s made the PR and the bug testers looking at PRs before they are committed will get inspired by your tests and potentially find other edge cases which break the PR.

There’s a lot of common code throughout Joomla, and there are sometimes places where an edge case might impact multiple components all using the same object like start and finish publishing fields.

It’s also very helpful for unsuccessful tests to see where it breaks to find a fix.

Learning how to test

If there are many terms in this article which are new for you, that’s no problem at all. To find all possible breaking combinations to test you need experience and sometimes just a lot of luck. Other times it’s just repetition that hones your skills. So it’s good to start as soon as possible...

During COVID isolation Joomla Bugs & Fun @Home has been running every Saturday where friendly Joomlers from around the world have been making themselves available to show new users the bits and bytes about good testing. You get first-hand experience and can learn what’s all about by diving directly into the world of testing. Plus the friendly helpers are available on every day of the week, so you can join the conversation any time.

Community testing - Pizza, Bugs & Fun

What would bug testing be without doing it together? For the past few years, Joomla has had an annual “Pizza, Bugs & Fun event” where Joomlers from all over the world have met in local groups and worked collectively on writing code, testing pull requests, creating documentation and translating language strings.

This year Pizza Bugs and Fun moves online and on October 17th we invite users around the world to join us to work together to get Joomla! 4.0 release ready. Be part of it, every helping hand is needed and yes, everyone can help, regardless of whether you’re a beginner or an expert. You can get involved even if you’ve never looked at a line of PHP in your life!

Find more information at the Pizza, Bugs & Fun website or contact us in Ringcentral (Glip) in the @PBF20 channel. The event starts at 9 am your time zone on October, 17th 2020, don’t miss this date!

As part of this year’s event, we’re giving away a survival kit to help you get through the event. To qualify to receive a kit (100 available, running out soon) you just need to complete a few simple tasks:

• Create a Ringcentral (Glip) account and search for the PBF20 team and join the conversation.
• Create a GitHub account to be able to test PR at the event.
• Create an account for the official Joomla! documentation to be able to write documentation as well.
• Test two pull requests or create a pull request yourself to solve a problem in Joomla! 4.
• Write at least two texts (no matter if they are new or if you are improving existing ones) with at least 200 words in the Joomla! documentation.

When you’ve completed those tasks, jump over to this page and complete the survival package form to get on the list for a kit.*

The test was clear

By the way, if you did not manage to break the PR you tested above, then you should also test it the way the author wants you to… rumours are out there, not every developer tests their code!

Some Resources and Further Reading

Joomla’s has a wide range of ways to get involved in addition to the Pizza, Bugs and Fun event. You can find out more ways to get involved on the Joomla Volunteer Portal.

There’s plenty of resources on bug testing out on the web. Here are a few that might help you as a beginner, or give you some fresh perspectives as a seasoned bug tester.

• How To Fix Bugs Step by Step
• 7 Common Types of Software Testing
• How software bugs can make fun of you

*Note that due to global postage slowdown, Pizza Bugs and Fun survival kits can not be guaranteed to reach participants in time for PBF20 on October 17th.