At a quarter past midnight, just when I went to bed, I decided to check my email. I was away in the Ardennes (Malmedy, Belgium) with a few work friends.

One email caught my eye:

Date: Wed, 12 Oct 2022 19:57:03 +0000
From: Netcraft Takedown Service <This email address is being protected from spambots. You need JavaScript enabled to view it.>
Subject: Website defacement discovered at jug-arnhem-nijmegen.nl [Issue xyz]

Last July I launched a new Joomla 4 with my Joomla User Group: https://jug-arnhem-nijmegen.nl/ . And three months later this brand-new website, with just a few extensions and fully up to date, might have been hacked!

The sender, Netcraft, works together with SIDN, the manager of the .nl domain, to keep websites in the .nl zone safe. They found a hacker file “hxxps://jug-arnhem-nijmegen[.]nl/Ox.php” that was used to deface websites. They asked me to fix the issue and inform them. If not, the domain name should be taken down by SIDN.

There was another email about the hack. My VPS hosting had been informed by Netcraft as well. They encouraged me to take action as well, and if I didn’t, the whole VPS would be taken down.

They were right, my website was hacked…

I used FTP to check the webroot for a file called Ox.php. And I found two file that we had not added to the website ourselves:

Ox.php 12-10-2022 20:41:24 - a php defacement that displayed a “Hacked by” message
sec.php 12-10-2022 17:47:56 - a backdoor script that gives access to the whole filesystem and Joomla’s database

Oops…

I immediately remembered the story of a German friend whose Joomla 4 website had been hacked a few months earlier. In the Joomla community, we all thought it was strange because she is an experienced Joomla professional who takes security seriously. She told me she discovered the hack a few days too late to get the full log files.

Standard Operating Procedure (SOP)

Let’s get my “Website hacked SOP” for situations like this:

1. Neutralize the hack
2. Secure hacker trails
   • Website files
   • Website database
   • Website log files
3. Prevent future hacks
   • Analyze how website was hacked
4. Restore backup
5. Inform data subjects

Step 1 - Neutralize the hack

First of all, I needed to make sure that the hackers could no longer use their backdoor scripts on the server. On the server I moved the website to another folder and changed the permissions to “700”. That means that only the owner can access the files directly on the server via SSH. All files in the website were now unavailable for the outside world.

Now that the website was unavailable, I created an index.html file with “This website is temporarily unavailable due to maintenance” message for all visitors.

SSH / sFTP logins

Using SSH I checked the successful logins on the whole VPS:

# cat /var/log/auth.log | grep Accepted

Fortunately, it was only me! All logins in the last week were done by me via SSH and sFTP. Note: this method is not 100% safe: In case that hackers obtained file write access outside the web root, they would be able to modify that auth.log.

Newly added files

To see which files had been added in the last 2 days, I used

# find . -mtime -2 -print

• Ox.php 12-10-2022 20:41:24
  a php defacement that displayed a “Hacked by” message
• sec.php 12-10-2022 17:47:56
  a backdoor script that gives access to the whole filesystem and Joomla’s database
• configuration.php 12-10-2022 17:49:30
  contained a small modification at the end, a new line:  “public $session_metadata_for_guest = true;”
• administrator/templates/atum/error.php  12-10-2022 17:46:58
  another backdoor script to upload files
• administrator/index.php 12-10-2022 20:43:40
  the original index.php had been replaced with a "hacked by" screen, preventing any admin to log in into the Joomla back-end

Note: this method is not 100% safe because it’s possible to change the file date/times.

Step 2 - Secure hacker trails

To analyze the hack at some later time, I safeguarded the 

• files (including date/time)
• database and
• server log files.

I compressed the files/folders in an archive, took a mysqldump from the database that I archived and compressed all log files to another archive file. I downloaded all the archive files to my own computer.

An hour later than planned, I went to sleep. Tomorrow I will analyze the hack...

Step 3 - Prevent future hacks

Next day, analysis time. I created a new document to note down everything I find out about the hack. Then I started my analysis of the Joomla database.

Database

The #__users table had only 2 users. My account still had the same password but my last login was: 22-10-12 15:46:28. The last two records in the #__action_logs table showed PLG_ACTIONLOG_JOOMLA_USER_LOGGED_IN for my account:

• 2022-10-06 07:24:04
• 2022-10-12 15:46:28

On October 12th in the afternoon I had successfully logged in. However at that time I had been far away from my computer. My browser history showed the October 6th visit, but confirmed that I have not been on the website on October 12th.

Someone else used my account to login, without changing my password!

Access log

So the hacker got super admin access to my website at 22-10-12 15:46:28. They added a few PHP files and probably used the PHP backdoor scripts that they left in my site. I used those date/time and those filenames as my starting point for analyzing my access log.

I noticed that IP address 103.147.9.x was active during my login time. The first time that IP popped up in my logfiles was:

103.147.9.x - - [12/Oct/2022:15:44:52 +0000] "GET /?view=article&id=51&catid=15 HTTP/2.0" 200 18275 "https://www.google.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36" "-"

So the hacker found my website via Google.com. 

When I searched in Google for that article id=51 page, I found that Google has indexed my page with “J!+InfoRequestSessionProfileQueries101Log” on it, meaning with Joomla’s debug mode on.

Between the first http request and the login was little time. What stands out is that they probably used Joomla’s com_ajax in combination with Joomla’s debug plugin:

• 103.147.9.x - - [12/Oct/2022:15:45:16 "GET /administrator/index.php?option=com_ajax&plugin=debug&group=system&format=raw&action=openhandler
  &15310a67461c94ffb5e6e6ce67bf67c3=1&max=20&offset=0 HTTP/2.0" 200 906 "https://jug-arnhem-nijmegen.nl/administrator/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36" "-"
• 103.147.9.x - - [12/Oct/2022:15:45:21 "GET /administrator/index.php?option=com_ajax&plugin=debug&group=system&format=raw&action=openhandler
  &15310a67461c94ffb5e6e6ce67bf67c3=1&method=POST&max=20&offset=0 HTTP/2.0" 200
• 103.147.9.x - - [12/Oct/2022:15:45:23 "GET /administrator/index.php?option=com_ajax&plugin=debug&group=system&format=raw&action=openhandler
  &15310a67461c94ffb5e6e6ce67bf67c3=1&op=get&id=X9913f9017153a97bb42ad2b98e5ba232 HTTP/2.0" 200
• 103.147.9.x - - [12/Oct/2022:15:46:28 "POST /administrator/index.php HTTP/2.0" 303
• 103.147.9.x - - [12/Oct/2022:15:46:29 "GET /administrator/index.php HTTP/2.0" 200

Then the hacker replaced the error.php page of Joomla’s Atum back-end template with a backdoor file:

• 103.147.9.x - - [12/Oct/2022:15:47:56 "POST /administrator/templates/atum/error.php HTTP/2.0" 200

After that, they unsuccessfully tried to use their backdoor script to get outside of the public_html folder and gain more access on the server.

Second hacker

The hacker probably shared this hack on some facebook group because somewhat later a second uninvited person appeared in the access log files. Some of their actions:

• 185.195.236.x - - [12/Oct/2022:18:39:03 +0000] "GET /?fbclid=IwAR3mgGPsxDHwH43s-bXrYRE3V-WAoNQGFUYtfa-EFU0cwElKJLCeh4tTio4 HTTP/2.0" 200 5539 "https://m.facebook.com/" "Mozilla/5.0 
• 185.195.236.x - - [12/Oct/2022:18:40:36 "GET /sec.php?dir=/www&aksi=view&dirf=/www/configuration.php HTTP/2.0" 200 4706 "https://jug-arnhem-nijmegen.nl/sec.php"

So the second hacker retrieved Joomla’s configuration.php with database and SMTP credentials.

Because it seemed that the hacker used core Joomla parts to gain unauthorized access to my site, I contacted David from the Joomla Security Strike Team (JSST). I gave him all my documentation about the hack and ssh access to the server.

A few hours later David informed me that they indeed found a security issue with Joomla’s debug mode and that they were working on a security patch. And I could remove the hacked site and restore my clean site.

Step 4 - Restore backup

Before restoring the backup, I changed ALL credentials:

• SSH/FTP access
• Database name, database username and password (they copied configuration.php!)
• SMTP credentials (they copied configuration.php!)
• The passwords of all Joomla User Accounts (only two fortunately)
• Backup server credentials in Akeeba backup

After that I restored the most recent backup from before the hack. 

Step 5 - Inform data subjects

When you have indications that personal data has been compromised, you also need to inform the local privacy authorities. And maybe you might have to report it to the local police.

In this case there were no indications that personal data (names + email addresses for the newsletter) had been stolen. I decided only to inform all the people subscribed to the mailing list. And I also informed Netcraft and my VPS provider that I fixed the issue.

Why did it happen?

Our JUG Arnhem-Nijmegen organized a Joomla User Group session for Tuesday 4 October. However my co-organizer was sick. And besides the presenter only one other person signed up for the event. We decided to cancel the meeting. 

Issue with sending newsletters

When sending the “sorry the JUG session has been canceled” newsletter via Acymailing, a problem occurred. Half of the mail queue was sent successfully. The other half gave “SMTP authentication failed” errors.

I could not find the cause of the issue. So I decided to switch on Joomla’s Debug mode because it gives helpful info in case of errors. This time it did not help me find the issue.

After making myself a cup of tea, I found the culprit: The DNS of jug-arnhem-nijmegen.nl had one MX record for mail.jug-arnhem-nijmegen.nl. But two different mail.jug-arnhem-nijmegen.nl records, both with different IPs. The correct IP address was the one used when the sending was successful. The faulty IP address caused the SMTP authentication failed errors. 

I finally was able to send all the newsletters, post some other messages online about the canceled meeting and….. totally forgot about switching off the debug mode. And Google indexed my “JUG session is canceled” article on my site and showed it with the debug details on the Google result page.

Note to self: 

• When using Joomla’s debug plugin on a live site, set its Access level to Super Admin!
• When finished debugging, switch OFF the debug mode. 

In the end

I’m happy that I found out about the hack so soon after it had happened, and that I was able to secure the hacker trails. My German friend confirmed to me that her hacked website also was in debug mode. Knowing that it was hacked via the same vulnerability gives something less to worry. Also although the vulnerability had a critical impact, the severity and probability was low: you had to have the debug mode enabled.

The Joomla Security Strike Team was able to fixed this security issue (thanks again!) and they made Joomla, with debug mode on, safer. On Tuesday 25 October 2022 Joomla 4.2.4 Security Release was made available which fixed this issue.

The cancelled October JUG Arnhem-Nijmegen meeting was rescheduled for November. At that Wednesday 2 November meeting, we had many visitors. And besides the interesting redesigned J4 migration presentation, there was a short presentation on "How the JUG website was hacked".