13-18th December 2023 :: What's up with the website? I'm getting 404 errors and "JANGAN GANNGU !!!" messages...

[Jools]

Firstly, thank you to all the good fish folks who offered support and encouragement as Planet was offline for five days. You rock, and we salute you!

[Jools]

Around 2pm GMT on Wednesday 13th I began to get emails suggesting something abnormal was happening with the site. This was unusual because I had not been logged into the site or had been working on it for a day or so. I was at work and can’t readily get into the site so I waited until the evening.

On logging into the server I found all the site files were missing and a message had been uploaded basically telling me someone has access to the server that should not. I immediately changed all passwords including the database users which run on a different server. I removed the message and left one saying the site was down and I was looking into it. I began the process of restoring the previous night's backup.

On Thursday morning my message had been changed to “JANGAN GANNGU !!!". Indonesian for “don’t bother”. Face palm moment: while I had changed all passwords, I had not checked who was logged in. I immediately rebooted the server which would kill any other sessions running.

I had a look at who had logged in. There were two logins (167.99.4.184 United States 4 Dec-13 04:59:51 and 125.163.1.16 Indonesia 2 Dec-13 06:06:06) which were not me. Someone, possibly two people, had logged into the server. I could trust nothing.

I also found that aquaticrepublic.com, aquaticrepublicnetwork.com, clearriverpartnership.co.uk, dignall.com and zebrapleco.com had all suffered the same fate. Not good. I took all those sites out of service and began thinking about what to do. Thursday was a long day at work. Thursday night I started the process of checking everything. All files; everything.

Planet alone has around 93,000 files across about 53GB of disk. OK, a lot of those are standard forum files, but there’s still a lot of code! And I don’t know the attack vector.

Planet is backed up daily online and monthly offline. That’s all the files and the database too.

Friday I managed to get a good copy of the forum by checking every file against a clean copy from the phpBB site, I checked that all the database looked ok. I also checked all files against my local copy and against the most recent backup. By Saturday morning I had a clean version of the site. It took hours to upload everything.

On Sunday I couldn't get the forum to work. The forum software (phpBB) runs the forum but I also use it for all user authentication; it's well-maintained and secure but it's complicated and tightly integrated with the custom ARN/PlanetCatfish code. It's also modern - bear in mind I learned all this stuff 25 years ago. So I am fairly current but a lot of techniques have changed. So, I learned a few new tricks and set about fixing it all up again.

How did they get in? Possibly a brute force attack, possibly a file with the password in it was inadvertently copied somewhere. I don't know.

Did they steal anything? I don't know, it's possible they could have downloaded the site, and the database but there is no evidence of this and the database logs don't appear to show any activity. In case you're worried about your password, we don't store it. As per best practice, we only store a hash of it.

Today is Monday. I've finally got the site up and running. Due to when the last online backup was taken, images uploaded after around 2023-12-05 1100 are not available and need to be reloaded.

I think this only affects @casscats, and some additions I made to the cat-elog (and in some cases wrong images will be against the wrong contributor). Going to make a cup of tea and tackle that.

Cheers,

Jools

[Shane]

What a pain!
-Shane

[characinkid]

WOW thats alot of work, thank you for all the hard work to bring this back up. I was checking daily and not getting my daily dose of PC!! Great to see it back.

[naturalart]

Yeomans work. Glad you were able to put it back together. Congrates!

[bekateen]

Thanks Jools! I could tell if you took that long to fix it, it must have been a nightmare!

Cheers, Eric

[Jools]

No worries guys, certainly kept me focussed for a few days!

Cheers,

Jools

[CoryWally]

Great work Jools, so glad you were able to restore the site. Hacks and Hackers are horrible to have to deal with.

[amiidae]

happy its back online again

[DBam]

Thanks Jools for all your hard work. I didn't realise how much I'm on this site until I couldn't reach it anymore.

[Jools]

Thanks Jools for all your hard work. I didn't realise how much I'm on this site until I couldn't reach it anymore.

It's a pleasure (usually!), you know, a lot of people have said that about not realising how often they dip into the site. It's good to know.

Cheers,

Jools

[NewPlecoKeeper]

Thank you so much Jools, we in Bergen are gratefull for your work!

[Tony4143]

Holy crap, sorry you had to deal with his but thank you for getting it back up so fast considering all the work you had to do,
-Tony