Help for Hack? on Forge/Digital Ocean

Hey there,

trying to checkout my website after the weekend I landed onto the 500 error page. Quickly looking at Forge's dashboard I realised that an error was appearing: No more space on device...

I thought that was strange as I didn't have an influx of users/data over the weekend. I checked on Digital Ocean the app's usage and yes, the disk usage had been slowly but consistently increasing since the 4th of April. Nice linear slope, until it reached 100% on the 9th. Strange again.

I logged in via ssh, df -h and then yes, the partition was 100% used. Little ls -la in the current folder, and I find myself facing a lot of small files of either 0b or 4343224 bites exactly. All with random names, and in the middle, the regular default folder and my app's name folder. As I couldn't even run a search I removed all these files (painstakingly) and manage to free up 300Mb. Ok now I could restart the server, MySQL and my app works again. But for how long?

I also notice in the df -h command, 6 /dev/loop0-6 small partitions. Are these supposed to be there in a Forge provisioned server?

Now my questions are:

How can I find out what happened?
How can I prevent it from happening again?
How can I find where is the rest of the disc usage? Because I was barely at 35% usage before, for the past 6 months.

Quick thing, my last push was in March 12th, so I don't think it's a coding issue.

Thanks for any help that can be provided.

CorvS

5 years ago

Level 27

What are your folder permissions? Especially of the folder you found all the files in. Did you check the auth.log (usually inside /var/log/) if someone got access to your server except you? Since you already mentioned that you "logged in via ssh", I'd assume that's the only way to get access to your server?

Romain

5 years ago

Level 30

Thanks for your reply,

looking at the logs, I see quite a lot of attempts with generic username: minecraft, user, sysadmin, testsite.... Besides root and forge users I also see a valid connection from a user smmsp (for 1 minute) is that normal? Quite a few login attempts from the same IP address on the same day. Im not sure how to detect a successful login by someone. They would have to use a proper user/password combination I assume, and that would reflect as a normal log entry (I guess?)

As for permissions they're the default ones when provisioning a server with Forge, I haven't changed anything. So the forge folder has: drwxr-xr-x => 755? 757?

And finally, yes SSH is the only entry point, no FTP. Basic Forge provisioning.

Please or to participate in this conversation.