Database getting too big for one server

Hi,

I have a database (MySQL) and especially 2 tables that are increasing in size rapidly (millions of rows per day).

They're taking a large part of the server space (Digital Ocean) which will soon reach its limit.

How would I scale this ? Not sure how to approach it in the best way...

Split the tables into several servers?

Thanks for any advice on this - if you've already encountered this.

Level 102

You could consider a database provider like planetscale or singlestore

I havent used either, but hear they are great. Fathom analytics (laravel based) is using singlestore https://usefathom.com/blog/worlds-fastest-analytics

2 likes

Level 73

As @sinnbeck said there are options explore with other server solutions, but before you do that, think upon this.

Questions:

How long do I need to keep the data?
Can I summarize the data in some way, and store that in another table, thus letting me remove records.

Possible solutions:

This is pretty easily solved, but is often forgotten, we create shitloads of data but we never remove any. So if you only need to keep a few weeks of data, run a nightly batch to remove everything older than those weeks.
If you are keeping track of some kind of transactions then you probably can summarize them into a single record for each day/week/month or whatever period you want.

2 likes

Level 13

Thank you for the advice and ideas !

I've already deleted the data and I can't possibly delete more. And summarizing is a great idea too.

Yes - was thinking about SingleStore. Let's see.

Please or to participate in this conversation.