Be part of JetBrains PHPverse 2026 on June 9 – a free online event bringing PHP devs worldwide together.

ElMorin's avatar

Work with a huge amount of data MySQL

Hello there,

First of all, I started programming last September and this is my first post there. Another thing, english is my second language, so sorry for mispelling...

Now that it's said, start talking about the subject. I was looking for the best way to use Eloquent and work with huge amount of data. My goal was to keep it fast. So, I started reading about how MySQL work and how to optimize it. Some say that it's better to have smaller tables and try to make less possible jobs by MySQL. If you have the choice, choose the make more work as possible by the app.

So there come Eloquent. All I read about this, is when you want to associate row from table A with rows from table B, use a pivot table. But, with my small knowledge, problem comes when you have a huge amount of data.

Take this example with 3 tables (users --- clients_users --- clients):

$client = Client::find(1);
$client -> dosomething;
$client -> save();

The app ask to MySQL to search for the client with the id=1 in clients table. What MySQL will do is to first search in the pivot table if the client is associate with the user, if it's true, go into clients table and get the row.

MySql did 2 process, if I'm right.

By the way, this is what peoples said me to do... But I was afraid that more my table clients is big, slower it'll be. There is my reasons:

I got user that belong to a dealer. This users can add clients and those clients belong to user and it dealer. So, if I got 100 dealers and each one got 10 users and each users add 100 clients per year, after one year the clients table will contain 100 000 rows. After 10 years it'll contain 1 000 000 rows. I don't really want this to become really slow...

There comes my issue. It work, but I want to know if it will really speed up the process. I'll make it short, i want to know what you think. If you want me to show you the code, ask for it, it'll be my pleasure.

First of all, I created some model: User, Client, Dealer and more.... but we don't need to know all of them to figure it. I also created the migration using the appropriate artisan command. Then, I create a bat file (I'm using xampp on my local environment) that contain some command that copy the migration clients and rename it d0clients. When I create a dealer, I also modify the .bat to become d.dealer_id.clients then launch a php artisan migrate command. After this, it undo the modification to the .bat file to be d0clients again.

At this point, it's fine each time I create a dealer, I got a new clients table for each dealer added. Now, every user will add their clients to their dealer's client table.

All of this needed some modification in the model, but, finally I can use Eloquent using one Model for each table type. By example:

$client = new Client;

Can be use to add client to all d*clients tables. The app find the right table to use then said to MySQL to add the client in the right table. So, MySQL do one process to add client and do one process to find them and update them.

Is this a "good" way to make it and keep the app fast or not? Do you need more examples?

0 likes
8 replies
jordan's avatar

"Oracle MySQL" and "fast" really don't fit in the same sentence. You'll be better off looking into better relational database alternatives, and trust me, there's better alternatives. Facebook are looking for an alternative and Google no longer uses Oracle MySQL.

I got user that belong to a dealer. This users can add clients and those clients belong to user and it dealer. So, if I got 100 dealers and each one got 10 users and each users add 100 clients per year, after one year the clients table will contain 100 000 rows. After 10 years it'll contain 1 000 000 rows. I don't really want this to become really slow...

1,000,000 rows is really nothing in terms of data. You don't have to worry. We deal with hundreds of millions of records in our data scraping database at work and it takes seconds to query the database. You have no need to worry.

Thanks, Jordan.

1 like
ElMorin's avatar

Thank you for the quick reply. Can you show me the direction? I can easily find an alternative, but do you have something in mind that I can read and learn to know what I can do to have better relational database.

The example was just to show how it can grow quickly. The other thing is that I don't use a powerful server like Google and Facebook do. But I still want something fast.

jordan's avatar

I sure can! If you're looking for a drop-in replacement for MySQL since some people don't like change, I would recommend MariaDB. If you would like change I would really recommend PostgreSQL. If you don't need a relational database, you can use a NoSQL server such as MongoDB.

Cheers, Jordan.

1 like
chrisgo's avatar

@ElMorin If you are really just a new developer on a new project, then do not worry about the number of rows in MySQL (or MariaDB) -- 1,000,000 records x 100 tables is nothing.

What you want to avoid is n+1 queries which is very easy to stumble into with ORM (Eloquent). If you do one query and return 1,000 records with the proper joins (read up on eager loading), that will be quick even if you are going through 10 other tables. If you return 1,000 records without the proper joins (lazy loading), then you will basically run 1 + 1,000 queries (total of 1,001) which will definitely kill your server.

1 like
michaeldyrynda's avatar
Level 41

If you're at the point with your application that a million rows is slowing you down, it's probably not the underlying database server that will be the source of your trouble. A lot of the time, slow queries come down to inefficient table design and in particular poor indexes.

I know @jordan mentions that Facebook are looking for alternatives and Google no longer use MySQL, but you have to remember these guys are dealing with orders of magnitude more data than most of us will be dealing with, and at that point, you'll have the staff behind you to more or less build your own database solution.

Now taking your example, @ElMorin, you'll have three tables: clients, users, dealers, with their basic structure as follows:

dealers - id, name
users - id, dealer_id, name, email, password
clients - id, user_id, name

From this, we can build three models and start to derive relationships:

namespace App;

use Illuminate\Database\Eloquent\Model;

// app/Dealer.php

class Dealer extends Model
{
    protected $fillable = [ 'name', ];

    public function users()
    {
        return $this->hasMany(User::class);
    }
}

// app/User.php

class User
{
    public function dealer()
    {
        return $this->belongsTo(Dealer::class);
    }

    public function clients()
    {
        return $this->hasMany(Client::class);
    }
}

// app/Class.php

class Client
{
    protected $fillable = [ 'name', ];
    
    public function user()
    {
        return $this->belongsTo(User::class);
    }
}

With the relationships you've established, you don't need a pivot table as I understand it. Your relationships are all one to many at this stage. The pivot table will come into it if a user belongs to many dealers or a client belongs to many users, but I don't see this need at the moment.

Now, if you needed to find any of these specific records, you can use the Eloquent methods:

$dealer = App\Dealer::find(1);  // SELECT * FROM dealers WHERE dealers.id = 1 LIMIT 1;
$user   = App\User::find(5);    // SELECT * FROM users WHERE users.id = 5 LIMIT 1;
$client = App\Client::find(3);  // SELECT * FROM clients WHERE clients.id = 3 LIMIT 1;

As you can see, any of these is only running a single query - it doesn't (and wouldn't, even with the pivot table) need to hit any other tables to get a single record.

If you wanted to load all users for a dealer when you load the dealer, you could do something like the following:

$dealer = App\Dealer::with('users')->find(1);

// Executes two queries:
// 1. To fetch the dealer record - SELECT * FROM dealers WHERE id = 1 LIMIT 1;
// 2. To fetch the associated users - SELECT * FROM users WHERE users.dealer_id IN ( 1 );

Now, if you needed to get all users and clients in one go, you could just add the extra relationship to your Eloquent model:

$dealer = App\Dealer::with('users.clients')->find(1);

// A third query is run - SELECT * FROM clients WHERE clients.user_id IN ( $ids_from_clients_result )

All up, you're executing three queries to get all of your related data for any given dealer.

Alternatively, you can define a has many through relationship on the dealer for your clients:

class Dealer
{
    public function clients()
    {
        return $this->hasManyThrough(Client::class, User::class);
    }
}

$dealer = App\Dealer::with('clients')->find(1);

// 1. SELECT * FROM dealers WHERE dealers.id = 1 LIMIT 1
// 2. SELECT clients.*, users.dealer_id FROM clients INNER JOIN users on users.id = clients.user_id WHERE users.dealer_id in ( 1 )

What this will let you do is grab all clients that belong to the top-level dealer, without having to hit the intermediary (users) table, and cuts the number of queries down to two. If you wanted to also get the owning user for each client, you can amend that to:

$dealer = App\Dealer::with('clients.user')->find(1);

Doing so will add the extra eager load as with the original App\Dealer::with('users')->find(1) query.

$dealer->clients->each(function ($client) {
    sprintf('%s | %s | %s%s', $client->user->dealer->name, $client->user->name, $client->name, PHP_EOL));
});
Quitzon Ltd | Kelton Brown | Aurore Ledner
Quitzon Ltd | Kelton Brown | Tressie Beier
Quitzon Ltd | Kelton Brown | Maci Boyer
...

I've committed this sample stuff to GitHub if you want to look into it a bit more, there's some model factories that will build up your database easily.

Hope this helps :)

3 likes
jlrdw's avatar

A billion rows shouldn't bog down mysql if the queries are written efficiently. Don't query 10000 rows at a time but only what you need and of course use pagination.

ElMorin's avatar

@deringer Thanks a lot. After you explained this to me, I think I understand, well... a bit more then yesterday... How all this work. Maybe I should ask it in one other question, but since you already did an example I'll ask it there. I did all the relation to my current app and added some new table. Now that I get Dealer - User - Client - Transaction, when the user login, I want to display all his transaction.

I do this :

$userclientstransactions = User::with('transactions')->find(Auth::user()->id);

And I return this to the view. This User::with('transactions') is this:

public function transactions()
{
    return $this->hasManyThrough(Transaction::class, Client::class);
}

The final result mus be:

client->first_name+' '+client->last_name | transaction->date | transaction->price

This because i want to display every transaction the user is working on starting with the most recent. But Client hasMany Transaction and Transaction belongsTo Client.

For now, i try to understand how to show each row like the final result, but I'm at this point:

foreach ($userclientstransactions->transactions as $transaction) {
    echo '<p>';
    var_dump($transaction->transactions);
     var_dump($transaction->price);
    echo '</p>';
 }

I'll continue searching on that, but if someone have some quick tips, it'll be appreciate!

-----EDIT----- I already done this in the past doing this:

$id = Auth::id();
$transactions = DB::table('transactions')->orderBy('updated_at','desc')->get();
$rowcolor = 0;
foreach ($transactions as $transaction) {
    if ( $rowcolor == 0 ) { $rowcolor = 1 ;} else { $rowcolor = 0 ;}
    if (($transaction->user_id)==$id) {

    $client = DB::table('clients')->where('id', $transaction->client_id)->first();

    $fulldate = ($transaction->updated_at);
    $date = explode("-", $fulldate);
    $day = explode(" ", $date[2]);
    if ($date[1]==01) {$month = 'JAN';}
    if ($date[1]==02) {$month = 'FEV';}
    if ($date[1]==03) {$month = 'MAR';}
    if ($date[1]==04) {$month = 'AVR';}
    if ($date[1]==05) {$month = 'MAI';}
    if ($date[1]==06) {$month = 'JUN';}
    if ($date[1]==07) {$month = 'JUL';}
    if ($date[1]==08) {$month = 'AOU';}
    if ($date[1]==09) {$month = 'SEP';}
    if ($date[1]==10) {$month = 'OCT';}
    if ($date[1]==11) {$month = 'NOV';}
    if ($date[1]==12) {$month = 'DEC';}
    $realdate = $day[0].'-'.$month.'-'.$date[0];

    echo   '<div class="'.$rowType.'"</div>
         <div id="" class="singleClientRow'.$rowcolor.'">
        <div class="inline icon-'.$iconType.' fs32 fc'.$color.'"></div>
        <div class="inline clientRowSplitter"></div>
        <div id="transDate'.$transaction->id.'" class="inline clientRowText">'.$realdate.'</div>
        <div class="inline clientRowSplitter"></div>
        <div class="inline clientRowText width200">'.strtoupper(($client->first_name)).'&nbsp;'.strtoupper(($client->last_name)).'</div>
         <div class="inline clientRowSplitter"></div>
         <div class="inline transactionButton" onClick="showTransaction('.$transaction->id.')">
        <div id="fixeIcon'.$transaction->id.'" class="inline show">
        <div class="icon-mail fs24"></div>
        </div>
        <div class="inline clientRowText">TRANSACTION</div>
        </div>
        <div class="inline clientRowSplitter"></div>
        </div>
        </div>';

But now I'm looking to do it using Eloquent and Laravel convention.

michaeldyrynda's avatar

You'll want to define the transactions relationship on your Client model.

class Client extends Model
{
    public function transactions()
    {
        return $this->hasMany(Transaction::class);
    }
}

You can then load all the transactions for the logged in user's clients:

$user = auth()->user()->clients()->with('transactions')->get();

Alternatively, you can put the transactions relationship directly on the user:

class User extends Model
{
    public function transactions()
    {
        return $this->hasManyThrough(Transaction::class, Client::class);
    }
}

And access the transactions directly:

$transactions = auth()->user()->transactions;

Choose the method that makes most sense, and depending on the context in which you'll be accessing the data.

In either event, once you have the transactions, you can iterate through them to display what you need:

foreach ($transactions as $transaction) {
    printf(
        '%s %s | %s | %s<br />',
        $transaction->client->first_name,
        $transaction->client->last_name,
        $transaction->date,
        $transaction->price
    );
}

Also, updated_at will be a Carbon instance by default. You can handle formatting of the date output much more elegantly:

$transaction->updated_at->format('d-M-Y');

It'll even handle localisation for you.

Please or to participate in this conversation.