Be part of JetBrains PHPverse 2026 on June 9 – a free online event bringing PHP devs worldwide together.

Drfraker's avatar

How to speed up this sql query

In my application ( which is multitenant) I have a query that can take up to 10 seconds depending on the number of rows in the database for a given office. I'd like to speed this up but I'm not sure how to do it. The table for this query "notes" is quite large and has around 800,000 rows in it. Additionally, the note_text on the notes table can be quite large.

Here's the controller, I believe the code that is slow is in the getFilteredNotes($request) method, and possibly related to getting the count. But I'm not sure. It is difficult to test because in development environment I don't have nearly as much data in the notes table.

My questions:

  1. Is there a clear problem with the query that is making it slow?
  2. Is there a way to measure this query for debugging without replicating production data? (Due to privacy regulation, I really don't want production data on my machine)
  3. Could removing the note_text field have an effect on the time of the query to complete?
<?php

namespace App\Http\Controllers\Api;


class NotesController extends Controller
{
    private $note;

    public function __construct(NoteRepositoryInterface $note)
    {
        $this->note = $note;
    }

    public function index(Request $request)
    {
        $paginationPage = $request->page ?? 1;
        $take = 25;

        list($query, $totalCount) = $this->getFilteredNotes($request);

        $notes = $this->getPaginationData($query, $take, $paginationPage);

        $clientIds = $notes->pluck('client_unique_id')->unique()->reject(function ($id) {
            return ! $id;
        });

        $clients = app()->make(ClientRepositoryInterface::class)->getClients($clientIds);

        $notes->map(function ($note) use ($clients) {
            $note->setRelation('client', $clients->where('api_unique_id', $note->client_unique_id)->first());
        });

        return new LengthAwarePaginator($notes, $totalCount, $take, $paginationPage);
    }
  

    /**
     * Get the notes for the view and filter by search text and user.
     *
     * @param Request $request
     * @return array
     */
    private function getFilteredNotes($request)
    {
        $filter = $request->filter;

        $query = Note::with('staff')
            ->select([
                'id',
                'client_unique_id',
                'user_id',
                'created_at',
                'deleted_at',
                'service_date',
                'note_text',
                ])
            ->when($filter, function ($q) use ($filter) {
                return $q->clientName($filter)
                    ->staffName($filter)
                    ->clientId($filter);
            })
            ->when(! $request->user()->isOwner(), function ($q) use ($request) {
                return $q->ownedByUser($request->user());
            });

        return [$query, $query->count()];
    }

    /**
     * Get the paginated results for the view.
     *
     * @param $query
     * @param $take
     * @param $paginationPage
     * @return mixed
     */
    private function getPaginationData($query, $take, $paginationPage)
    {
        return $query->orderBy('service_date', 'desc')
            ->take($take)
            ->skip(($paginationPage * $take) - $take)
            ->get();
    }
}

0 likes
5 replies
click's avatar

Do you have Laravel Debugbar enabled? If not, add it to your project. It is helpful to see which queries are executed and how long they take. This makes pin pointing the problem much easier.

https://github.com/barryvdh/laravel-debugbar

There could be a few things "wrong"

  1. One thing that could be slowing things down is doing the extra query to count the total records. Do you really need a full paginator or could you also use a simple prev/next ? If so you could use the SimplePaginate instead of the LengtAwarePaginator https://medium.com/@asked_io/optimizing-laravel-5-mysql-for-9m-rows-step-1-56285bad2d65
  2. I see you order on service_date do you have an index on that field?
  3. Do you have indexes on the database fields you use to filter?
  4. How does the filter on staff name and client name look like? If it is a LIKE %search% comparison it could slow things down because it needs to run through all of your records and is not able to use your database index.
  5. I'm not sure what exactly is happening with the $clients variable and the mapping but it looks a lot like the famous N+1 problem. Are you not able to resolve this with smarter eager loading?

In short, first figure out which queries are executed and which ones are slow or which ones you can optimize by using eager loading.

And to answer your other questions:

Is there a way to measure this query for debugging without replicating production data? (Due to privacy regulation, I really don't want production data on my machine)

You could use database seeders with the help of model factories and faker. See https://laravel.com/docs/5.7/seeding

Could removing the note_text field have an effect on the time of the query to complete?

Probably not, it looks like you only retrieve 25 records with this value and do not filter or sort on this field so that shouldn't be the biggest problem here.

1 like
Drfraker's avatar

@CLICK - Wow thanks for the in depth reply.

I will look at the article and see about the simple paginator instead of length aware. I think it is the count() that is taking so much time. I'll also check my indexes to ensure they are set up correctly. Also, yes the filter is using LIKE %search% maybe I'll have to find a better way to do that too.

jlrdw's avatar

What is

list($query, $totalCount)

See:

https://laracasts.com/discuss/channels/guides/length-aware-paginator

and

https://christophersax.com/2016/custom-lengthawarepaginator-in-laravel/

A count on a large resultset is fast if only counting the id of matching results:

As example:

$result = $this->db->select('SELECT COUNT(dogid) as total FROM ' . PREFIX . 'dogs WHERE dogname LIKE :search and adopted = 1', array('search' => $dogsearch));

// not laravel example

Would be fast with 100 or 100,000 in resultset.

But paginating by index is also fine, but I would never want to retrieve a million results at once.

I would drop eloquent and use db facade for such a query.

Drfraker's avatar

@JLRDW - the list() function assigns variable as if they were an array. You can see that the getFilteredNotes() function that is being called returns an array. Well by using list($query, $totalcount) the variables are set automatically from the return value.

jlrdw's avatar

Are you getting the entire result set in an array or collection , that is not the way to properly paginate.

See the guide in the link I posted.

Sorry if I misunderstood something.

Please or to participate in this conversation.