MauricioMateus's avatar

Laravel Insert urls with query string causes performance issue

Hello:

Please see the example below:

$data = collect(range(0, 300))->map(fn () => [ // 'googleDriveDocument' => Str::random('180'), // 'googleDriveDocument' => 'https://docs.google.com/document/d/16jabcPOytUWDTXv1QjAau5123453K5r2NbxtU/edit?usp=drive_link', 'googleDriveDocument' => 'https://docs.google.com/document/d/16jubu12345OytUWDTXv1QjAau5abcde5r2NbxtU/edit', ])->toArray();

If I run Model::insert($data) for the random strings, the insert takes 2 seconds. But if I insert urls, it takes 16 seconds for the same number of rows.

If I delete the query string from the urls, the performance is the same.

This turns into minutes and even hours as the size of the insert grows (seems like O^2) when the url has a question mark and linear complexity for the alpha numeric strings.

Could this be related to Laravel trying to prevent sql injection, or could it have to do with the prepared statments?

Kind regards

0 likes
7 replies
LaryAI's avatar
Level 58

The performance issue you're experiencing when inserting URLs with query strings in Laravel is likely not directly related to SQL injection prevention or prepared statements. Instead, it might be related to how the database engine handles the data, especially if the URLs are being indexed or if there are constraints on the column.

Here are a few steps you can take to diagnose and potentially resolve the issue:

  1. Check Database Indexes:

    • If the column where you're inserting the URLs is indexed, the database might be taking longer to insert because it needs to update the index with each new entry. Consider whether the index is necessary for this column.
  2. Column Data Type:

    • Ensure that the column data type is appropriate for storing URLs. Typically, a TEXT or VARCHAR type is used. If the column is too small, it might be causing additional overhead.
  3. Batch Inserts:

    • If you're inserting a large number of rows, consider breaking them into smaller batches. This can sometimes improve performance by reducing the load on the database in a single transaction.
  4. Database Configuration:

    • Check your database configuration for any settings that might be affecting performance, such as transaction isolation levels or buffer sizes.
  5. Profiling and Logging:

    • Use Laravel's query logging or a database profiling tool to see if there are any unexpected delays or operations occurring during the insert.
  6. Database Engine:

    • Different database engines handle large text inserts differently. If you're using MySQL, for example, consider whether switching to a different storage engine (like InnoDB or MyISAM) might help.

Here's a code example to demonstrate how you might batch your inserts:

use Illuminate\Support\Facades\DB;

$data = collect(range(0, 300))->map(fn () => [
    'googleDriveDocument' => 'https://docs.google.com/document/d/16jubu12345OytUWDTXv1QjAau5abcde5r2NbxtU/edit',
])->toArray();

// Batch size
$batchSize = 50;

// Insert in batches
foreach (array_chunk($data, $batchSize) as $batch) {
    DB::table('your_table_name')->insert($batch);
}

By inserting in smaller batches, you might see an improvement in performance. If the issue persists, further investigation into the database server's performance and configuration might be necessary.

JussiMannisto's avatar

Could this be related to Laravel trying to prevent sql injection, or could it have to do with the prepared statments?

No.

Snapey's avatar

you are not inserting urls. They might look like Urls but they are just strings. Nothing more.

The database sees them as strings only varying in length slightly.

Something else is going on which is impossible to see from this short snippet.

MauricioMateus's avatar

@Snapey yes, I couldn't fix the performance issue until I removed the question mark before doing the insert. I could not explain the issue. That type of string causes a performance issue when doing large inserts, which does not happen if the strings don't contain a question mark

fullstackalien's avatar

I'm seeing this right now as well. The query finishes quickly. The new rows are in the table, but something not in the logic chain is getting hung up. I have ATTR_EMULATE_PREPARES set to true in my config and without it the query executes immediately, but with enabled the query takes about 3 minutes.

Still looking into this. I'll report back.

fullstackalien's avatar

Ok, so isolated a little more. It is the logQuery function in Illuminate/Database/Connection.php that is hanging, which is kind of what I expected.

Strange enough its the event dispatcher for the logQuery. When I comment it out it process very quickly. I'm doing some debugging in that event but nothing is showing up in my logs. So, it's making me think that it never dispatches and potentially is timing out before finishing.

    public function logQuery($query, $bindings, $time = null)
    {
        $startTime = microtime(true);
        $this->totalQueryDuration += $time ?? 0.0;

        logger()->info(__METHOD__.' QueryExecuted event about to trigger at '. microtime(true) - $startTime.' total duration '.$this->totalQueryDuration);

        // $this->event(new QueryExecuted($query, $bindings, $time, $this));

        logger()->info(__METHOD__.' query executed event in '.microtime(true) - $startTime.' total duration '.$this->totalQueryDuration);

        $query = $this->pretending === true
            ? $this->queryGrammar?->substituteBindingsIntoRawSql($query, $bindings) ?? $query
            : $query;

        logger()->info(__METHOD__.' get raw sql for logging '.microtime(true)-$startTime);

        if ($this->loggingQueries) {
            $this->queryLog[] = compact('query', 'bindings', 'time');
        }

        logger()->info(__METHOD__.' finished in '.microtime(true) - $startTime);
    }
fullstackalien's avatar

Well, that was a fun little side quest. The issue is a combination of enabling ATTR_EMULATE_PREPARES , having question marks present in your insert string values, and the Telescope package.

I called Telescope::stopRecording() prior to running these inserts and the problem is resolved. The QueryExecuted event was being dispatched properly, when Telescope acted on it something was hung.

I don't need Telescope to record here, so I'm moving on. I'm curious what is going on in Telescope, but I'm going to have to move on for now.

1 like

Please or to participate in this conversation.