alixaxel's avatar

Bulk DB Inserts Randomly Regressing?

Hey all, first post here so forgive me if I'm breaking any convention.

I'm having this weird problem with an Artisan command that inserts huge (several GBs) amounts of data in a SQLite database. Basically, the command appears to run fine for a long time, periodically reporting how far along in terms of completion it is. When the command finishes (apparently successfully) I check my database and several million records are missing completely and when I check the logs I only see a handful of constraint violations (all of which are expected/acceptable). I've never seen SQLite behave this way and I'm starting to suspect it may be something related to Laravel or my Eloquent models.

I start my command with a few SQLite PRAGMAs to improve insert performance:

public function __construct()
{
    ini_set('memory_limit', '512M');

    DB::connection()->disableQueryLog();

    $pragmas = [
        'automatic_index' => 'ON',
        'cache_size' => '4096',
        'cache_spill' => 'OFF',
        'foreign_keys' => 'ON',
        'journal_mode' => 'TRUNCATE',
        'journal_size_limit' => '0',
        'legacy_file_format' => 'OFF',
        'locking_mode' => 'NORMAL',
        'page_size' => '4096',
        'recursive_triggers' => 'ON',
        'secure_delete' => 'ON',
        'synchronous' => 'OFF',
        'temp_store' => 'FILE',
    ];

    foreach ($pragmas as $key => $value) {
        DB::connection()->getPdo()->exec(sprintf('PRAGMA %s=%s;', $key, $value));
    }

    $this->i = 0;

    parent::__construct();
}

Just in case you're wondering, I've tried with and without those PRAGMAs.

The rest of the command logic can be summarized as follows:

public function handle()
{
    $i = 0;

    while ($i < 10000000) {
        if ($i % 2048 == 0) {
            DB::connection()->beginTransaction();
        }

        try {
            Stuff::create([
                'id' => $i,
                'md5' => md5($i),
            ]);
        } catch (\Exception $exception) {
            Log::error($exception->getMessage(), [$exception->getFile(), $exception->getLine()]);
        }
        
        if (++$i % 2048 == 0) {
            DB::connection()->commit();
        }
    }

    if ($i % 2048 != 0) {
        DB::connection()->commit();
    }
}

This flow of inserting in bulk transactions is usually recommended for improving the performance. What totally baffles me is that up to a point all records are inserted and durable but then something mysterious happens and nothing else gets inserted. If I resume the same command from the last known inserted ID, the inserts continue - but even then, sometimes the database still regresses a few million records without any notice.

I'm just wondering, has someone experienced this strange behavior before?

From all my heavy (vanilla) usage with SQLite, this is the first time I see it happening.

0 likes
0 replies

Please or to participate in this conversation.