On laravel site I make import of news using rss sources (now several dozens, later can be bigger). It works as from db tables rows are read which has rss link, which is used for rss data reading.
By 1 rss link 20-40 rows are read and written into database.
I wonder in which way have I to use database transaction for big ammount of data? 1 transaction for 1 rss link?
What have I to pay attention with big data imported to reduce productivity of my site?
Some special tools for this ?
"laravel/framework": "^10.48.4",
Thanks in advance!
I'm not really sure what you're asking about, but:
Make sure you are not inserting way too many rows in one single transaction to prevent complete table locking (as I'm assuming you are using Auto Increment keys which suffer from this problem). Insert the rows in small batches and perhaps do it in a background job.
Please or to participate in this conversation.