futurefuture's avatar

Very Long Job

Hey, was looking for some advice here to see if maybe anyone had any insight:

I'm running an eCommerce site with approx. 17000 products. We have been collecting analytic data such as impressions, add to carts, etc on Amazon Redshift, and we've built a dashboard for our partners to view their products analytics.

We've noticed that running these individual queries (i.e add to carts) against start date and end date in real time was pretty slow on Redshift, so we decided to compile all the data into daily reports that are enriched with some of our internal data that we could query against (so we are looping in PHP as well).

This has seemingly been much, much quicker already, but the querying to generate the reports has been offloaded to background jobs.

These jobs still take a bunch of time, especially as more partners are onboarded for this.

My questions are:

  1. Is generating reports the best approach to something like this?

  2. Is using Laravel Jobs the best way compiling such data? I.e, should I be using a separate language like Python to compile this data?

  3. Would anyone know if there are any resources available for standard practice on such a need?

Thanks in advance,

0 likes
0 replies

Please or to participate in this conversation.