Best Way to Load Large Amounts of Data for Optimal Page Load Time
Hello! I have lots of data that needs to be displayed in a big table. This data is stored in all sorts of tables due to the way the relationships are setup and how we receiving our reporting data.
-----------------------------------------------------------------Relationships-----------------------------------------------------------------
Here are the important datasets and the models they "have many" of...
website: campaign, group, keyword, facebook_ad, google_ad, yahoo_ad, valid_click_ad
campaign: group, keyword, facebook_ad, google_ad, yahoo_ad, valid_click_ad
group: keyword, facebook_ad, google_ad, yahoo_ad, valid_click_ad
keyword: valid_click_ad
Also every campaign, group, and keyword belong to a user.
-----------------------------------------------------------------Data to Display-----------------------------------------------------------------
------------------Website------------------
I need to be able to select campaigns/groups/keywords by date range and/or user, then display the following data (all sudo code)
website->name
website->facebook_ad->sum(spend)
website->facebook_ad->sum(impressions)
website->google_ad->sum(spend)
website->google_ad->sum(impressions)
website->yahoo_ad->sum(spend)
website->yahoo_ad->sum(impressions)
website->valid_click_ad->sum(clicks)
website->valid_click_ad->sum(revenue)
website->valid_click_ad->sum(case(tq >= 0, tq, 0))
website->valid_click_ad->sum(case(tq >= 0, 1, 0))
website->valid_click_ad->count
------------------Campaign------------------
Then for each website, I need to show each of it's campaigns and this data:
campaign->name
campaign->facebook_ad->sum(spend)
campaign->facebook_ad->sum(impressions)
campaign->google_ad->sum(spend)
campaign->google_ad->sum(impressions)
campaign->yahoo_ad->sum(spend)
campaign->yahoo_ad->sum(impressions)
campaign->valid_click_ad->sum(clicks)
campaign->valid_click_ad->sum(revenue)
campaign->valid_click_ad->sum(case(tq >= 0, tq, 0))
campaign->valid_click_ad->sum(case(tq >= 0, 1, 0))
campaign->valid_click_ad->count
------------------Group------------------
Then for each campaign, I need to show each of it's groups and this data:
group->name
group->facebook_ad->sum(spend)
group->facebook_ad->sum(impressions)
group->google_ad->sum(spend)
group->google_ad->sum(impressions)
group->yahoo_ad->sum(spend)
group->yahoo_ad->sum(impressions)
group->valid_click_ad->sum(clicks)
group->valid_click_ad->sum(revenue)
group->valid_click_ad->sum(case(tq >= 0, tq, 0))
group->valid_click_ad->sum(case(tq >= 0, 1, 0))
group->valid_click_ad->count
------------------Keyword------------------
Then for each group, I need to show each of it's keywords and this data:
keyword->name
group->valid_click_ad->sum(clicks)
group->valid_click_ad->sum(revenue)
group->valid_click_ad->sum(case(tq >= 0, tq, 0))
group->valid_click_ad->sum(case(tq >= 0, 1, 0))
group->valid_click_ad->count
-----------------------------------------------------------------Main Questions-----------------------------------------------------------------
So you can see that this can easily get out of hand, especially with the selectors of date range and user.
My biggest question to tackle is how best to achieve loading this data. Is there a way to achieve this with eager loading that won't take forever (I would prefer to have all data loaded upfront for simplicity's sake) or should I wait to load hidden with lazy load (I am displaying this data in drop-down tables)?
What is the best way to select data between date ranges and by user? Especially with all these sums and sub-models?
Any ideas on the best way to tackle this to get my page loading in a reasonable amount of time would be greatly appreciated.
Please or to participate in this conversation.