One to many relationship in a somehow big table - slow performance

I have this table source_data with source_data_details as its child with one to many relationship

source:
=======
id
name
create_by
create_datetime
status

source_field:
=============
id
source_id (FK from source.id)
name
data_type
max_length
default_value
special_field
status

source_data:
============
id
source_id (FK from source.id)
create_datetime
generated_code

source_data_details:
====================
id
source_data_id (FK from source_data.id)
source_field_id (FK from source_field.id)
sourve_data_value

allocation:
===========
id
source_data_id (FK from source_data_id)
create_by
create_datetime
researcher_id
status

basically what is happening is that, I can create a source, and specified the field for each source. I have the data inserted into the source_data and then allocate the data to people for them to research.

And now I have one source with 12 fields with 330673 records in source_data and 3961051 records in source_data details.

I am retrieving them using Laravel Eager Loading and it is really such a pain because it is very slow (8s-10s) in Test environment and 5s in Production. And yes I am using pagination to limit the data I am retrieving and retrieving 5 records at a time.

Additional information is that I also have added index to source_data_details.source_data_id and it did not really much, it just made the average retrieving time better (nearly 8seconds constant)

In my source_data model:

public function details()
{
    return $this->hasMany('SourceDataDetailsModel', 'source_data_id', 'id');
}

public function allocation()
{
    return $this->hasOne('AllocationsModel', 'source_data_id', 'id');
}

This is my eager loading:

$query = \SourceDataModel::with(
                                   array
                                   (
                                       'details',
                                       'allocation'
                                   )
                               );

 $query->where('source_id', '=', 1);
 Paginator::setCurrentPage($request['current_page']);
 $source_data = $query->paginate($request['record_count']);

Some of the things I am considering now:

• Remove source_data_details and put it into source_data (this will remove the dynamic feature) which will affect quite alot in other modules

Before I do this, I wanted to know if there is any other way like maybe switching to fluent from eloquent?

Any suggestion is very much appreciated.