Sort Collection by accessor with pagination

I have two many to many related models:

1. Contact
2. Catchup

The Contact model has two accessors to determine the last catchup, and next catchup (based on a calculation):

public function catchups()
    {
        return $this->belongsToMany(Catchup::class);
    }


    public function getLastCatchupAttribute()
    {
        return \Carbon\Carbon::create($this->catchups()->max('date'));
    }

    public function getNextCatchupAttribute()
    {
        return $this->last_catchup->add($this->catchup_value, $this->catchup_increment);
    }

I have a view (in livewire component) which lists all the contacts, showing their most recent catchup and next catchup:

use WithPagination;

    public $perPage = 10;
    public $search = '';
    public $orderBy = 'first_name';
    public $orderAscending = true; //true = ascending, false = descending

    public function render()
    {
        $contacts = Contact::search($this->search)
            ->where("user_id","=",Auth::user()->id) //Important to scope down to the specific user's contacts
            ->orderBy($this->orderBy, $this->orderAscending ? "asc" : "desc") 
            ->Paginate($this->perPage)     ;

The user can sort by a number of fields on the table. I am having issues adding the Accessor fields to the sort though, as an eloquent query cannot be sorted/ordered by an accessor. If I collect all of the contacts and apply sortBy to the resulting collection, I lose the ability to use paginate.

Any ideas on how I can achieve this?