BelongsToMany with more than one table

(copied from StackOverflow)

I have a table that is linked to multiple other tables in a many-to-many fashion.

In my example, pmds (primary medical doctors) are records in the users table that

1. have a role_id that corresponds with pmd in the roles table; AND
2. are associated with the sites table through the intermediate table site_user; OR
3. are associated with the agency table through the intermediate table agency_user

The following function in User.php gets the information I want...

    public function allPmds()
    {
        return User::whereHas('role', function ($q) {
            $q->where('name', 'pmd');
        })->where(function ($q) {
            $q->whereHas('siteUsers', function ($q) {
                $q->where('site_id', $this->id);
            })->orWhereHas('agencies', function ($q) {
                $q->where('agency_id', $this->agency_id);
            });
        });
    }

But what I really want is eager loading. Something like this:

public function pmds()
    {
        return $this->belongsToMany('App\User')->where( <essentially the same code as above> );
    }

How can I do this in Eloquent?