Multiple pivots related to one model?

I have a situation I can't seem to figure out the logic for. I have a Users table and each User can be assigned a role (in a separate table). Each User also belongs to a Department.

Some users have the role of Supervisor, and they are supervisors of other users. I figure it's better to assign a Supervisor to a Department instead of each user. So, I created two pivot tables:

1st pivot table called user_departments user_id department_id

2nd pivot table called supervisor_departments user_id department_id

If I call $user->departments, I get the values from the 1st pivot table and that is correct.

If I call $supervisor->departments, I get the same values from the 1st pivot, when I should be getting values from the 2nd pivot.

Here are the models:

Department model

public function users()
{

    return $this->belongsToMany('App\User', 'user_departments',  'department_id', 'user_id');

}

public function supervisors(){

    $this->belongsTo('App\Supervisor','department_id', 'user_id');

}

User Model

public function departments()
{
    return $this->belongsToMany('App\Department', 'user_departments','user_id', 'department_id');
}

Supervisor Model

public function departments(){

    $this->hasMany('App\Departments')->withPivot('supervisor_departments');

}

Hopefully, someone can pinpoint what I'm missing or doing wrong. Thanks in advance!