Get specific manyToMany relationship additional relationship

Lets say i have users and clients many to many relationship and there in the pivot table I have:

$table->foreignId('user_id');
$table->foreignId('client_id');
$table->foreignId('role_id'); -> additional relationship to spatie roles

How would I get all users per client (clients per user) with their roles per that relationship?

public function users () {
        return $this->belongsToMany(User::class, 'user_client')->withPivot(['role_id']);
}
public function clients () {
        return $this->belongsToMany(Client::class, 'user_client')->withPivot(['role_id']);
}

I know I can get users roles or clients roles, but how would I get roles for users per client without making it as separate joined UserClient model? How would I do, $user->can('edit.client') based on user_client table? Are wildcards the only way where I would need to generate permissions and roles per each client?