Separate roles/permissions for users belonging to multiple organizations

I have an application where a user can belong to multiple organizations. I want to set it up in a way that a user can have different roles/permissions for each organization. I am using Laravel and plan on implementing Spatie/laravel-permission. What is the best way to implement this?

I have tried setting up two guards, one for the main user account and another for the pivot model between the user and the organization they log into. So basically when they log into the app using the main user model, I ask them which organization they would like to log into, when they choose the organization I will then also set up an auth session on the pivot model that links the user to the organization and access the roles off that model. This works, but having to manage the auth sessions is kind of a pain.

// User Model
class User extends Authenticatable
{
    public function organizationUsers()
    {
        return $this->hasMany(OrganizationUser::class);
    }
}

// OrganizationUser Model

class Organziationuser extends Authenticatable
{
    use HasRoles;

    public $guard_name = 'organization_user';

    public function organization()
    {
        return $this->belongsTo(Organization::class);
    }

    public function user()
    {
        return $this->belongsTo(User::class);
    }
}

I would expect a user to be able to log into the application using a single login, but also be able to have different permissions for different organizations.

I wonder how it would look if i added an organization_id to the model_has_roles and model_has_permissions tables??