@haskell so you want a multi-auth? The User and the Group to be able to use it, or just the Group.
I believe the steps you are missing are setting a custom guard and a custom middleware to handle the checks.
Take a look at this here for multi-auth:
https://github.com/laravel/passport/issues/161#issuecomment-460034599