Chaining Multiple Belongs To Many Relationships

Consider the following Schema:

users
    id
    ...

roles
    id
    ...

permissions
    id
    ...

role_user
    role_id
    user_id
    ...

permission_role
    permission_id
    role_id
    ...

And the following Relationships:

User.php

public function roles()
{
    return $this->belongsToMany(Role::class);
}

Role.php

public function users()
{
    return $this->belongsToMany(User::class);
}

public function permissions()
{
    return $this->belongsToMany(Permission::class);
}

Permission.php

public function roles()
{
    return $this->belongsToMany(Role::class);
}

Now then, let's say that I want to find what Permissions a User has:

User.php

public function permissions()
{
    return ???;
}

Or perhaps I want to look up the Users from the Permission:

Permission.php

public function users()
{
    return ???;
}

How can this functions be completed? For the relational functions, I want to be able to call both $permission->users() and $permission->users.

I know how to do Query Scopes for these types of Relationships (albeit complicated):

User.php

public function scopeWithPermission($query, Permission $permission)
{
    $query->join('role_user', 'users.id', '=', 'role_user.role_id') // Join with `role_user`
            ->join('permission_role', 'role_user.role_id', '=', 'permission_role.role_id') // Join with `permission_role`
            ->where('permission_role.permission_id', $permission->id); // Filter by Permission ID
}

And I know how to get the results that I'm looking for, but not they're not in a Relationship, meaning I can't call both $permission->users() and $permission->users:

Permission.php

public function users()
{
    // Determine the Roles associated with this Permission
    $roles = $this->roles()->with('users');

    // Determine the Users associated with the Roles
    $users = $roles->get()->pluck('users');

    // Turn the Collection of Collections into a Single Collection
    $users = call_user_func_array('array_merge', $users->toArray());

    // Pluck the IDs from the Users
    $ids = array_column($users, 'id');

    // Remove Duplicate IDs
    $ids = array_unique($ids);

    // Return the Users associated with this Permission
    return User::whereIn('id', $ids);
}

$permission->users();

Illuminate\Database\Eloquent\Builder

$permission->users()->get();

Illuminate\Database\Eloquent\Collection {
    all: [
        App\Users {
            ...
        },
        ...
    ]
}

$permission->users;

LogicException with message 'Relationship method must return an object of type Illuminate\Database\Eloquent\Relations\Relation'

I'm game for creating a new Relation called belongsToManyThroughMany if that's what it comes to. :P