Eloquent Many to Many select User without certain Roles

So I have User & Role models with many-to-many relationship, I have 3 roles: super, admin and moderator with 4 users let's says: John, Mike, James and Larry.

John is a super, Mike has admin and moderator roles, James is an admin and Larry is a moderator. To displaying users who doesn't have a certain roles I created this scope:

public function scopeDoesntHaveRoles($query, $roles = [], $column = 'id') {
    return $query->whereDoesntHave('roles')->orWhereHas('roles', function ($q) use ($roles, $column) {
        $q->whereNotIn($column, $roles);
    });
}

When I call Users::doesntHaveRoles([1])->lists('name', 'id') to get users who doesn't have super role, it works and returns:

{"2":"Mike","3":"James","4":"Larry"}

But, when I trying to list users who doesn't have admin role Users::doesntHaveRoles([2])->lists('name', 'id'), yes James is not shown there but Mike is appeared while he is actually has admin role:

{"1":"John","2":"Mike","4":"Larry"}

I think it's because Mike is also has moderator role, do you see something wrong in my scope? or do you have other solutions?

Thanks