Ordering by a Relationship

I'm working with Spatie's permissions package and trying to order my users by their role but applying a specific order of those roles by the role name. Currently with my query it its not ordering and I'm curious to know if I'm doing something wrong here.

public function scopeOrderByRole($query)
    {
        return $query->whereHas('roles', function ($query) {
            return $query->orderBy(DB::raw('
                case
                    when roles.name = "super-admin" then 1
                    when roles.name = "company-admin" then 2
                    when roles.name = "maintainer-admin" then 3
                    when roles.name = "user" then 4
                end
            '), 'desc');
        });
    }