Laravel sort by pivot table column

I am trying to sort an eloquent query in laravel by a column from the pivot table.

Basically as long as the product is favorited by the user, it is added to the pivot table, and it should first show the favorited ones, and then the remaining ones. Since I will be using foreach in the blade.

products table contains these columns:

    id, category, product_name, priority, status, created_at, updated_at

users table contains these columns:

    id, name, email, created_at, updated_at

with the following in the users model:

        public function favorites()
    {
        return $this->belongsToMany(Products::class, 'favorites', 'user_id', 'product_id')->withTimeStamps();
    }

favorites table contains these columns:

    id, user_id, product_id, created_at, updated_at

In the controller, the eloquent query is currently:

    $productinprogress = Products::all()->sortByDesc("priority")->where('status', 'inprogress');

I am just confused as to how to tackle this.

Your help is appreciated. Thank you so much.