Policy Logic

Hello! I have Users and Orders in my app. User can only view his own orders, but if he is Admin (field isAdmin in users table with boolean) - he can view any user's orders.

Route::get('/{user}/orders', 'OrdersController@index')->name('orders.index');

I try to check the rights of User in index method, but doing something wrong

public function index(User $user)
{
    if (auth()->id() != $user->id || !auth()->user()->isAdmin) {
        abort(403);
    }
}

if I leave only the first part of the condition its working good, but only for regular User who can not watch other people's orders

public function index(User $user)
{
    if (auth()->id() != $user->id) {
        abort(403);
    }
}

Tell me please what wrong with that logic, and what is the right way to do it.