How to restrict with records returned in query

I have 4 models in my app, category, product, sku, and client. Category has many product and products have many skus. Clients have many skus and skus have many clients.

class Category extends Model
{
    public function products(): HasMany
    {
        return $this->hasMany(Product::class);
    }

    public function skus()
    {
        return $this->hasManyThrough(Sku::class, Product::class);
    }
}

class Product extends Model
{
    protected $with = ['skus'];

    public function skus(): HasMany
    {
        return $this->hasMany(Sku::class);
    }

    public function category(): BelongsTo
    {
        return $this->belongsTo(Category::class);
    }
}

class Sku extends Model
{
    public function product(): BelongsTo
    {
        return $this->belongsTo(Product::class);
    }

    public function clients(): BelongsToMany
    {
        return $this->belongsToMany(Client::class)
            ->using(ClientSku::class)
            ->withPivot('price');
    }

    public function scopeOfClient(Builder $query, Client $client): void
    {
        $query->whereHas('clients', function ($query) use ($client) {
            $query->where('id', $client->id);
        });
    }
}

class Client extends Model
{
    public function skus(): BelongsToMany
    {
        return $this->belongsToMany(Sku::class)
            ->using(ClientSku::class)
            ->withPivot('price');
    }
}

I want to restrict in certain cases which categories certain clients see, and I can do this with the following query.

Category::whereHas('skus', function ($query) {
    $query->ofClient($clientId);
})->get();

The query returns the correct categories that the client should see, but it returns all the skus as well. How can I restrict it so that it only shows the skus for that client (i.e. only skus that are in the client_sku pivot table)?