Laravel orderByRaw with a condition

I have a model ModelA mapping to table model_a. It has one to many relationship with table model_b and the relationship is named modelab

Now in model_b there are following columns

id | key | value

What I am trying to achieve is

User will send the value of key and order (ASC, DESC). What I want is based on the key provided the values should be sorted. For instance let's say there are 3 items in model_a each having 2 attributes in model_b. So data looks something like this

data = [{
 "id": 1,
 "name": "ABC",
model_ab: [
{"id": 1, "key": "date", "value": "2022-01-22"},
{"id": 2, "key": "index", "value": "1"},
]
},

{
 "id": 2,
 "name": "PQR",
model_ab: [
{"id": 3, "key": "date", "value": "2022-02-22"},
{"id": 4, "key": "index", "value": "2"},
]
},
{
 "id": 3,
 "name": "XYZ",
model_ab: [
{"id": 5, "key": "date", "value": "2022-03-22"},
{"id": 6, "key": "index", "value": "3"},
]
}
] 

Now if user provides key to be "date" and order to be "DESC" then I want data to be arranged in ascending order based on date from model_ab. So the order will be something like this

data = [
{
 "id": 3,
 "name": "XYZ",
model_ab: [
{"id": 5, "key": "date", "value": "2022-03-22"},
{"id": 6, "key": "index", "value": "3"},
]
},
{
 "id": 2,
 "name": "PQR",
model_ab: [
{"id": 3, "key": "date", "value": "2022-02-22"},
{"id": 4, "key": "index", "value": "2"},
]
},
{
 "id": 1,
 "name": "ABC",
model_ab: [
{"id": 1, "key": "date", "value": "2022-01-22"},
{"id": 2, "key": "index", "value": "1"},
]
}
]

How can I achieve this with eloquent queries? I have tried something like this.

$data = ModelA::with(['model_ab' => function ($q) use($key, $order){
  $q->orderByRaw("CASE WHEN model_b.key = ? THEN model_b.value END $order", [$key]);
}])->paginate(10) ;

Somehow this doesn't work. Am I doing something wrong?

Thanks in advance :)