Laravel order by column and relationship column

I have a query which has needs to order by additional data on relationship columns. This is my most recent attempt:

$odds = Odd::with([
    'box', 'set'
])->get()->sortBy(function($query) {
    return $query->box->position;
})->sortBy('created_at');

The current database tables are like so:

id.    box_id.   created_at
1      1         2021-04-02 17:15:04
2      2         2021-04-02 17:15:04
3      3         2021-04-02 17:15:04
4      4         2021-04-02 17:20:05

Box
id     set_id    position
1      1         3
2      1         1
3      1         5
4      2         2

The query above is able to sort on odds.created_at, but will not sort on the box.position. The result needs to be in date order then with in date group needs to be ordered by position, like so:

id.    box_id.   created_at            set_id    position
2      2         2021-04-02 17:15:04   1         1
1      1         2021-04-02 17:15:04   1         3
3      3         2021-04-02 17:15:04   1         5
4      4         2021-04-02 17:20:05   2         2

I have also tried these methods:

 $odds = Odd::with([
    'box', 'set'
])->get()->sortBy('created_at')->sortBy('box.position');

 $odds = Odd::with([
    'box' => function ($query) {
            $query->orderBy('position');
        },
    'set'
  ])->orderBy('created_at')->get();