OrderBy eager load result

I've a set of models and each model has a specific set of specification values eager loaded.

an example for the eager loaded spec value would be something like:

#items: array:14 [▼
    0 => SpecValue {#1007 ▼
      #table: "specification_values"
      #connection: null
      #primaryKey: "id"
      #perPage: 15
      +incrementing: true
      +timestamps: true
      #attributes: array:9 [▶]
      #original: array:9 [▼
        "id" => 9
        "specification_id" => 9
        "model_id" => 149
        "value" => "10,000 impressions"
        "filtered_value" => "10,000"
        "created_at" => "2015-12-31 16:47:11"
        "updated_at" => "2015-12-31 16:40:52"
        "name" => "max monthly duty cycle"
        "title" => "Monthly Volume"
      ]
      #relations: []
      #hidden: []
      #visible: []
      #appends: []
      #fillable: []
      #guarded: array:1 [▶]
      #dates: []
      #dateFormat: null
      #casts: []
      #touches: []
      #observables: []
      #with: []
      #morphClass: null
      +exists: true
      +wasRecentlyCreated: false
    }
    1 => SpecValue {#1008 ▶}
    2 => SpecValue {#1009 ▶}
   13 => SpecValue {#1020 ▼
      #table: "specification_values"
      #connection: null
      #primaryKey: "id"
      #perPage: 15
      +incrementing: true
      +timestamps: true
      #attributes: array:9 [▼
        "id" => 5
        "specification_id" => 5
        "model_id" => 149
        "value" => "01-11-2006"
        "filtered_value" => "01-11-2006"
        "created_at" => "2016-02-27 12:21:15"
        "updated_at" => "2016-02-27 12:21:15"
        "name" => "date"
        "title" => "Date"
      ]
      #original: array:9 [▶]
      #relations: []
      #hidden: []
      #visible: []
      #appends: []
      #fillable: []
      #guarded: array:1 [▶]
      #dates: []
      #dateFormat: null
      #casts: []
      #touches: []
      #observables: []
      #with: []
      #morphClass: null
      +exists: true
      +wasRecentlyCreated: false

As you can see, I've multiple specification values.

Now, I want to order the models depending on the date from the spec values. (The last array showed above).

This is the query I use now

$categories = Category::with(['models'=>function($query) use ($brand_name){
            $query->with(['specValues'=> function($query){
                $query->join('specifications AS s', 's.id', '=', 'specification_values.specification_id') ;
            }])->with('brand')->with('images')->whereHas('brand', function($query) use ($brand_name){
                $query->where('name', $brand_name);
            });
        }, 'models.brand'])->get();

But this won't sort the model with the spec date.

How can I sort this?