Eager loading a multiple indexes relationship

I'm currently working on a project in which I have to use a 10 years old database which has a pretty bad structure. Anyway I have two tables: one for the products and one for their colors. Every product is identified by a reference and every color is identified by its own reference + its product's one. The two references are located in two separated columns. For instance I have a product which has the reference 123. This product has 3 colors: 123-01, 123-02 and 123-03. These two values are split into two columns: nocol (product reference) and nopat (color reference). I can't have a unique id for each color as the database is dropped every night so the ids are always changing (I know this is pretty weird).

For now I have three models: Product, Color and CartItem. My Color model has a one-to-one relationship with the Product model. My CartItem has a one-to-one relationship with my Color model which is defined like this:

public function color() {
  return $this->hasOne('App\Color', 'nocol', 'color_ref')->where('nopat', $this->product_ref);
}

This is working fine because I can retrieve the Color from the CartItem with a simple $cartItem->color. The problem is that it doesn't work with eager loading. When I try to retrieve my CartItems with something like this:

CartItem::with('product', 'color')->get();

I end up with a null relationship when I dump the result.

Is there any other way to write this kind of relationship when the index consists in two separated columns?