Two way pivot table.

I am currently building a dating application that uses a swipe feature like Tinder. Because of this, I need to have a single table called user_match that acts as the pivot between users. This table is currently setup as follows:

+------------+-----------------+------+-----+---------+----------------+
| Field      | Type            | Null | Key | Default | Extra          |
+------------+-----------------+------+-----+---------+----------------+
| id         | bigint unsigned | NO   | PRI | NULL    | auto_increment |
| user_id_a  | bigint unsigned | NO   | MUL | NULL    |                |
| user_id_b  | bigint unsigned | NO   | MUL | NULL    |                |
| created_at | timestamp       | YES  |     | NULL    |                |
| updated_at | timestamp       | YES  |     | NULL    |                |
+------------+-----------------+------+-----+---------+----------------+

I would like to get a user's matches via $user->matches(). The blocker I'm running into with Eloquent is that pivot tables are, what it seems to be, one way. However, the user object I'm working with can have it's ID in either user_id_a or user_id_b. This breaks $this->belongsToMany(self::class, 'user_match', 'user_id_a', 'user_id_a') as the query generated is:

select *
  from `users`
 inner join `user_match` on `users`.`id` = `user_match`.`user_id_a`
 where `user_match`.`user_id_a` = ?
   and `users`.`deleted_at` is null

This is because the join is where it makes this a one way query. Adding orWherePivot('user_id_b', $this->id) won't help.

Is there an Eloquent solution to this kind of implementation, or am I better off writing a custom query that hydrates user models?