If I understood correctly you need another relation Athlete -> belongsToMany -> Round with pivot table athlete_round.
Don't mess with complex pivot tables linking more than two tables, Eloquent don't support them as far as I know. Stick to simple standard relations and combine them to fetch what you need.
Also json column is bad idea for you. Data must have a strict structure, this is general rule.
Well thought indexes and constraints are vital for you.
Also, what is "scores"? Is it another entity/model or just calculated value based on results of game/round?