Splitting returned model into categories on the view

I have a number of products which each belong to a category. the Model relationships are set up and working. I also have 3 groups which are currently just three fields in the products table.

Group 1 Group 2 Group 3

Each group contains products that belong to multiple categories and a product could belong to any number of the 3 groups or none. Now I have a route to show which products belong to each group

/products/group1

I run a query such as

$products = Product::where('group1', 1)->get();

on this view I want to split the results into the categories, the only way I can see to do this without running multiple queries in the controller would be to remove the group fields from the products table and create a many to many relationship for each of the groups.

Any other suggestions or ideas on how to tackle this?

I'm guessing the many to many would be best but it just seems overkill to create 3 the extra models and tables when what I really want to achieve could be done with these fields.

I hope this is clear, open to any suggestions.