First of all, you say a user can only belong to one brand. You don't really need a many-to-many relationship here, s one-to-many would do just fine.
Since you have a many-to-many relationship, this package should help you out to manage it in Nova: https://novapackages.com/packages/everestmx/nova-belongs-to-many-field