~/.composer/vendor/laravel/valet/cli/stubs/valet.conf is the file that gets copied to our ~/.valet/Nginx/*.dev virtualhosts.
I guess the only way to modify the port is to change it in the stub and then regenerate the sites? Basically, it uses nginx to serve the sites. So changing listen 80 or listen 443 is what you want.
It seems that that's going to auto-removed whenever I update Laravel. This seems more like a hack than a real configuration option. I guess I'll have to just write an on/off toggle script for now instead.