Turn off Xdebug for Valet only

I'm running Xdebug on a OSX homebrew installation, mainly to debug in phpstorm. However, Laravel Valet is also using the same configuration where I hardly need Xdebug turned and rather have the performance of running php without Xdebug.

I'm trying to figure out if it would be possible to have laravel Valet running from a custom php.ini, or if there would be another solution to it...