As a suggestion, instead of trying to return the file from the api. return a temporary signed url to the file on your api server. First, create a web route on your api server to download the file. Then return the signed url like so:
URL::temporarySignedRoute(
'download', now()->addMinutes(30), ['id' => 1]
);
Make sure to either attach the “signed” middleware to the web route or manually check that the request has a valid Signature.