Filament/Livewire Upload Failing in Stage and Prod

Hi All,

I've got an issue where uploads to DigitalOcean Spaces fail in Staging and Prod but work perfectly locally. Locally, I get a 200 OK response, and in Staging/Prod, I get 401 Unauthorized. I've double-checked configs, environment variables, API Keys, and Policies to ensure I'm not missing anything, and everything appears to have the correct values.

I'm using awcodes/filament-curator for media management.

There are some differences between Local and Staging/Production, as the latter two have additional monitoring/tooling, but nothing that interferes with the upload.

In an effort to debug, I tried changing the livewire.temporary_file_upload.disk config value to public, but this did not change any outcomes.

'temporary_file_upload' => [
        'disk' => 'public', 
        'rules' => null,
        'directory' => null,
        'middleware' => null,
        'preview_mimes' => [   
            'png', 'gif', 'bmp', 'svg', 'wav', 'mp4',
            'mov', 'avi', 'wmv', 'mp3', 'm4a',
            'jpg', 'jpeg', 'mpga', 'webp', 'wma',
        ],
        'max_upload_time' => 5, 
    ],

Using Telescope, this is what I was able to extract from the request. I am at a complete loss as to what could be causing this issue. Any help y'all can provide would be greatly appreciated!

Local Request:

{
	"cookie": "[REDACTED]",
	"te": "trailers",
	"sec-fetch-site": "same-origin",
	"sec-fetch-mode": "cors",
	"sec-fetch-dest": "empty",
	"referer": "https://mpc.test/portal/media/create",
	"sec-gpc": "1",
	"dnt": "1",
	"origin": "https://mpc.test",
	"content-length": "22746",
	"content-type": "multipart/form-data; boundary=---------------------------89199902241699356162743815900",
	"x-csrf-token": "[REDACTED]",
	"accept-encoding": "gzip, deflate, br",
	"accept-language": "en-US,en;q=0.5",
	"accept": "application/json",
	"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:122.0) Gecko/20100101 Firefox/122.0",
	"host": "mpc.test"
}

Local Response:

{
	"paths": [
		"/3HGlDtnYlNCl1FQSY8VoPpChkKzAib-metaSEMtbG9nby1ibGFjay13ZWJzaXRlLmpwZw==-.jpg"
	]
}

Staging/Production Request:

{
	"do-connecting-ip": "[REDACTED]",
	"cdn-loop": "cloudflare",
	"cookie": "[REDACTED]",
	"sec-fetch-site": "same-origin",
	"sec-fetch-mode": "cors",
	"sec-fetch-dest": "empty",
	"referer": "https://my.hopechurchstl.com/portal/media/create",
	"sec-gpc": "1",
	"dnt": "1",
	"origin": "https://my.hopechurchstl.com",
	"content-type": "multipart/form-data; boundary=---------------------------131969581742699529032200988978",
	"x-csrf-token": "[REDACTED]",
	"tracestate": "[REDACTED]",
	"traceparent": "[REDACTED]",
	"newrelic": "[REDACTED]",
	"x-newrelic-id": "[REDACTED]",
	"accept-language": "en-US,en;q=0.5",
	"accept": "application/json",
	"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:122.0) Gecko/20100101 Firefox/122.0",
	"x-forwarded-proto": "https",
	"content-length": "22748",
	"x-forwarded-for": "24.107.212.22,172.71.254.88",
	"accept-encoding": "gzip",
	"host": "my.hopechurchstl.com"
}

Staging/Production Response:

{
	"message": ""
}

LaryAI

2 years ago

Level 58

It appears that the issue you're facing with uploads to DigitalOcean Spaces in Staging and Production environments is related to authentication, as indicated by the 401 Unauthorized response. Here are some steps you can take to troubleshoot and resolve the issue:

Check Environment Variables: Ensure that the environment variables related to DigitalOcean Spaces (like DO_SPACES_KEY, DO_SPACES_SECRET, DO_SPACES_ENDPOINT, and DO_SPACES_REGION) are correctly set in your Staging and Production environments. They should match the values that work locally.
Permissions and Policies: Verify that the permissions and policies set on your DigitalOcean Spaces bucket allow for the necessary read/write operations from the Staging and Production servers.
CORS Configuration: Check the CORS configuration on your DigitalOcean Spaces bucket. The CORS settings must allow requests from your Staging and Production domain names.
Middleware Conflicts: Although you mentioned that there's nothing that interferes with the upload, double-check any middleware that might be running in Staging/Production that isn't running locally. This includes checking for any security headers or configurations that might be blocking the request.
Server Time Sync: Ensure that the server time is synchronized with NTP. A time drift can cause issues with signed requests.
Logging and Monitoring: Increase the logging level on your Staging/Production servers to capture more detailed information about the request and response. This might provide more insight into why the request is being rejected.
Livewire Configuration: Double-check the Livewire configuration for file uploads. Ensure that the temporary_file_upload.disk is set to a disk that uses the DigitalOcean Spaces driver and that the configuration matches between local and Staging/Production.
Test with Tinker: Use Laravel Tinker on your Staging/Production server to manually attempt an upload to DigitalOcean Spaces using the same credentials and configuration. This can help isolate the issue to either your application code or the server/environment setup.
Network Configuration: Ensure that there are no network configuration issues that might be preventing your Staging/Production server from communicating with DigitalOcean Spaces.
Check for Service Outages: Check DigitalOcean's status page for any ongoing issues that might be affecting Spaces.

If after checking all the above, the issue persists, you may want to reach out to DigitalOcean support for further assistance, as the problem might be on their end or require more detailed investigation into your account specifics.

Remember to keep sensitive information such as API keys and secrets secure and avoid exposing them in logs or error messages.

mostofa_me

1 year ago

Level 1

have you got any solution? i am having the same issue

onairmarc

1 year ago

Level 5

@mostofa_me I can't remember exactly how we fixed this, but I do remember that shortly after, we created a private fork to make further customizations and fix other issues we were facing.

Please or to participate in this conversation.