Forge backups not working - downgrade of AWS CLI is needed!

The 15th of January AWS updated the AWS CLI and SDK. Since that the Forge backups has not been working but reported error:

upload failed: - to s3://databases/daily/20250120105421/realpay.sql.gz An error occurred (InvalidRequest) when calling the PutObject operation: The value specified in the x-amz-trailer header is not supported

I'm using Scaleway object storage. I tried to change to Digital Ocean object storage but got the message below.

It seems like Forge should downgrade the AWS CLI/SDK version to make it work. Last known good version is v2.22.35

DigitalOcean Spaces Customers using the latest versions (released on Jan 15, 2025) of the AWS CLI or any AWS SDK, or relying on a tool or application that depends on any of these versions of the AWS CLI or an AWS SDK, are experiencing failed uploads to Spaces buckets in all regions.

This is caused by a change made by AWS on January 15, 2024 to the AWS CLI and almost all AWS SDKs (including boto3) to require use of “Data Integrity Protections for Amazon S3” (https://docs.aws.amazon.com/sdkref/latest/guide/feature-dataintegrity.html).

The currently recommended solution is to downgrade (roll back) to a version of the AWS CLI or AWS SDK released prior to January 15, 2025, or to an earlier version of tools or applications that do not depend on one of these versions of the AWS CLI or AWS SDK. The list of known impacted AWS client tools are: AWS CLI v2.23.0 (workaround - move to last known good version: v2.22.35) AWS SDK for Python (boto3) v1.36.0 (workaround - move to last known good version: 1.35.99) AWS SDK for Go 2025-01-15 release (workaround - move to last known good release: 2025-01-14)

LaryAI

1 year ago

Level 58

To resolve the issue with Forge backups not working due to the AWS CLI update, you can downgrade the AWS CLI to a version that is known to work correctly with your object storage provider. Here’s how you can do it:

Downgrade AWS CLI

Uninstall the Current AWS CLI Version:

First, you need to uninstall the current version of the AWS CLI. You can do this by running the following command:
```
sudo apt remove awscli
```
If you installed it using a different method (e.g., pip, homebrew), make sure to use the appropriate command to uninstall it.
Install a Specific Version of AWS CLI:

You can download and install the last known good version (v2.22.35) directly from the AWS CLI version archive. Here’s how you can do it:
```
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-2.22.35.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
```
Make sure to replace the URL with the appropriate one if you are using a different operating system.
Verify the Installation:

After installation, verify that the correct version is installed by running:
```
aws --version
```
This should output something like aws-cli/2.22.35.

Additional Considerations

Check for Compatibility: Ensure that any scripts or tools you are using are compatible with the version you are downgrading to.
Monitor for Updates: Keep an eye on updates from AWS and your object storage provider for any patches or updates that might resolve the issue without needing a downgrade.
Alternative Solutions: If downgrading is not feasible, consider using a different tool or method for backups that does not rely on the AWS CLI or SDKs affected by this change.

By following these steps, you should be able to restore functionality to your Forge backups using the AWS CLI.

[email protected]

1 year ago

Level 2

Hi Lary,

Thanks a lot for the great answer. Downgrade succeeded and my backups work again :-)

Just some comments:

AWS CLI was initially installed at my forge server with Snap so the current version should be removed with: sudo snap remove aws-cli

Also remove the old settings: sudo rm -f /snap/bin/aws

Then you can download the old version (do it in a seperat folder to make the clean up easier): curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-2.22.35.zip" -o "awscliv2.zip" unzip awscliv2.zip

When installing with apt you need to adjust the bash before running install because the bin destination changed: nano ~/.bashrc export PATH=/usr/local/bin:$PATH source ~/.bashrc

Now you can install aws (with update flag): sudo ./aws/install --update

And check the version: aws --version

It should now be: aws-cli/2.22.35 Python/3.12.6 Linux/6.8.0-51-generic exe/x86_64.ubuntu.24

And backup scripts will run without errors.

[email protected]

1 year ago

Level 2

I also got a reply from the Laravel team. I think this is the best way. The guide uses Snap (as originally) and it prevents against updates that might reverse back to the problem.

Temporary Workaround: Install a Specific AWS CLI Version via Snap

Uninstall the current AWS CLI (Snap)

sudo snap remove aws-cli Install AWS CLI version 2.22.35 (revision 1148)

snap install aws-cli --classic --revision=1148 (Revision 1148 corresponds to 2.22.35.)

Prevent Automatic Upgrades

snap refresh --hold aws-cli This will prevent Snap from automatically upgrading AWS CLI to a newer (potentially broken) version.

Once you’ve completed these steps, you can manually run a backup through Forge to confirm that the issue is resolved.

Please or to participate in this conversation.