rawilk's avatar
Level 47

Zero-Downtime Deployment Review

If anyone out there is familiar with zero-downtime deployments and would be interested in reviewing/critiquing my attempt with it on my personal site repository, I'd greatly appreciate it. I know I could just use a service like Envoyer, however I wanted to tinker around with it and learn more about zero-downtime deployments.

I think I have my deployment process working well now, but I'm just curious if anyone with experience with this kind of thing has suggestions on things that could be improved either for security or performance.

For the deployment process, it starts out with a GitHub action that builds a new .env file from secrets in my 1Password vault and then ssh's that file into the server. The action also makes some other preparations on the remote server as well before triggering a deployment through forge.

Once the forge deployment starts, it provides the necessary variables to my deploy.sh script that I send to server and then executes it.

Here is the GitHub action workflow for the deployment: https://github.com/rawilk/randallwilk.dev/blob/main/.github/workflows/prod-deploy.yml

Here is the deployment script that runs on the server: https://github.com/rawilk/randallwilk.dev/blob/main/bin/deploy.sh

0 likes
3 replies
LaryAI's avatar
Level 58

To achieve zero-downtime deployments, it's crucial to ensure that your deployment process is seamless and doesn't interrupt the user experience. Here are some suggestions and best practices you might consider to improve your deployment process:

  1. Atomic Deployment:

    • Ensure that your deployment process is atomic. This means that the new version of your application should be fully prepared before switching over from the old version. You can achieve this by using symlinks to point to the current release directory.
  2. Use a Release Directory Structure:

    • Organize your deployment directory with a structure like:
      /var/www/
        ├── releases/
        │   ├── 20231010120000/
        │   ├── 20231010130000/
        ├── current -> /var/www/releases/20231010130000
        ├── shared/
      
    • Each deployment creates a new timestamped directory under releases/, and the current symlink points to the latest release.
  3. Database Migrations:

    • Run database migrations in a way that they don't lock tables for a long time. Consider using tools or techniques that allow for non-blocking migrations.
  4. Environment Variables:

    • Ensure that your .env file is securely transferred and only accessible by the necessary processes. Consider using environment variable management tools or services.
  5. Rollback Strategy:

    • Implement a rollback strategy in case something goes wrong. This can be as simple as changing the current symlink back to the previous release.
  6. Health Checks:

    • Before switching to the new release, perform health checks to ensure the application is running correctly.
  7. Logging and Monitoring:

    • Implement logging and monitoring to quickly identify and resolve any issues that arise during or after deployment.
  8. Testing:

    • Automate tests to run as part of your deployment process to catch any issues before they reach production.
  9. Security:

    • Ensure that your deployment scripts and processes are secure. Limit SSH access and use key-based authentication.
  10. Performance:

    • Optimize your build and deployment scripts to reduce downtime. Consider caching dependencies and assets.

Here's a basic example of how you might structure a deploy.sh script to use symlinks for zero-downtime deployment:

This script is a simplified example and should be adapted to fit your specific needs and environment. Always test your deployment process in a staging environment before applying it to production.

Please or to participate in this conversation.