Deployment Strategies

In this article I want to explore some typical deployment strategies:

• Blue/Green
• Canary
• A/B Tests/Experiments
• In-place

Blue/Green Deployment

Blue/Green Deployment:

• involve deploying the new version of the application alongside the old version, allowing a smooth switch between the two
• running concurrently two production versions but there is only one that gets live traffic
• e.g. blue is the version that’s live and the latest update (new version) is green
• since the blue and green deployments will not be alive at the same time, the latest version does not actually have to be backward compatible
• This is a problem for application dependencies, such as databases, but for them we can also use the blue / green approach
• Since green is not receiving live traffic yet, until moving it live, we have the chance to conduct real testing in a production environment.
• For trivia about the name origin, see Blue–green deployment - Wikipedia or Blue-Green deployment ($1846041) · Snippets · GitLab.

Process:

• Blue Environment: The current live environment (e.g., EC2 instances in an Auto Scaling Group) runs the current version of the application.
• Green Environment: The new version is deployed to a separate environment (e.g., a new set of EC2 instances or a different target group in Elastic Load Balancer).
• Switch: Once the new version is verified and tested in the green environment, traffic is switched from the blue environment to the green environment.
• Rollback: If issues arise, traffic can be switched back to the blue environment, allowing for a quick rollback.

Benefits:

• Zero Downtime: Allows for continuous operation of the application during the deployment since the switch happens between fully operational environments.
• Reduced Risk: Provides an opportunity to test the new version in a live environment before fully committing.
• Easy Rollback: If something goes wrong, reverting to the old version is straightforward by switching traffic back to the blue environment.
• Instant rollout/rollback
• Avoid versioning problems, the entire state of the application is modified in one go

Drawbacks:

• Cost: Higher cost due to the need to maintain two separate environments during the deployment.
• Complexity: More complex to set up and manage, requiring careful coordination of environment management and traffic switching.
• Proper testing of the entire platform should be conducted prior to release to production.
• Handling stateful software can be difficult.

Canary Deployment

Canary Deployment:

• Software release strategy used to reduce risk and validate new features or updates before they are fully rolled out to all users
• Involves releasing the new version of software to a small, controlled subset of users (the "canary" group) before making it available to the entire user base
• Traffic is gradually moved from version A to version B
• Traffic is generally divided on the basis of weight e.g. 90% of requests go to version A, 10% to version B
• We redirect a small amount of user traffic to the new version and the rest to the existing version
• We decide which users will be the first to see the new edition, and we can still change which users we want to have in our test group
• Helps to identify and address potential issues with the new release in a real-world environment while minimizing the impact of any problems that may arise.
• Before a complete launch, we get the chance to test modifications with a portion of the real traffic
• By using canary deployments, organizations can enhance their deployment processes, reduce risks, and ensure a smoother transition when introducing new features or updates.

Key Concepts of Canary Deployment:

• Gradual Rollout:
• The new version is initially deployed to a small percentage of users or servers.
• If the canary release performs well, the rollout is progressively expanded to more users or servers.
• Monitoring and Feedback:
• Continuous monitoring of the canary release is performed to track metrics such as performance, error rates, and user feedback.
• This helps to quickly identify any issues or bugs that may have been missed during testing.
• Rollback Capability:
• If significant issues are detected, the deployment can be rolled back to the previous stable version.
• This minimizes the impact on users and allows for quick remediation of problems.
• Risk Mitigation:
• By deploying the new version to a limited user base first, the potential negative impact of any issues is contained.
• This allows for iterative improvements and fixes before a full-scale rollout.

How Canary Deployment Works:

• Preparation:
• Infrastructure: Ensure that your infrastructure supports deploying multiple versions of the application concurrently.
• Testing: Thoroughly test the new version in a staging environment that mirrors the production environment.
• Deploy Canary Release:
• Deploy to Canary Group: Release the new version to a small, controlled group of users or a specific subset of servers.
• Traffic Routing: Use techniques like load balancers or feature flags to direct a portion of the traffic to the canary release.
• Monitor and Analyze:
• Collect Metrics: Monitor application performance, error rates, user feedback, and other key metrics.
• Analyze Results: Compare the canary release metrics with those of the previous version to identify any issues or performance degradation.
• Expand or Rollback:
• Expand Rollout: If the canary release performs well, gradually increase the number of users or servers receiving the new version.
• Rollback: If issues are detected, halt the rollout and roll back to the previous stable version while addressing the problems.
• Full Deployment:
• Once the canary release is verified and issues are resolved, proceed with deploying the new version to the entire user base.

Benefits:

• Early Detection of Issues: 
• Identifies potential problems with the new release before it affects all users.
• Reduced Risk: 
• Limits the impact of deployment issues to a small subset of users.
• Improved User Experience: 
• Allows for gradual introduction of new features, minimizing disruption for end users.
• Iterative Improvement: 
• Provides the opportunity to make incremental improvements based on real-world feedback.Released edition to a subset of users.
• Practical for controlling error rate and efficiency.
• Fast rollback

Drawbacks:

• Slow rollout

Example of Canary Deployment

Let's say you have a web application and you want to deploy a new feature. Here’s how you might use canary deployment:

• Deploy the New Version: Deploy the new feature to 5% of your production servers or users.
• Monitor Performance: Track the performance of the new feature on these servers or users.
• Evaluate Results: If everything is running smoothly and no significant issues are detected, increase the percentage of servers or users receiving the new version.
• Expand Gradually: Continue to monitor and expand the rollout until 100% of users have the new feature.
• Rollback if Needed: If you encounter major issues, revert the deployment and address the problems before retrying.

Tools and Techniques for Canary Deployment

• Load Balancers: Route a percentage of traffic to the canary version.
• Feature Flags: Enable new features for a subset of users without deploying new code.
• Deployment Platforms: Many cloud providers and deployment platforms (e.g., Kubernetes, AWS Elastic Beanstalk, Google Cloud Platform) offer built-in support for canary deployments.

A/B Testing Deployment

A/B Testing Deployment:

• Similar to the release of the canary, but how the traffic is divided is a business decision and is typically more difficult
• Widely used for experiments - e.g. new features, maybe not yet finished
• We identify groups based on user habits, location, age , gender, or other variables that will tell us which version has the most positive impact on sales. We can then determine if A or B is ready for all of your users.
• Here is a list of conditions that can be used to distribute traffic amongst the versions:
• By browser cookie
• Query parameters
• Geolocalisation
• Technology support: browser version, screen size, operating system, etc.
• Language

Benefits:

• Several variants run parallel to this
• Complete control over management of the traffic

Drawbacks:

• Needs a smart load balancer
• Difficult to troubleshoot errors, distributed tracing becomes mandatory for a given session

In-Place Deployments

In-place deployments update the application directly on the existing instances. 

Process:

• CodeDeploy replaces the application code on the existing instances.
• It follows a rolling update strategy, where instances are updated one at a time or in batches.
• During the update, CodeDeploy can stop the application, install the new version, and restart the application.

Advantages:

• Simplicity: Easier to set up and manage since it involves updating the same set of instances.
• Cost: Generally lower cost as it doesn’t require additional infrastructure.

Disadvantages:

• Downtime: Can experience downtime or reduced availability during the deployment if not managed carefully.
• Risk: If a deployment fails, it can affect all instances since they are updated in place.
• Rollbacks: Rollbacks can be more complex, as the system may need to revert the entire set of instances.

Use Case:

• Suitable for applications where minimal downtime is acceptable and where the impact of deployment failures is manageable.

References:

3 Best Deployment Strategies for Your Startup | by Praful Dhabekar | DevOps Dudes | Medium
Whats the best practice for Go deployments for a small startup? : r/golang
Canary Deployment: What It Is and Why It Matters
What we can learn from the CrowdStrike outage | by QAComet | Jul, 2024 | Medium