Presenters
Source
The CI/CD Evolution: From Skype’s Secret Canaries to AI-Powered Pipelines 🚀
Hey tech enthusiasts! Ever wonder how the sausage of software gets made? We’re diving deep into the fascinating world of Continuous Integration and Continuous Delivery (CI/CD) with Rob Arrowsmith, a veteran engineer who’s been shaping this space for over a decade. From his early days at Skype, where they were quietly innovating shipping practices, to his current role at Octopus Deploy, Rob shares hard-earned lessons that illuminate the path forward.
From Yolo to Continuous Deployment: The CI/CD Journey 🛣️
Remember the “Yolo” days of software deployment? Where you’d SSH into production and hope for the best? Rob humorously recalls those times, setting the stage for the evolution of deployment strategies.
The journey typically progresses through these stages:
- Yolo: The wild west of direct production deployments.
- Continuous Integration (CI): Merging code changes frequently into a single branch and running automated tests.
- Continuous Delivery (CD): Ensuring your code is always in a deployable state, with the ability to deploy to production at any time. This involves testing the deployment process itself.
- Continuous Deployment: The ultimate stage where code changes are automatically shipped to production without human intervention.
The Key Difference: Continuous Delivery means your code is ready to deploy, while Continuous Deployment means it is deployed automatically.
When to Pause: Not all teams are ready for Continuous Deployment. Highly regulated industries or those with complex compliance requirements might need manual approvals or scheduled deployments. The core benefit, however, remains: mitigating risk by catching issues as early as possible.
Kubernetes: The Cloud-Native King, On-Prem Style 👑
Kubernetes has become the de facto standard for container orchestration, but its impact extends far beyond the cloud. Rob highlights a surprising reality: many organizations run Kubernetes on-premises.
- Leveling the Playing Field: Kubernetes, born from Google’s Borg, was released to enable easier workload migration between cloud providers like AWS, Azure, and Google Cloud.
- On-Prem Powerhouse: Despite its “cloud-native” moniker, Kubernetes is widely adopted for on-premise deployments, offering greater control, especially in sectors like finance.
- Declarative Magic: Kubernetes, like tools such as Terraform, embraces a declarative approach. You define the desired state of your infrastructure, and Kubernetes ensures it’s maintained, automatically handling failures and self-healing.
Real-World Adventures: Imagine Kubernetes clusters running on research vessels at sea! Rob shares anecdotes of customers deploying to these unique environments, facing challenges like limited connectivity and the need for robust, offline-capable deployment strategies.
GitOps: Beyond the Git Name 🌐
GitOps, a term coined by Weaveworks, leverages Git as the single source of truth for declarative infrastructure. Its core pillars are:
- Declarative: Defining the desired state of your infrastructure.
- Versioned and Immutable: Storing this state in a way that’s auditable and unchangeable (like Git commits).
- Pull vs. Push: GitOps agents pull the desired state from a repository and apply it to the cluster.
- Continuous Reconciliation: The system continuously ensures the actual state matches the desired state.
The Misconception: While the name suggests a strict reliance on Git, Rob emphasizes that none of these pillars inherently require Git. The core principle is maintaining a versioned, immutable source of truth for your desired state.
The Practical Take: For many teams, the goal isn’t dogmatic adherence to GitOps, but simply shipping software effectively. Rob cautions against the “everything is a nail” approach, advocating for pragmatic solutions that might blend GitOps principles with other, more imperative methods where they make sense.
Progressive Delivery: Rolling Out Safely 📈
Progressive delivery is the evolution of continuous delivery, focusing on releasing changes in controlled, incremental steps to minimize risk.
- Canary Deployments: Releasing a new version to a small subset of users (like the historical practice of using New Zealand as a canary!) to monitor its performance before a full rollout.
- Blue/Green Deployments: Running two identical environments (blue and green), deploying to one, validating it, and then switching traffic to the new version.
- Feature Toggles (Feature Flags): The star player in progressive delivery. These allow you to decouple deploying code from releasing features.
Why Feature Toggles Shine:
- Granular Control: Toggles can control individual lines of code, offering much finer-grained control than deploying entire application versions.
- Precise Targeting: Target specific user segments based on complex rules.
- Instant Rollbacks: Turn features off instantly, in seconds, without redeploying. This is a massive advantage over traditional rollbacks, especially with database schema changes.
The Rollback Reality Check: Rob highlights that true rollbacks are often a myth, especially with stateful systems and database schema changes. The mantra is: roll forward. If something goes wrong, the safest and fastest approach is often to deploy a hotfix rather than attempting to revert to a previous state that might be incompatible.
Feature Toggle Hygiene: While powerful, feature toggles can become a mess if not managed. Rob stresses the importance of cleaning them up once they’re no longer needed to avoid “stale” toggles cluttering the codebase.
The Rise of Platform Teams and AI’s Impact 🤖
The software development landscape is constantly shifting. Rob touches on two significant trends:
- Platform Teams: As organizations scale, platform teams emerge to provide standardized, self-service infrastructure and deployment pipelines. This frees up application teams to focus on building features, rather than managing complex CI/CD infrastructure.
- AI in CI/CD: While still early days, AI is poised to transform CI/CD. Rob anticipates increased velocity due to AI-generated code. However, the focus might shift from pipeline speed to risk reduction associated with AI-generated code. Progressive delivery techniques like feature toggles will become even more critical.
Development Environments: From Local to Ephemeral ☁️
The way developers test and iterate is also evolving:
- Traditional Environments: Dev, Test, Staging, Prod remain common.
- Ephemeral Environments: The exciting trend is the rise of ephemeral environments. These are temporary, on-demand environments spun up for a specific feature branch, allowing for pre-merge testing and collaboration. Once the feature is merged, the environment is torn down. This significantly speeds up the feedback loop.
Octopus Deploy: A Tale of SaaS and On-Prem 🐙
Rob shares insights into building and operating Octopus Deploy’s SaaS offering, highlighting the architectural shift from VMs to Kubernetes and a cell-based “reef” architecture. He also discusses the unique challenges of supporting both SaaS and on-premise versions:
- On-Prem Persistence: A significant portion of their customer base, including banks and governments, prefers on-premise deployments for control and compliance. This necessitates supporting older versions for extended periods, creating complex upgrade paths.
- The Upgrade Gauntlet: It can take months, even years, for on-premise customers to adopt new versions, meaning Octopus must ensure compatibility across a wide range of past releases.
This dual approach, while challenging, caters to a diverse market and underscores that the “SaaS is eating the world” narrative isn’t the whole story.
Key Takeaways for Engineers ✨
- Start Small with Progressive Delivery: Add a single feature toggle to your workflow. Experience the power of immediate control and the “calm” it brings when issues arise.
- Embrace “Roll Forward”: Don’t get fixated on rollbacks. Focus on getting fixes out quickly.
- Clean Up Your Toggles: Feature toggles are powerful, but require discipline to avoid becoming a maintenance burden.
- Kubernetes is Everywhere: Understand its declarative nature and its applicability beyond the cloud.
- GitOps is a Philosophy, Not Just Git: Focus on the principles of declarative, versioned, and reconciled state.
Recommended Reads:
- The Phoenix Project by Gene Kim: A foundational read on DevOps principles and the value of bridging development and operations.
- Radical Candor by Kim Scott: Essential for improving communication and empathy within teams.
- Diaspora or Schismatrix by Greg Egan: For those who love hard sci-fi with a strong dose of mathematics and mind-bending concepts.
The world of CI/CD is constantly evolving, and understanding these trends and best practices is crucial for any engineer aiming to ship better software, faster and more reliably. Happy coding! 👨💻