Skip to main content

Building continuous delivery pipelines in Hybrid Clouds




Building Continuous Delivery Pipelines


Complex release management and multiple release pipelines in a dynamic product development environment catering to high demand of products in short interval of time requires an hybrid cloud approach whereby you can use the elasticity of vast public cloud as and when required and top it off with abundant use of container world to address short lived releases in CI/CD environment providing self healing builds with easy reruns on failures. The new approach of addressing the demands of startup or enterprise companies scaling sudden exponential growths requires throttling release pipelines from your onprem datacenter or private cloud to public cloud without any upfront cost of planning data center requirements every quarter and staggering the production. The first step is to understand How pipelines work in most of the current implementations.

Set up a continuous integration system to automatically test new changes to a repository











CI/CD EcoSystems:-


Popular CI/CD deployment Infrastructures

  • Jenkins
  • BuildBot
  • GoCD


Using Jenkins pipeline for Continuous delivery - Java Runtime

 


 Continuous Integration
Continuous Integration is a development practice that requires developers to integrate code into a shared repository at regular intervals.

 Jenkinsfile (Declarative Pipeline)

pipeline {
    agent any

   stages {
        stage('Build') {
           steps {
               sh 'make'
           }
        }
        stage('Test'){
            steps {
                sh 'make check'
                junit 'reports/**/*.xml'
           }
        }
        stage('Deploy') {
            steps {
                sh 'make publish'
            }
        }
    }
}






The Pipeline workflow


  • Individual Development and unit tests
  • Personal multi-node, package based OpenStack env  for dev and validation
  • Push to git/gerrit for review and automated testing (tempest)
  • Pull in deploy-kit
  • Deploy-kit tooling builds deploy artifacts
  • Auto deploy to public cloud test env
  • Deploy to physical staging env -->production 
  • Gerrit for reviews and submit changes
  • Jenkins test pipeline
  • Jenkins Job builder - create merge, etc
  • Jenkins uploads release artifacts and deploy unit/validation  test


BuildBot -Open Source Continuous Delivery


Buildbot supports not just continuous-integration testing, but automation of complex build systems, application deployment, and management of sophisticated software-release processes.



BuildBot pipeline  - Python based





BuildBot Dashboard - Integrations



BuildBot uses open source product integrations to H/W resources, Test Suites/Builders, Build Reporting  


BuildBot Pipeline Use Case


Product :- SDN and SDS based SDDC software

  • 4 Build pipelines - Master, RC, Released dot Build, Related patch builds
  • 3 hours of Build Verification tests run with every check-in on any pipeline
  • 30 Stable Builders run nightly on 4 pipelines
  • Long Running Builders weekly 3 times on all pipelines
  • H/W and Virtual Builder resources shared by all pipeline

Deploying CI/CD pipeline for a family of product releases - Challenges


  • Scalability - Build infrastructure, Number of pipelines, Number of users

  • Resource Contention - Same physical hardware and Virtual environment being consumed and repurposed causing loss of release time in triaging issues not related to product

  • DevOps - System Maintenance Installs Upgrades

1.) Scalability of build infrastructure as with increase/decrease number of people check-in code and run unit tests in CI platform. This results in contention for virtual and hardware resources in data center, loss of time by engineers investigating failures or triaging defects related to failing CI/CD infrastructure problems
2.) Queuing up check-in caused by number of builders waiting to be deployed and this causes loss of time in terms of product delivery. Can 3-month release cycle in agile process be reduced to on demand continuous delivery?
3.) Constant patching by DevOps on infrastructure to keep the CI/CD pipelines up and running with no mtbf.  Build infrastructures and pipelines are tightly coupled with the amount of resources availability in data center. Introduction of a new product release or feature release to existing causes DevOps to panic and 50% of the time due to inability to measure things like % of data center utilized? How much is the capacity of data center on which we are trying to run the CI/CD pipelines? How many pipelines can my datacenter support to deliver multiple products for continuous delivery?



What do I need for my pipelines to work ?




1.)  Elasticity of public cloud in my data center – Move to Hybrid virtual data center
2.)  Being able to extend my pipelines or CI/CD environment from private cloud to public cloud on demand – Hybrid Cloud
3.)  Ability to control cost of datacenter by offloading to public cloud and pay as we use model
4.)  How seamless is the deployment of CI/CD build infrastructure to this new virtual Hybrid datacenter – Does it truly offer the value and ROI?

Comments

Post a Comment