Sorry, the language specified is not available for this page

    Take your CI/CD to the next level

    January 25, 2018

    How do your business and product teams engage in innovation and experimentation quickly? By having the luxury of making changes fast while not worrying if some new feature will break or will not be successful. We can unlock this agility by eliminating potential points of waste in our development and delivery processes.

    The goal of a good Continuous Integration / Continuous Delivery (CI/CD) process should be to reliably deploy ‘good’ ideas to production quickly (least amount of time possible between code/feature check-in and production deployment). If an idea is ‘good’ or not, can only be truly known once in production, therefore the need to try changes quickly.

    Every CI/CD process contains these four high level phases – Source, Build, Test and Production. Continuous Integration covers the first two, while Continuous Delivery covers all the phases and creates a deployable artifact, albeit with a manual control gate before production. Continuous Deployment automates that final gate as well.

    Here are some practical guidelines, best practices and learnings we have seen while trying to move our CI/CD processes to the next level. I have organized them under five categories, as follows.

      1. CI/CD Workflow Infrastructure
        • Automate provisioning of your build/deploy infrastructure and resources – using CFN, Terraform etc.
        • Have a good Test/Deployment Management tool/service – like Jenkins / AWS CodePipeline – to execute and visualize the various stages. This will help provide clarity to the ‘tribal knowledge’ as it has the following benefits
          • Models the pipeline through a visual workflow, spreads the knowledge and simplifies understanding of your deployment processes end-to-end
          • Smoothens the operation, by providing ways to incorporate “stop the line” capabilities, manual approval steps between stages etc.
        • Optimize for low spend i.e. pay as little as you can
          • Use spot instances for building, or
          • Leverage pay as you go managed service like AWS CodeBuild, that scales with your needs
      2. Build Phase
        • Centrally manage app configuration files
          • Move your ‘’ or similar application config file from your deployment package to a central repository like S3
        • Manage your secrets by moving them out of config files and placing them in a encrypted secret store. Ensure the store has good permissions security and also highly available
        • Pin external dependencies to specific versions – i.e. not using the “Latest” tag. This will help provide predictability for your builds
        • Package only runtime requirements for production, for example, not build tools, just the application and what’s required to run the executable
        • Optimize for speed
          • layer caching or extracting a base image
          • co-locate build with artifact repo
      3. Testing
        • Get good at managing the multiple environments before the prod environment. You might want to consider a separate environment each for Beta, Integration, Security/Load Testing etc. Simplify this with automation and incorporation into the CI/CD pipeline stages
        • Automate testing in a Beta or Test environments
          • Canary testing – initially some simple URL based testing and move on to advanced acceptance testing
          • Load testing using a home-grown service or something like BlazeMeter
          • Security testing in all phases - Build, Image creation and hardening, Infrastructure creation, Integration and finally Regular operations’ phases. For more details see my blog on InformationWeek about security in DevOps.
      4. Deployment
        • Get creative in your deployments. Choose an option that fits your business needs and yes, definitely automate
          • In-place
            • Doubling the size of your cluster - if your app/cost permit it
            • Rolling – if we don’t want to grow above 100% ever, but can shrink during deployment
          • Canary – a subset of requests goes to the new revision – helps monitor metrics before full deployment
          • Blue/Green
            • with DNS Swap – building parallel infrastructure
            • target group swap using Host header routing
          • Allow updates of individual components. Microservices, anyone?
          • Know if your application works against the same database schema when you release new app code. You would think this is obvious but this is an easy miss for beginning micro-service deployments
          • Consider possibly automating application deployment into a separate region – this can act as your Disaster Recovery as well
          • Boost your confidence levels by integrating automated acceptance testing against any new environments or a canary before cutover
      5. Monitoring & Operations
        • Health checks help prevent many a failed deploys. Use them to your advantage
        • ensure solid notifications for your CI/CD process – for example, CI/CD process changes like code commits, status changes for builds, tests and deployments with corresponding results
        • try to
        • automate your operations wherever possible
        • have governance for compliance with the company’s policies around security, industry regulations etc. – using a tool/service like
          • AWS Config that allows you to assess, audit and evaluate your resources using rules, or
          • service like SunGard AS’s managed offering that provides among other things
            • best practice access management using temporary credentials/roles and
            • intuitive rules based configuration approach for better cloud footprint management
            • learn from your logs and metrics by applying machine learning
            • automate your operations

    Achieving a self-healing, fully automated, secure and quality focused CI/CD state requires a diligent approach. The patterns and practices listed in this article around advanced aspects within automated building, testing, deployment, monitoring and operations will help guide you on next steps.

    Just as a popular meme goes - “Stay curious my friends” - I hope this gives you something to think about on your DevOps maturity journey.

    Other Posts You Might Be Interested In

    How to Move an Enteprise to Continuous Integration: An Intro

    Part one of a three-part series There is an article that has been quoted many times over the years, that originally appeared in The Wall Street Journal in 2011, written by... Learn More

    Finding The Right Fit For Your Container Management Tooling Strategy

    Container technology has been around for more than twenty years, but broad use only took hold in the last couple of years. If you have been following the container space,... Learn More

    Sungard AS Helps Support COVID-19 Research via Folding@home

    No one really knows the exact origin of the phrase “Necessity is the mother of invention,” but it is certainly not a new concept. Regardless of its origin, the concept is... Learn More