DevOps needs no introduction - It will not be an exaggeration if DevOps would be titled one of the most disruptive software delivery practices. It drives the cultural change in how individuals react and work together towards a common goal. It also bridges the gap between Business and IT through Automation and promotes Continuous Delivery. At Qentelli, we see DevOps as a holistic framework enabling Continuous Delivery of Value for businesses looking to expedite digital transformation. We believe that that the best way to start this journey is by building a good DevOps Pipeline. In this article we understand what a DevOps Pipeline is and how to build one using Azure.
DevOps Pipeline
A DevOps pipeline is a set of tools and automated processes utilized by the software engineering team to compile, build, and deploy code. Building an effective DevOps pipeline enables companies to rapidly develop, test, and deploy new code on an ongoing basis.
DevOps enables continuous delivery of features adding value to a working product. Continuous Delivery held this line of thought and teams achieve this functionally using a Deployment Pipeline. A DevOps Pipeline is a set of automated processes where all users (Business, Dev, and Ops) work together to continuously develop and deploy their code to production.
Imagine a continuously rotating mechanical belt from the start of the software development till the end. As this belt keeps on moving, different developers working on different requirements push their codes on to the belt. These code pieces get assembled as they move further. Various tools take care of build compilation, testing, and security integrations. Continuous is an other differentiated characteristic of a DevOps pipeline. Once every code piece is assembled, tested, approved, and merged, they are shipped as the final product. As this continuous belt is running from start to end, there is also a reverse belt or continuous feedback loop attached to it. This belt takes any defect or bugs back to its origin point for resolution and once resolved it is back on the main belt. This helps in building a continuous pipeline of defect-free products aka Continuous Delivery of Value.
Phases of a DevOps Pipeline
Since there isn’t one standard DevOps pipeline, an organization’s design and implementation of a DevOps pipeline depends on its technology stack, a DevOps engineer’s level of experience, budget, and more. A more common framework is a eight phased DevOps Pipeline -
With this framework, we aim to provide guidelines on building a basic DevOps pipeline. Our DevOps consultants tailor this framework for global clientele with varied team sizes and industries.
1. Plan – The planning phase includes incremental planning for the next set of iterations or sprints or releases. Scrum and XP are two popular Agile methodologies with a focus on planning and adaptation based on real-time feedback. Deciding on either of these approaches depends on individuals, organizational structure, and teams’ release cadence. With regular stand-ups, backlogs, and visualization metrics such as burnup / burndown charts, they provide end-to-end visibility for teams.
Both these approaches have groundswell and help in planning and delivering in micro-increments without compromising on quality. They can cut meantime to roadmap planning and adaptation from weeks and months to days and hours. Project Management and Tracking tools help in managing progress, tickets, issues, feedbacks, and milestones etc.
2. Code – The next phase is Coding where Developers pick the user stories that they will be working on in the current Sprint / Iteration. With uniformity in toolchains, maintaining a consistent set of tools and frameworks, makes it easy for developers to develop consistent code-styling with no or minimal code smell issues and code anti-patterns.
DevOps teams adopt IaC (Infrastructure as Code) process to rapidly provision infrastructure resources. Developers take care of provisioning servers and testing environments using code. This removes the infrastructure dependencies on system administrators leading to testing codes in a production-like environment resulting in much better quality. Environment consistency makes security explicit and saves time on satisfying policies for environment provisioning.
3. Build – The Build phase includes committing code into a code repository. Developers create a pull request to commit their piece of code. As a pull-request is submitted – peer-review is done and after approval, the code moves to the next phase of automated build testing. The peer-review consumes less time and helps in identifying issues early thereby saving cost and effort.
Once the pull request is approved, a series of automated tests like unit tests kick in. If the build fails, a notification is sent to the developer to investigate the issue. This is to ensure that the developer whose code broke the build can fix it immediately before other developers check their code.
It is highly recommended to check-in code continuously to rerun builds and tests. This ensures the issues are resolved early. If the issues cannot be resolved in the current Iteration / Sprint, then they will be considered as part of Planning phase for the next Iteration / Sprint.
4. Test – As the build passes and moves on to the next stage, it enters the crucial testing phase. The build is deployed to the Staging / Testing environment for extensive testing. This includes performance, load, user acceptance, and security testing. The Staging / Testing environment is configured as a replica of a production environment to understand application performance in the real-time scenario.
Simultaneously, some of the functional and non-functional areas that are prone-to-human error will go through automated testing - such as security scans, infra and compliance, performance, load testing, etc. These tests act as a testbed to ensure everything is running as required. The maturity and level of test automation practices determine the release readiness of the application.
5. Release – The release phase is considered as a climax in the DevOps pipeline. By this phase, every code has passed automated and security check-gates and is ready for deployment.
Based on the Business Goals, organizations decide on the release management approach – Manual aka Continuous Delivery or Automated aka Continuous Deployment. There are various release management tools available in the market to enable multiple releases every day or every week. Continuous Delivery gives organizations and Business specific control over the release schedule so they can decide when to go live with the new functionality.
Release management teams and tools are responsible for:
- Managing interdependencies
- Review, planning, maintenance, and up-gradation of the infrastructure landscape
- Keeping up with release planning and management, process, and tools adherence, sprints, resource utilization, and rollback strategy
- Post-deployment monitoring and feedback, and improving processes
6. Deploy – At this stage, the build is ready to be pushed into production. The production environment can be configured using Infrastructure-as-Code. Organizations can implement processes and policies as per their deployment strategies like blue-green deployment, canary deployment, and others.
As quality and continuous improvement is gaining importance, it is ideal to integrate testing into the deployment phase as well. This allows real-time feedback from customers and faster resolution of issues. If something is broken while in production, a rollback strategy gives the ability for the team to run the last stable build to ensure continuity for their customers while quickly fixing the issues in the latest build.
7. Operate – Once the application has been deployed into the production environment, the Operations team continuously monitors the infrastructure and application performance. Infra and Ops team maintains the optimal scaling of the environment depending on the peaks and troughs.
As the array of infrastructure management services are increasing, the operations team is continuously tasked to ensure maximum uptime and to take care of Security, Pipeline Management, Cloud Resources, Virtualization, and Kubernetes / Docker orchestration.
8. Monitor – Although we have mentioned Monitoring as the last phase of the DevOps pipeline, Monitoring is embedded across all the phases. Due to the enormous amount of data that gets generated across the entire SDLC, it is often possible for the teams to overlook few aspects.
Monitoring gathers various data workflows to improve the DevOps pipeline and build high-quality applications. These teams would be likely to spend fewer hours chasing, triaging, and resolving various vulnerabilities with little prioritization.
Azure DevOps Pipeline
Azure DevOps enables end-to-end DevOps capabilities by providing version control, reporting, requirements management, project management, automated builds, testing, and release management capabilities. It covers the entire application lifecycle. Here is an overview of different Azure DevOps components to help you build a DevOps pipeline –
Azure Boards let teams track workflows using Kanban boards from idea to release and monitor progress throughout the SDLC.
Azure Pipelines automates entire build, testing, and deployment stages to create reliable and consistent pipeline workflows. With support for Microsoft hosted Linux, macOS, and Windows, it is super-easy for teams to manage hardware and VMs.
GitHub and Azure can be seamlessly integrated using the developer’s GitHub account. Developers can move their code repositories from the GitHub account to the deployment phase using their GitHub account. Azure DevOps comes with native support for deployments to Azure Kubernetes Services, Azure Web Apps, Azure SQL Database, Azure Functions, and more.
Azure Test Plans provides an exhaustive test suite to run manual as well as automated testing. As quality becomes paramount, Azure Test Plans provide all the capabilities required for different kinds of testing such as manual testing, user acceptance testing, exploratory testing, and feedback collection from stakeholders.
Azure DevTest Labs feature from Azure DevOps eases the process of infrastructure provisioning. Developers can create the desired environment directly from Continuous Integrations tools using Azure plug-ins or REST API. This simplifies cost management by setting up virtual machines and empowering developers to shut down and start them based on the requirements.
Azure Monitor provides comprehensive insights from Azure and on-premises environments. This helps in storing and managing data from various stages of the DevOps pipeline and apply advanced analytics engine for faster resolution of issues. With visibility across pipelines, it helps in maximizing the performance and availability of all applications.
To improve DevOps pipelines continuously, teams need a granular and focused approach to build pipelines. Here are few tips to keep in mind that will help you build better DevOps pipelines –
- Code Repository is a web application facility with file storage where programmers can store large amounts of source code. There are several code repositories available in the market. For example – GitHub, BitBucket, Azure DevOps, GitLab are some of the widely used code repositories. The first step in the DevOps pipeline is to link the code repository to the pipeline to take further actions.
- Triggers define the DevOps strategy adopted by the team. There are several ways to invoke a build pipeline like Polling SCM, building periodically, build manually, trigger after a build completes, etc. To enable CI, the build pipeline needs to poll the repository continuously on a specific branch to look for new commits. The branching strategy adopted by the team plays a significant role in the DevOps process.
- Choosing a Branching Strategy depends on the size and maturity of the team, rate of development, and discipline. It is always suggestible to maintain only one long-living branch and short-lived branches for feature development or bug fixing. Generally, GitFlow branching is the way to go for teams getting into DevOps transformation. More matured teams can go with Trunk Based Development.
- Package Management help in handling application dependencies. Applications rely on libraries/packages to make use of existing programs to ease development. The build pipeline might need an additional step to download all those dependencies for us. Some package management tools include npm, NuGet, Maven, Homebrew, Chocolatey, etc.
- Compiling code is the stage where the human-readable source code gets translated into executable machine code. Depending on the application, the build task converts the code into compiled binaries in various formats.
- Tests are an integral part of the DevOps pipeline. These can be unit tests or integration tests. It is advisable to run the unit tests in the build pipeline, measure the code coverage, and report the results back in every pipeline. There are several unit testing frameworks to write tests and tasks in the pipelines to run them and measure the code coverage.
- Static code analysis is an optional step in the DevOps pipeline to analyze static code of any security vulnerabilities, duplications, code smells, and to follow a standard coding convention. It is advisable to adapt to this culture to write better code.
- Packaging the application depends on the type of application, the binaries might need packaging. Packaging all the binaries into a zip file or a jar file to favor the deployment.
- Deployment and associated practices are key to long-term success in the DevOps pipeline. In the modern DevOps world, it is recommended to treat the infrastructure as code, especially if that infrastructure is hosted in Cloud. The deployment pipeline should not only deploy code but also maintain a desired state configuration to keep environments in sync and track all the changes.
- Post-deployment activities include running functional tests, monitoring the application, alerting, and feedback. It is vital to have automated functional tests post-deployment to track the behavior of the app, to catch the bugs early in the lifecycle, to capture server metrics, and to provide feedback to the teams.
DevOps Pipeline is your strategic asset
Building the DevOps pipeline does not have to become a slave to rigid, guidelines-based documentation. It should give the opportunity and power to create a higher standard of continuous delivery practices. As the industry moves towards a DevOps future with AI-ML-driven advanced pipelines, they can use DevOps pipelines as a strategic asset. Have a DevOps project in Mind? Let our DevOps solutions team connect with you – info@qentelli.com
