CI/CD Pipeline Innovations

Project teams fill in a mapping file (Jenkinsfile) to specify their application’s structure and testing requirements and our CICD Template Framework automates the pipeline onboarding process. This can save each application team more than 200 hours per year. For example, we onboarded a mission-critical application containing 740,000 lines of Java code to the CICD pipeline in less than 4 hours. Citizant has used the Template Framework on more than 100 applications running on a variety of hosting platforms.

The CICD Template Framework is highly scalable, extensible, and portable. We seamlessly ported our CICD Template Pipeline for one federal agency to an AWS Gov Cloud environment hosted by another agency. We executed all stages of the CICD pipeline to deploy a sample application with more than 200 rulesets in under 2 hours. Previously, standing up an infrastructure to perform all pipeline stages would have taken months.

Citizant designed and developed an innovative Configuration as Code (CAC) framework that allows all CICD project template pipelines to be stored in the IRS Enterprise GitHub instance as Configuration Items (CIs). Storing CI/CD templates as first-level artifacts provides the benefits of version control, traceability, security, maintainability, simple installation in new environments, and quick restoration of the pipelines during a disaster recovery effort.

To give developers self-sufficiency when managing application deployments on containers, Citizant developed a Container Template Framework for CI/CD pipelines to automate build and deploy on OpenShift v3 and v4 platforms using Dockerfile. This framework allows building pods and containers on the fly with auto-scaling of applications during peak events. We have also integrated the RedHat Application Migration Toolkit (RHAMT) on more than 100 application pipelines to scan the application code for container-readiness issues and generated a readiness report each time the application is built. The average time from a developer committing code to deploying an application on secure containers is about 8 minutes (excluding management approval), after passing the defined “Definition of Done” (DoD) thresholds on each pipeline stage.

To enhance the CI/CD Pipeline, Citizant developed a configuration-driven framework to automatically generate all required AppScan security files each time the project is built. Developers use the AppScan analysis results to identify and resolve all security issues before deployment, preventing costly re-work late in the development life cycle. Our DevOps engineers also designed the CI/CD pipeline such that all project JUnit and automated tests are executed against the latest version of the code every time a project’s pipeline is run. This provides the developers and the Enterprise Testing organization with rapid feedback on the code’s current state. One agency saved 97-98% of testing time on multiple enterprise applications compared to manual testing.

Citizant’s typical pipeline includes different levels of testing (unit, functional, regression, performance, integration) performed throughout the CI/CD pipeline and are configurable to fail. Among the tools we have integrated into our pipeline are Selenium, Rational Functional Tester, Rational Quality Manager, SoapUI/Ready API, and Rational Performance Tester. As a benchmark, one application running on our pipeline has 14,000 JUnit tests that run in less than 20 minutes. Through automated testing, we have saved 12 applications a total of 360 hours for each run through the pipeline. Manual testing that previously took 15 hours now takes less than 10 seconds – with no human intervention. The pipeline can auto-scale to multiple pods for containerized applications to run tests on-demand.

Citizant’s DevSecOps engineers used Infrastructure as Code (IaC) methods to integrate Ansible Tower and Ansible Core with the CI/CD pipeline. On AWS GovCloud, the pipeline can automatically spin up a RedHat Enterprise Linux 7/8 Virtual Machine (VM) or Windows Server 2019 virtual machine by executing automated Ansible playbooks without manual intervention. With the Sysdig security platform integrated with the pipeline, we can automatically run our Security Patches Playbook to ensure these environments conform to Standard Baseline Configuration Guidelines. This end-to-end operation takes about seven minutes. Instead of patching a runtime server and creating a snowflake instance, now we can cut maintenance costs and time by applying patches to the image and making it available in the customer’s server marketplace. This automated IaC approach reduced server provisioning time from 34 days to only minutes.

By provisioning the hosting infrastructure automatically with IaC, our DevOps engineers demonstrated the end-to-end running of an enterprise application on the pipeline – from code commit and new VM build to deployment and execution of test cases, including destruction of the newly created VM – in under 22 minutes.

Citizant developed a Reusable Microservices Pipeline Framework (RMPF) to automate the onboarding of microservices applications to the CI/CD pipeline. RFPF features include auto scaling, container-readiness scanning with RHAMT, and security scanning. RMPF enabled a large enterprise application to self-onboard 45 microservices to the pipeline without impact to other services. The average time from code commit to deployment is about 8 minutes.

Citizant’s CI/CD pipeline is integrated with Platform as a Service (PaaS) tools – OpenShift 4 with Podman, Kubernetes, and Ansible. This helps reduce costs by lowering server footprint, removing snowflakes servers (reduce technical debt), and mitigating environment drifts. As a data point, we consolidated 3,000 servers to 180 containerized servers. At a cost of roughly $1,500 per server, our container deployment strategy resulted in a potential aggregate savings of more than $4 million.

We have also evaluated the Department of Defense’s Iron Bank for obtaining DevSecOps tool images. We also use the Red Hat Application Migration Toolkit (RHMTA) for generating the container readiness reports for more than 100 applications The RHAMT report allows the project team to be container ready by providing guidance on conforming to “12 Factor App” industry standard. Our pipeline supports container deployment for Spring Boot and JBoss applications using OpenShift. We have performed 12 production container deployments of the IRS Tax Litigation Counsel Automated Tracking System (TLCATS). At least seven other applications are containerized and deployed on OpenShift in non-production clusters. We have enabled logging features by integrating with Splunk for event and stream analysis. We have also implemented side-car container patterns for logging, single sign-on, and the OpenShift template for SiteMinder, a tool for enterprise Web access management.

Compliance as Code is baked into Citizant’s CI/CD pipeline. Each stage in the pipeline has gated compliance thresholds. If any stage does not pass the defined threshold, the pipeline stops and sends notifications for remedial action. These thresholds are established by working with various stakeholders such as Enterprise Architecture, Cybersecurity, and operations teams. One customer’s Cybersecurity team has created a Secure Code Matrix (SCM)-CMMI level for each application, which is baked into our pipeline. All metrics generated by the pipeline are tracked and reported for actionable outcomes. Even the pipeline itself is subject to Change Control Board (CCB) acceptance. We developed an automated tool to review compliance against the Definition of Done for a pipeline, which feeds into the CCB.

Before we implemented our CI/CD pipeline at one agency, a development team would manually submit their source code to CISO team for scanning when the app was ready for production. The agency typically took 2-3 weeks to generate a security report, with additional expensive delays for issue resolution or executive waivers. To resolve this challenge, we collaborated with project teams and the CISO team to integrate the CISO’s preferred tool into our pipeline. We iterated frequently until we could demonstrate that the report generated from our pipeline matched the report generated manually by the CISO team. As a result of this collaboration, the CISO trusted the pipeline security report and would approve code for production deployment based on this report.