PipeCD for CI/CD

Conductor

In recent years, Linux containers have revolutionized the way developers and administrators roll out and operate applications. They usually come with Kubernetes in tow, which basically serves up a virtual data center, providing dynamic functions such as network and storage, which means workloads can be orchestrated by a fleet of systems.

However, Kubernetes (K8s) is known for its complexity. The symphony of pods, services, namespaces, and volumes alone is enough to drive many a K8s newcomer to distraction. When you add custom resource definitions and endless YAML files, your frustration meter can peak out.

An example quickly illustrates the complexities in rolling out applications in Kubernetes. To package a container with an application in a pod, add some persistent storage, and make the whole thing accessible from the outside, you need at least four independent Kubernetes resources. Seasoned K8s jockeys will be aware of this, but not those who have just begun their own container orchestration journey.

A few tools make life easier for both newcomers and experienced administrators, such as package managers Helm and ArgoCD [1]. ArgoCD not only promises to roll out applications in Kubernetes, but also offers comprehensive lifecycle management. It (Figure 1) is considered the standard solution for continuous integration and continuous development (CI/CD) with Kubernetes, but it is unpopular with many developers (see the "ArgoCD" box).

ArgoCD alternatives have long been available. FluxCD [6], which is far leaner than ArgoCD but can easily compete with it in terms of functionality, has gained some notoriety. However, FluxCD also lacks key features, such as the ability to use platforms other than Kubernetes. PipeCD is an alternative.

PipeCD

PipeCD [7] has made a name for itself in recent months because the software claims to offer pipelines for CI/CD, in some cases with a far wider range of functions than its peers, while still being lightweight and easy to use.

At first glance, the PipeCD architecture (Figure 2) appears even more complex than that of ArgoCD, which comprises just three internal components. PipeCD, on the other hand, is more extensive: In addition to the central intelligence – the PipeCD daemon – the PipeCD control plane comprises a data store, a file store, a cache, and a management interface named pipecd-ops.

PipeCD does not implement the data storage itself. Instead, it relies on MySQL to store its metadata and on Amazon Simple Storage Service (S3) to store files. If so desired, it can also roll out MySQL and MinIO for S3 itself. In general, setting up PipeCD in a Kubernetes cluster is easy to manage. The developers do supply ready-made templates for Kubernetes, and they are easy to apply with kubectl apply. Most of the services required for PipeCD are then already running in the cluster.

However, you still need an instance of piped, which should not be confused with the PipeCD daemon itself. Instead, it is a kind of agent that waits for PipeCD to assign tasks, which it then processes. In the Kubernetes context, it results in interesting effects. Initially, you will typically only roll out the main services of PipeCD, but not the piped instances for your developers; they have to take care of this themselves.

Moreover, the PipeCD architecture means that piped instances do not necessarily have to be Kubernetes-specific. Accordingly, one central argument by the PipeCD developers in favor of their solution is that, unlike ArgoCD, PipeCD is not Kubernetes-centric. Instead, piped can be installed on a virtual instance on the Google Compute Platform or run as a standalone service on a legacy virtual machine (VM).

The tool also supports Terraform, which further increases the choice of potential target platforms, because deployment in AWS is also possible through PipeCD, right down to special resources such as EKS. Where ArgoCD is only capable of communicating with Kubernetes, PipeCD is far more versatile.

Gitless GitOps and GitOps

Another key difference between ArgoCD and PipeCD is that PipeCD is explicitly intended for the use of GitOps and Gitless GitOps scenarios. The term "GitOps" has established in agile IT the principle of maintaining the development status of individual applications or environments in Git and rolling them out automatically directly from Git. If a developer or administrator changes something in Git, the CI system reacts and automatically adjusts the current deployment with as little interruption to service as possible. The current version of the software, including the configuration, is then identical to the data stored in Git.

However, ArgoCD is relatively static with regard to the integration of Git directories. A single service ensures that changes in connected Git directories do not go unnoticed. However, changes to the deployment are made by another component within ArgoCD, which the administrator or developer executing the deployment must operate separately.

PipeCD is more universal: Once a Git directory is connected, the PipeCD daemon monitors it itself. Changes implicitly cause the pipeline defined for the respective directory to start, and the necessary changes then find their way into production.

PipeCD not only supports GitOps [8], but also Gitless GitOps [9]. At first, this distinction seems like splitting hairs, because the main difference between Gitless GitOps and GitOps is that Gitless GitOps does not directly access the original Git directory, but only artifacts resulting from changes in Git (e.g., updated container images). In terms of day-to-day handling, however, administrators and developers see a huge difference.

If you want to connect a solution such as ArgoCD to a Git directory, you will inevitably have to handle credentials and user management for the Git solution, which harbors security risks. On top of this, you need Git integration for the CI/CD application, which again complicates handling. If, instead, you have the option of automatically creating new artifacts from GitLab or GitHub when changes are made and uploading them to a private or public directory, CI/CD only needs to monitor the artifact directory.

In Gitless GitOps, the Git directories do not need to be accessed for development, which means you only need to deal with access data for accessing the artifacts. Accordingly, intruders cannot reach the Git directory even if they manage to break in. Conveniently, PipeCD also has its own credential management system. Credentials do not appear in the clear anywhere, can be referenced by variables in pipelines, and are stored securely in an internal vault in PipeCD.

Getting Started

After you have installed the PipeCD services in the cluster, first set up piped by making the main PipeCD server accessible from outside with a service definition in Kubernetes or a suitable equivalent (e.g., a load balancer). A sample project (Figure 3) is created automatically, and you can expand it if you wish.

Additionally, PipeCD is multiclient capable and can handle user authentication over LDAP or single sign-on. A PipeCD daemon instance therefore can easily accommodate many users and their projects. However, each project requires its own piped instance, which is comparatively easy to set up; the PipeCD GUI has a separate page for available piped instances. You just need to create a new entry there and field the temporary project ID and key for logging piped into PipeCD.

The following step is a little more complicated because it depends on where you want the instance to run. If it is on Kubernetes, PipeCD at least comes with a ready-made resource definition. Replace the project ID and the login key for piped with values from the web interface; then, follow up by running

kubectl -f apply

against the file. A little later the piped instance (Figure 4) appears in PipeCD.

You can now manage the instance without another visit to the command line and, if needed, update it at the push of a button. If you want to move the instance elsewhere, the steps differ, but some things remain the same. You will need to adjust the project ID and the key for registering piped in the target instance.

As soon as an executable piped exists, most PipeCD functions are available to the user: Pipelines can be created, Git directories can be integrated into them, and deployment strategies can be rolled out. All of these tasks for Kubernetes setups are handled by Kubernetes resource definitions.

Smart Implementation

Now the real work with PipeCD begins. The PipeCD developers attach great importance to opening up as many options as possible. The documentation, for example, refers extensively to blue/green and canary deployments. If you don't regularly develop applications yourself, you won't be familiar with these terms (see the "Blue/Green and Canaries" box). For the time being, it is enough to know that the terms refer to the way in which a CI/CD solution rolls out updated configurations and updated applications. Blue/green and canary deployments [10] are two different approaches.

Blue/Green and Canaries

In deployments that are based on the blue/green principle, two versions of an environment are basically always running. One is the current production environment and the other is the newer version that developers and admins are currently preparing.

PipeCD supports the deployment scenario out of the box. Once all the work and necessary tests have been completed, the new version switches to live operation in PipeCD and the old version is kept as a fallback. Thanks to the flexibility offered by Kubernetes, this arrangement also works with just a few command-line commands.

If the new version of the software turns out to be error-free, the developer responsible replaces the previous environment with a clone of the application that is then in production. At the same time, they then have a new starting point for ongoing development work. If something is still not running as desired, a rollback (Figure 5) is quickly initiated.

Canary deployments take a different approach. Although two versions of the software are also present: The old and new run in parallel in live operation. The developer uses the tools of the hosting platform to ensure that the new features only reach a small proportion of users called "the canaries," or involuntary guinea pigs, if you prefer. If errors or problems occur, it is assumed that one of the canary users will complain. The problem would then be solved by updating the canary version. This process is repeated until the new version works perfectly for the canary users.

The changeover then takes place so that the new version is rolled out to all users. If no further problems occur, the legacy environment is decommissioned and the cycle starts all over again. If you encounter any problems after the changeover, you roll back and the improvements are implemented again, and again initially for the canary group.

Neither of the two approaches can be described as better or worse than the other across the board. The personal preferences of admins and developers define what happens. As I mentioned earlier, PipeCD fully supports both models. You just need to provide the pipeline for an application with matching parameters. PipeCD takes care of the rest automatically on the basis of the connected piped instances.

As with most administrators, you have probably experienced first hand that things can sometimes go wrong when you update software. Classic deployment approaches are based on long-term support software and assume that the production environment practically does not change at all during operation. Updates are only available as part of long-announced periods of downtime, during which the service is not available.

Before a software update is carried out in the scope of such a maintenance window, the development team puts the new version of a tool through its paces. In recent years, however, this model has proven to be partly disfunctional and massively slows down development. Additionally, large monolithic updates mean that errors and bugs cannot be immediately traced to a clear root cause. Testing does nothing to change this situation. If, after a major update has been installed, it turns out that a basic function no longer works, it will, at the very least, trigger a frantic search for the bug. At worst, you will be facing a rollback to the old version, but often enough, a rollback is impossible.

Agile software development has therefore adopted the motto "release early, release often" as its guiding principle. People who incrementally release new versions of their software with small changes will quickly notice whether something is not working and be able to intervene immediately. The disadvantage is that the production environment is constantly changing. CI/CD tools are primarily used to manage and simplify the changes.

Knowing What's Going On

Regardless of whether developers opt for blue/green deployments or the canary approach, they need to monitor the new software version meticulously after the rollout. In a CI/CD environment, monitoring is only possible if it is closely interlinked with the deployment tool. The PipeCD developers are aware of this point, which is why they equipped their solution with trending and analysis tools. These tools cover almost every aspect of monitoring, alerting, and trending.

In the case of applications with a REST interface, for example, PipeCD monitors incoming and outgoing requests when instructed to do so, measuring their duration from start to finish and recording their scope. If the time per request were to increase massively after the deployment of a new application version, the developers would see this immediately in PipeCD.

The software also meticulously monitors the availability of the services rolled out with PipeCD. If a service goes offline after an update, PipeCD sounds an alert. It also collects the log messages of a deployment. For example, it can glean the data from the Kubernetes logs for the respective pods. Because applications in Kubernetes containers usually log to stdout or stderr, the messages appear in the logs of the containers and ultimately also in the container GUIs thanks to PipeCD's log harvesting.

In the logs, details are always conveniently available at a central location for each deployment. An application's status can be visualized practically in real time. PipeCD also accesses the monitoring data, which it collects automatically according to the configured pipelines. If you take it to the extreme, you can even define criteria for an automatic rollback in PipeCD from the collected metrics data.

Building Artifacts

As mentioned earlier, PipeCD is a solution for Gitless GitOps, which fits in with the fact that artifacts (i.e., the results of an automated build or CI/CD process) can be integrated into a pipeline directly from the application. If so desired, you can work with Git directories or use artifacts in the definition of your own pipeline, which shifts some of the work to be done elsewhere and outside the sphere of influence of PipeCD.

Anyone who builds artifacts directly from GitLab or GitHub, for example, will need to set up their own CI/CD pipelines there. However, the rewards are reliable isolation of Git commits and the software running in production, significantly higher platform security from eliminating the ties between Git and the deployment platform, and a clearer allocation of the responsibilities of the individual services.

Conclusions: Valid Alternative

Today, ArgoCD is considered to be the standard solution for CI/CD with Kubernetes. However, being the king of the hill does not automatically mean offering the best solutions for typical issues in container application development. Anyone who is not completely happy with the de facto standard will find that PipeCD is a valid alternative – for many reasons: The tool is far leaner and has less overhead than ArgoCD and it uses and combines tools in numerous places instead of demolishing some of their functionality and replacing it with its own inventions.

PipeCD is ideal for workloads that rely on security, comprehensive metrics data and the choice between GitOps and Gitless GitOps. Unlike its competitors ArgoCD and Flux, it also supports target platforms other than Kubernetes, although K8s is and will remain the typical environment for PipeCD. In terms of multifunctionality, it beats the direct competition by a country mile. Anyone who is planning or implementing a CI/CD environment will definitely want PipeCD on their radar as an alternative to ArgoCD and the like.