Skip to main content

BuildPiper Architecture

Here's a brief explanation and overview of BuildPiper architecture and its basic components. These components work together to support product features and functionalities.

This is the BuildPiper Architecture Diagram displaying all the critical components that enable BuildPiper to manage Kubernetes & Microservices application delivery.

Docs Version Dropdown

BP Gateway#

BP Gateway is the main entry point of the platform. It acts as a Gateway within the product. It efficiently distributes the incoming network traffic across a group of servers supporting the platform. Once the traffic crosses the BP gateway, it then either reaches the Frontend (BuildPiper UI) or BPCTL.

Frontend (BuildPiper UI)#

This is a feature-rich, easy-to-use, highly intuitive user interface to leverage product functionalities. BuidPiper can be configured and configured Jobs can be triggered via the state-of-the-art user interface.

BPCTL (CLI)#

BuildPiper provides a CLI( Command Line Interface) to interact with BuildPiper’s Platform API. This CLI is called BPCTL. BPCTL works via Manifest files, which can be applied via BPCTL apply command. BPCTL Manifest definition supports a wide variety of kinds which can be leveraged to configure microservices and associated constructs in BuildPiper. Almost anything which can be setup using the BuildPiper UI can also be set up via BPCTL, using the Manifest files. For more details about BPCTL and its capabilities, please refer to the BPCTL documentation.

Sample Manifest files for configuring different aspects of BuildPiper are available and stored in the below GIT Repo link.

https://github.com/OT-BUILDPIPER-MARKETPLACE/bpctl

Platform API#

The core of BuildPiper is the Platform API. It exposes the REST API via which the product can be configured, setup of microservices can be done and Jobs can be triggered. Platform API is the central control plane which interacts with the scheduler engine, BP Agents, Deploy Agents and MySQL database to orchestrate the functionalities of the product.

BuildPiper is an asynchronous system. The Platform API control plane acts as the central orchestrator which accepts requests and puts them into the queues or schedulers or in the database. Post that different consumers like BP Agents, Deploy Agents, Schedulers pick up the asynchronous tasks and perform the assigned tasks. The Platform API also exposes APIs to monitor the progress of tasks which may be initiated or put in a queue.

Redis#

An open-source, in-memory data store used as a cache.

Database#

It is the database store (MySQL).

Deploy API#

It supports the deployment process. It is designed to perform the following tasks,

  • Automatically Generates Manifest files from the values provided in the UI or by BPCTL.
  • Generates Manifest files from Helm template and values files.
  • Provides Support for different deployment methodologies like Rolling, Blue Green, Canary etc.
  • Empowers team with the ability to perform Rollout Validation.
  • Give detailed insights into Deployment Analytics and Deployment Visualization.

Scheduler#

It helps to schedule all the asynchronous tasks performed by the product.

Shared File System#

All data regarding the build, deployment, monitoring and much more is stored in the shared file system. Since the whole system is auto scalable, keeping all the platform elements in sync with each other is important. For this, all components within the BuildPiper architecture like the Platform API, Deploy API, BP Agent and so on are kept in sync through the shared information via the shared file system.

Job Template and Step Catalogue#

BuildPiper is an out-of-the-box and ready-to-use product. It still allows complete flexibility to re-configure the behaviour and functionalities of the product. This is possible via the Job Template and Step Catalog framework of BuildPiper.

Job Template is a group of Jobs which can be performed by BuildPiper. Every job is in turn a combination of a series of steps.

A Step is a modular piece of functionality which can be done in a docker container or a VM. The product comes with a large number of already configured steps which can be leveraged to perform jobs. Clone code, send slack message, pre-hook, JIRA integration are a few examples of steps that the product offers. New steps can be added into the product and then leveraged in job templates.

The ability to alter the behaviour of Jobs as per microservice or as per language or any other criteria makes it immensely powerful for teams to perform custom checks or execute custom steps for a particular group of microservices.

This framework also makes standardization and governance very easy.

As of today, BuildPiper supports the following Jobs by default,

  • Build
  • Deploy
  • Promote
  • Rollback
  • API call
  • Jira Integration
  • Deploy Config Maps
  • Deploy Secrets

Docs Version Dropdown

For instance: The diagram above shows a Job Template that contains different Jobs. The Build Job Definition comprises the following steps. This behaviour of Build Job can be easily altered by adding CI-related checks if required,

  • clone_repo: Cloning of the build details from an already existing template.
  • prehook: Adding custom hooks for the creation of the docker image.
  • build_image: Building the docker image.
  • push_image: Pushing the docker image to the repo.
  • post hook: Adding custom hooks after the creation of the docker image.

BP Agent#

BP Agent is the orchestrator which executes Jobs and steps within the Job Template. BP agent picks the jobs from Queue and then executes them as per the availability of consumers.

BP agent can be setup in autoscalable mode, ensuring that the infra will autoscale if there is additional load on the system and queue depth is increasing. This ensures that the off-peak infra is really low and the system scales only if there is a load on the system.