At Blend, we’re dedicated to bringing simplicity and transparency to consumer finance. In our six years building a lending platform better suited for the 21st century—one that delivers greater security and transparency and is accessible to all consumers—we’ve grown a lot. Just two years ago our team was barely 100 employees total, but today, we’re at more than 350, including 90 engineers.
Throughout this rapid growth, we’ve relied on a set of core engineering principles, like iterative delivery, to help us learn about what our customers want quickly. In contrast to the traditional slow enterprise release, we’ve adopted an iterative, forward-thinking approach to optimize our speed from conception to delivery by investing in automation and tooling.
The growing need to evolve automation
Our product teams at Blend operate in a fast-paced environment to deliver features into the hands of our enterprise customers continuously. To do this, we rely on Jenkins to manage the release pipeline for our microservices. Many Jenkins jobs are tenant-specific (i.e. two customers, Bank A and Bank B, represent two different tenants in our system), and new services are created regularly, which can lead to a pileup of jobs across multiple Jenkins servers.
As our engineering team headcount grew past 60, we realized that maintaining and configuring Jenkins jobs had become unscalable: Using Jenkins UI to manually create new jobs based on “templates” could cause inconsistency among jobs and is prone to human error. While we were afraid of scaling such a tedious and manual process, the lack of clarity surrounding job ownership also became an issue. We knew we needed to evolve our way of working. We switched to Jenkins DSL in the hopes of programmatically controlling our delivery pipeline to decrease risk and gain efficiency.
In this post, we’ll describe how we leveraged Jenkins DSL to facilitate software delivery that meets the needs of our growing team and customer base.
Why did we choose Jenkins DSL?
Jenkins DSL is a Groovy-based domain-specific language (DSL) that allows developers to define Jenkins jobs and configuration in a programmatic form. Groovy facilitates parsing JSON/XML and handling HTTP requests in scripts. Declaring Jenkins jobs programmatically also offers the abstraction and extensibility of a programming language, including the ability to factor out shared components.
The main benefits we found after switching to Jenkins DSL included:
- Our engineering teams run jobs on multiple Jenkins servers, and many tenant-specific jobs spawn from the same template, so keeping them in sync requires deliberate human effort. Declaring jobs in code ensures consistency among generated jobs based on the same boilerplate and across our Jenkins servers, reducing the likelihood of human error.
- Checking the release pipeline into source control increases auditability and encourages collaboration. With this, we disabled configuration options on Jenkins for everyone outside the Infrastructure team, forcing all other teams to configure jobs via DSL and every change to go through code review.
- We arranged DSL scripts according to teams and engineering domains. This led to a more standardized process of extending functionality and assigning responsibility.
- Codifying the release pipeline has reduced the overall complexity of job configuration and increased the transparency of service deployment for engineering at Blend. Common code blocks including Slack notifications and Git configuration are shared among teams.
The new release pipeline
At Blend, the Platform team is responsible for deploying Blend’s core web services at the heart of our business. The deployment process is complex and time-consuming—and happens almost every day.
Every release has to go through multiple stages of deployment (including staging, preprod, and beta), as well as regression testing before it can reach production. On top of the platform team’s process, product managers and deployment leads work closely with our clients and other internal teams to ensure that all new features satisfy the specification at each stage.
Even before deployment, we’re constantly integrating new changes and testing them extensively before they get anywhere near production. Every pull request is required to pass a suite of thousands of unit tests, API tests, and end-to-end tests before merging into master.
As we moved our release pipeline to Jenkins DSL, we were able to configure most Jenkins jobs based on a “pipeline template” using the same underlying DSL scripts. Jenkins DSL allowed our team to configure many types of deployment settings to fit our needs. Each deployment stage includes a slightly different combination of build artifacts and environment variables, so we added branching and parameters throughout the workflow.
We adopted the factory method pattern for creating job objects extensively in our DSL repository. In DSL, the so-called seed job is the master script that triggers job creation. The code snippet below depicts how part of our release jobs are created with environment variables from configuration settings. For our core web service release, build and deploy jobs are created and kicked off by a class factory called BuildAndDeployAll. BuildAndDeployAll triggers subsequent downstream release jobs like PostDeployAll for configuring tenant-specific settings and SmokeTestAll for smoke testing.
The high-level steps of each deployment stage can be summarized as follows:
Similar to common practices, our first step is to build a Docker image and static assets. Each build job is associated with a shell script responsible for the corresponding task. The stringParam method is used to export predefined Jenkins parameters into shell commands. Keep in mind that the build stage is performed only once per release.
Once the base image is built, the deploy job is triggered to provision and deploy containers with environment- and tenant-specific variables. Creating a new deployment can be done simply by appending new tenant information to the DSL scripts.
Tasks including tenant-specific app configuration and data migration/validation are performed in the post-deploy section. These jobs are mutually exclusive and are run in parallel on multiple Jenkins executors.
Before moving onto the next deployment stage, all new instances are required to pass a suite of smoke tests similar to those in the “merging a pull request” step. Slack notifications are enabled to report test failures, and it is up to the Platform team to coordinate release issues within the engineering organization.
The transition of turning continuous delivery into code helped serialize the release pipeline with clarity and give us the flexibility to make adjustments confidently to meet future demands, all while bolstering a more collaborative and secure engineering culture.
Over the past few months, these have been our team’s main takeaways:
- An “end-to-end” release pipeline as code has improved the reliability and security of our service deployment. The new paradigm lowers the risk of human error and unlocks the benefits that come with source control.
- We can now hand the responsibility of deploying services over to the service owners themselves. With Jenkins DSL they can now own their release in code.
- Based on our current microservice architecture, giving complete ownership of a service, from development to delivery, to its team contributes significantly to our overarching mission of building a robust and modern consumer finance ecosystem.
Leveraging opportunities for automation has been key to growing our engineering operations sustainably. As we continue to adopt Jenkins DSL fully across our organization, we are excited to learn and share new practices for scaling our engineering facility.
We have career opportunities open across all departments at Blend and would love to have you join us and become part of this effort. Check out our current openings here.