Migrating from CircleCI to AWS CodeBuild (consolidating costs).
CircleCI is a hugely popular CI/CD service. But it can be expensive. Migrating to AWS CodeBuild reduced costs and enabled declarative, Terraform based pipelines.
As the founder of a small SAAS startup, CircleCI was a no-brainer for continuous deployments. But as our needs grew the financial cost of CircleCI began to outweigh its benefits (ease of use and documentation). As we already relied on AWS for infrastructure and storage, we decided to migrate the build, test, and deployment pipelines of mailslurp.com to AWS CodeBuild.
CodeBuild is relatively under-rated service that integrates really nicely with AWS’s other platforms (and pricing model). It has far fewer features than a dedicated platform like CircleCI but can do what we need — and do that at a much lower price.
But more importantly, AWS CodeBuild allows one to manage pipelines in a declarative, versioned way (yes, CircleCI has a config file but this is only defines jobs and workflows; CodeBuild let’s you define the settings of the pipelines themselves).
FYI we still love ❤ CircleCI ❤ this is just a guide for those interested in trying something different.
Contents
- This article contains a background to our migration
- Code samples showing how we did it in Terraform
- And how we enabled scheduled Smoke Tests with notifications
Motivation
Our company, MailSlurp, is a hosted API for testing applications using real, ephemeral email addresses. You can create new email addresses on the fly and use them to sign up to your app; fetch authentication codes; and test a happy path. It’s written in Kotlin and deployed using Docker, Kubernetes, and AWS EKS.
Our CI/CD pipelines were initially managed in CircleCI. This included a suite of integration and smoke tests that were scheduled to execute every 10 minutes. Failures in these tests would result in Slack notifications via CircleCI’s wonderful built in notification options.
The intensity of our deployment cycle and Smoke Test schedule lead to CircleCI bills that affected sustainability. Young, self-funded startups need to cut costs wherever they can and CircleCI seemed like a good place to start.
Aside from reducing costs, migrating CI/CD from CircleCI to AWS would reduce platform complexity by consolidating our business around one set of service models and concepts.
Alternatives
There’s a plethora of open and closed source solutions for continuous deployment. Hosted options like Github Actions, Buddy Works, or roll-your-own applications like Go.cd and Jenkins. CodeBuild appealed because it strongly integrates with AWS services and is very affordable.
Most of the MailSlurp infrastructure is built using Terraform AWS resources. By moving to CodeBuild we gained the additional advantage of moving our pipeline configurations into our Terraform repository — very cool!
Goals
By migrating our pipelines we wanted to achieve three main goals:
- Minimal changes to our existing code base
- Scheduled pipeline builds
- Slack notifications for failures
- BONUS: Terraform versioning of resources
CircleCI basics: what did our pipelines look like?
CircleCI configures pipelines via a .circleci/config.yml file in your project’s Github or BitBucket repository. Steps in this file specify a Docker image or machine type to execute your build in, and what steps to take before during or after a given build. CircleCI also has a sophisticated API for scheduling workflows and triggering notifications.
Here is an example of our previous CircleCI config file for the API component of MailSlurp.
version: 2# define tasks or jobs
jobs:
test:
docker:
- image: openjdk:8
steps:
- checkout
- run: ./gradlew clean test --stacktrace --info
deploy:
docker:
- image: docker:18.06.1-ce-git
steps:
- checkout
- setup_remote_docker
- run:
name: Install dependencies for AWS
command: |
apk add py-pip
pip install awscli
- run:
name: Build image and push to ECR
command: ./package.sh# define workflows - a job sequence with conditions
workflows:
version: 2
build_and_deploy:
jobs:
- test
- deploy:
requires:
- test
filters:
branches:
only: master
This config instructs CircleCI to:
- Create a workflow called
build_and_deployfor this repository. - Create a
testanddeployjob that execute a number of steps. - Trigger the pipeline for every push (running the deploy job only on master).
These are the concepts we had to migrate to CodeBuild to replicate our existing CI/CD workflow and capabilities. On to that next.
Creating the AWS infrastructure
First thing’s first, as with many AWS services, we need to set it up. MailSlurp uses Terraform to manage it’s cloud resources. Terraform is an amazing open-source toolchain for managing infrastructure in a versioned, declarative way. Basically, you write what you want and Terraform creates it for you on your chosen cloud provider.
To set up the CodeBuild pipeline for our API we added a module to our infrastructure repo. Here’s what it looks like:
pipeline/
├── build.tf
├── cloudwatch.tf
└── vars.tfFirst we define some variables describing the pipeline:
variable "name" {
description = "Name of pipeline"
}variable "build_image" {
description = "What Docker image to run in"
default = "aws/codebuild/standard:1.0"
}variable "source_type" {
description = "Where to pull repo from"
default = "BITBUCKET"
}variable "source_location" {
description = "The git url for the repo"
}variable "sns_arn" {
description = "An AWS SNS topic to post errors to"
}
Then we use these variables to create the main components of the pipeline.
# storage for the pipeline
resource "aws_s3_bucket" "pipeline" {
bucket = "${var.name}-pipeline-cache"
acl = "private"
}# iam role that pipeline will need
resource "aws_iam_role" "pipeline_role" {
name = "${var.name}-pipeline-role"assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "codebuild.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
EOFlifecycle {
create_before_destroy = true
}
}# iam policy permissions for the pipeline
resource "aws_iam_role_policy" "pipeline" {
role = "${aws_iam_role.pipeline_role.name}"policy = <<POLICY
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Resource": [
"*"
],
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
]
},
{
"Effect": "Allow",
"Action": [
"ec2:*",
"ecr:*",
"sns:*"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"${aws_s3_bucket.pipeline.arn}",
"${aws_s3_bucket.pipeline.arn}/*"
]
}
]
}
POLICY
}# the pipeline itself
resource "aws_codebuild_project" "pipeline" {
name = "${var.name}"
description = "Teraform managed"
build_timeout = "5"
service_role = "${aws_iam_role.pipeline_role.arn}"artifacts {
type = "NO_ARTIFACTS"
}cache {
type = "S3"
location = "${aws_s3_bucket.pipeline.bucket}"
}environment {
compute_type = "BUILD_GENERAL1_SMALL"
image = "${var.build_image}"
type = "LINUX_CONTAINER"
# allow docker in docker
privileged_mode = true
}source {
type = "${var.source_type}"
location = "${var.source_location}"
git_clone_depth = 1# needed for private repositories (requires an initial manual step to grant access)
auth {
type = "OAUTH"
}
}tags = {
Name = "${var.name}-pipeline"
"Environment" = "Test"
}
}# enable BitBucket/Github to notify CodeBuild on changes
resource "aws_codebuild_webhook" "pipeline" {
project_name = "${aws_codebuild_project.pipeline.name}"
}
Lastly we add some logging capabilities so that errors are forwarded to an SNS stream via CloudWatch event targets (more on this later):
locals {
rule_name = "${var.name}-pipeline-alert-rule"
target_id = "${var.name}-pipeline-alert-target"
}resource "aws_cloudwatch_event_rule" "pipeline" {
name = "${local.rule_name}"# a pattern for parsing event logs for failed CodeBuild buils
event_pattern = <<PATTERN
{
"source": [
"aws.codebuild"
],
"detail-type": [
"CodeBuild Build State Change"
],
"detail": {
"build-status": [
"FAILED"
],
"project-name": [
"${aws_codebuild_project.pipeline.name}"
]
}
}
PATTERN
}resource "aws_cloudwatch_event_target" "pipeline" {
target_id = "${local.target_id}"
rule = "${local.rule_name}"
# this connects the event target with out SNS topic
arn = "${var.sns_arn}"input_transformer {
input_paths {
buildId = "$.build-id"
projectName = "$.detail.project-name"
buildStatus = "$.detail.build-status"
}input_template = <<EOF
{ "project": <projectName>, "status": <buildStatus> }
EOF
}depends_on = ["aws_cloudwatch_event_rule.pipeline"]
}
Phew. That was quite a bit of code to do something that would only takes a click or two in CircleCI’s dashboard. Well, you could do it the exact same way in AWS CodeBuild (there is a nice web interface) but we chose to use Terraform so that our pipelines are versioned. That means the settings of the pipelines themselves are checked into Github and we can make changes in a safe, peer reviewed way.
But how do we define our build jobs and workflows?
CodeBuild uses a buildspec.yml file in the root of your repository. It’s analogous to CircleCI’s .circleci/config.yml file. It does however have a lot less features — but honestly, we don’t use many of CircleCI’s advanced features anyway.
version: 0.2phases:
install:
commands:
- apk add py-pip && pip install awscli
- ./gradlew clean test --stacktrace
- ./package.sh
This runs the same tasks as our CircleCI config but on every branch. As package.sh script only pushes containers to ECR that’s okay for us. It doesn’t deploy anything into production.
If you want to have branch filtering and different jobs for different stages, then you need to add some additional AWS CodePipeline resources. For the sake of this example we will keep it simple.
Enabling Slack Notifications for failed builds
You may have noticed the use of an AWS SNS topic in the previous Terraform code. That SNS topic is how we can send notifications to a Slack channel for failed builds in CodeBuild.
Here is some Terraform code that creates an SNS Topic and uses an external Slack module to forward the message:
resource "aws_sns_topic" "sns_alarm" {
name = "sns-alarms"
}data "aws_iam_policy_document" "sns_alarm" {
statement {
sid = "TrustCloudWatchEvents"
effect = "Allow"
resources = ["${aws_sns_topic.sns_alarm.arn}"]
actions = ["sns:Publish"]principals {
type = "Service"
identifiers = ["events.amazonaws.com"]
}
}
}resource "aws_sns_topic_policy" "builds_events" {
arn = "${aws_sns_topic.sns_alarm.arn}"
policy = "${data.aws_iam_policy_document.sns_alarm.json}"
}module "notify_slack" {
source = "git::https://github.com/terraform-aws-modules/terraform-aws-notify-slack.git"
sns_topic_name = "${aws_sns_topic.sns_alarm.name}"
slack_webhook_url = "XXXXXXXXXXXXXXXXXXXXXXX"
slack_channel = "alerts"
slack_username = "MailSlurpBot"
}
Enabling scheduled smoke tests
MailSlurp had a suite of smoke tests that ran as a scheduled workflow in CircleCI. Migrating these to CodeBuild involved similar steps as above but with an additional manual action needed to create a CodeBuild trigger. This, unfortunately, had to be done in the AWS Console for now and not in Terraform, but it was pretty straight forward. Scheduling is specified in Cron syntax and was simple to set up.
Putting it all together
After the migration we’ve seen a considerable reduction in CI costs. This is a big win for a small company. It took some work to set up our new pipelines but the savings plus the added benefit of versioned, declarative pipelines made CodeBuild a compelling CI system.
Time will tell how CodeBuild serves us but for now we are thoroughly enjoying it. If you’re interested in testing your application with real email addresses check out our company at MailSlurp.com.
We still love ❤ CircleCI ❤ but, for our needs, closer integration with our existing AWS infrastructure has proved a success.
