My Recent Circle Project Deep Dive
As I'm quite proud with what our current CircleCI flow is doing. Especially as I managed to cut the run time in 70% from the original build, and got all of the "tasks" run separataly.
Here is the document I wrote down to explain to the team what is going on inside Circle:
Circle CI
We use Circle version 2.1 for our builds. That means we can use their latest features like: workflows, commands and orbs.
Workflows
We now have three different workflows for specific set of branches:
- deploy_staging_prod
- test_braches
- deploy_integration
Each of them has its own set of jobs to run that are specific to our needs.
deploy_staging_prod
branches: prod, prod_preview, staging, staging_preview
this workflow is for our production deploys and it consists of the full stack of jobs we have.
Jobs:
- checkout_code
- unit_test
- lint_code
- prep_env_vars
- build_code
- deploy
- publish_sentry
test_braches
branches: all branches, except: master, master_preview, prod, prod_preview, staging, staging_preview
this workflow is for our in development branches. It aims to be very fast and only give developers quick feedback on their code.
Jobs:
- checkout_code
- unit_test
- lint_code
deploy_integration
branches: master, master_preview
This build our integration preview where the team can preview the work before taking it to the next phase of staging and production. Here we skip the code checks as we relied on that on the above feature branches checks. But this should be fast so the rest of the team could see the output of dev team.
Jobs:
- checkout_code
- prep_env_vars
- build_code
- deploy
Jobs
Jobs are a part of worfklows and the contains a list of rules to perform for a specific goal. For example checking out the code, installing dependecies and run tests.
We have a list of different jobs in the system and we will describe each one.
- checkout_code
- unit_test
- lint_code
- prep_env_vars
- build_code
- deploy
- publish_sentry
checkout_code
This job is in charge of fetching the code, installing npm dependecies and caching both of them for future jobs to use. It takes advantage of Circle caching in cache of same git commits or unchange lock files.
unit_test
This job is pulling the code from our cached data. Then it runs our unit tests to verify the code works as expected.
lint_code
This job is pulling the code from our cached data. Then it runs linting on the code.
prep_env_vars
This job is pulling the code from our cached data.
Then it performs bash commands to generate the need .env.production file.
Attaching "sentry release" version if we are in a deployment branch.
In the end it persists the created file to workflow workspace.
build_code
This job is pulling the code from our cached data, and pulling the created env file from the workspace data. Then it will bundle the app into the release version files. In the end it will attach the bundled files in to the workflow workspace.
deploy
This job is pulling the code from our cached data, and pulling the created env file from the workspace data. Then it will run the deployment bash script that will upload the code to S3 and invalidate our cdn.
publish_sentry
This job is pulling the code from our cached data. Gets the current git commit id and creates and releases to sentry.
Commands
Commands are a great way to "DRY" (don't repeat yourself). You define a set of steps you need and you can use one liner commands in your main jobs that take advantage of those commands.
Our different commands:
- get_app_code
- get_project
- slack_on_deployment
- skip_if_cache_exists
- save_cache_flag
get_app_code
This command is simply fetching our cached codebase.
get_project
This command is fetching our cached codebase and the cached npm dependencies.
slack_on_deployment
This is how we notify slack on jobs progress. So we want to send a slack messsage in case a job failed or succeeded. It has severl parameters that can be used for different behaviours.
skip_if_cache_exists
There are certain jobs we would like to skip if nothing was changed. For example we dont need to run unit tests if there was no code change. So we are caching a flag and in case the flag exists we just skip the rest of the job.
save_cache_flag
This command create a file as flag that will be cached for future jobs to check and skip on with the commands "skip_if_cache_exists"
executors
These are another way to define the containers our jobs will run on. Currently we have only one, but if there are jobs that require PHP, Golan etc. You can create an executer an reuse it in your jobs.
version: 2.1
executors:
my-executor:
docker:
- image: circleci/python:3.7.1-node-browsers
orbs:
slack: circleci/slack@3.4.1
jobs:
checkout_code:
executor: my-executor
steps:
- checkout
- save_cache:
key: code-{{ .Environment.CIRCLE_SHA1 }}
paths:
- ./
- restore_cache:
key: deps-{{ checksum "yarn.lock" }}
- run:
name: yarn-install
command: yarn install
- save_cache:
key: deps-{{ checksum "yarn.lock" }}
paths:
- ./node_modules
- slack_on_deployment:
success_message: "deployment started for [$CIRCLE_BRANCH]"
failure_message: "deployment failed to start for [$CIRCLE_BRANCH]"
prep_env_vars:
executor: my-executor
steps:
- get_project
- run:
name: Prep Env Vars
command: |
while IFS='' read -r var || [[ -n "$var" ]]; do
./scripts/prep-var.sh $CIRCLE_BRANCH $var
done < BUILD_VARS
- run:
name: Attach Git Version To Sentry env
command: |
if [[ "$CIRCLE_BRANCH" =~ ^(staging|staging_preview|prod|prod_preview)$ ]]; then
SENTRY_VERSION=$(git rev-parse --verify HEAD)
echo "SENTRY_RELEASE_VERSION=$SENTRY_VERSION" >> .env.production
else
echo "$CIRCLE_BRANCH is not in the list"
fi
- persist_to_workspace:
root: ./
paths:
- .env.production
- slack_on_deployment:
fail_only: true
failure_message: "deployment failed for [$CIRCLE_BRANCH] because of env_prep"
unit_test:
executor: my-executor
steps:
- skip_if_cache_exists:
skiptype: "unittests"
- get_project
- run:
name: Unit Test
command: yarn test --maxWorkers=4
- save_cache_flag:
skiptype: "unittests"
- slack_on_deployment:
fail_only: true
failure_message: "deployment failed for [$CIRCLE_BRANCH] because of unit tests"
lint_code:
executor: my-executor
steps:
- skip_if_cache_exists:
skiptype: "linting"
- get_project
- run:
name: Lint Code
command: yarn lint
- save_cache_flag:
skiptype: "linting"
- slack_on_deployment:
fail_only: true
failure_message: "deployment failed for [$CIRCLE_BRANCH] because of linting"
build_code:
executor: my-executor
steps:
- get_project
- attach_workspace:
at: ~/project
- run:
name: build_code
command: yarn build && yarn build-storybook -o public/storybook
- persist_to_workspace:
root: ./
paths:
- ./public
- slack_on_deployment:
fail_only: true
failure_message: "deployment failed for [$CIRCLE_BRANCH] because of code build"
deploy:
executor: my-executor
steps:
- get_app_code
- attach_workspace:
at: ~/project
- run:
name: Deploy to S3
command: ./scripts/deploy.sh
- slack_on_deployment:
success_message: "deployment successful for => [$CIRCLE_BRANCH]"
failure_message: "deployment failed for [$CIRCLE_BRANCH]"
publish_sentry:
executor: my-executor
steps:
- get_app_code
- run:
name: install sentry-cli
command: curl -sL https://sentry.io/get-cli/ | bash
- run:
name: set Sentry release version
command: echo $(git rev-parse --verify HEAD) > SENTRY_RELEASE_VERSION
- run:
name: create sentry release version
command: sentry-cli releases --org $SENTRY_ORG --project $SENTRY_PROJECT new $(cat SENTRY_RELEASE_VERSION)
- run:
name: finalizing sentry release
command: sentry-cli releases --org $SENTRY_ORG finalize $(cat SENTRY_RELEASE_VERSION)
- run:
name: finalizing sentry release
command: sentry-cli releases --org $SENTRY_ORG deploys $(cat SENTRY_RELEASE_VERSION) new --env $CIRCLE_BRANCH
commands:
get_app_code:
steps:
- restore_cache:
name: restore app code
key: code-{{ .Environment.CIRCLE_SHA1 }}
get_project:
steps:
- restore_cache:
name: restore app code
key: code-{{ .Environment.CIRCLE_SHA1 }}
- restore_cache:
name: restore dependencies
key: deps-{{ checksum "yarn.lock" }}
slack_on_deployment:
description: |
notify deployments slack channel
parameters:
only_for_branches:
default: "master,master_preview,prod,prod_preview,staging,staging_preview"
description: comma-separated list of branches for which to send notifications. No spaces.
type: string
fail_only:
default: false
description: should notify on fail only or success as well
type: boolean
success_message:
default: ""
description: msg on success
type: string
failure_message:
default: ""
description: msg on failure
type: string
steps:
- slack/status:
channel: "deployments"
success_message: "<<parameters.success_message>> - $CIRCLE_WORKFLOW_ID"
failure_message: "<<parameters.failure_message>> - $CIRCLE_WORKFLOW_ID"
only_for_branches: <<parameters.only_for_branches>>
fail_only: <<parameters.fail_only>>
skip_if_cache_exists:
description: |
a command to exit the job for selected branch
parameters:
skiptype:
description: type of job to skip
type: string
steps:
- restore_cache:
key: skipcheck-<<parameters.skiptype>>-{{ .Environment.CIRCLE_BRANCH }}-{{ .Environment.CIRCLE_SHA1 }}
- run:
name: if cache exists exit
command: |
FILE=~/cachedflags/job.<<parameters.skiptype>>.flag
if test -f "$FILE"; then
echo "$FILE exist"
circleci step halt
else
echo "$FILE doesnt exist"
fi
save_cache_flag:
description: |
a command that will create the cache
parameters:
skiptype:
description: type of job to skip
type: string
steps:
- run:
name: create job flag file
command: mkdir -p ~/cachedflags/ && touch ~/cachedflags/job.<<parameters.skiptype>>.flag
- save_cache:
key: skipcheck-<<parameters.skiptype>>-{{ .Environment.CIRCLE_BRANCH }}-{{ .Environment.CIRCLE_SHA1 }}
paths:
- ~/cachedflags/job.<<parameters.skiptype>>.flag
workflows:
deploy_staging_prod:
jobs:
- checkout_code:
filters:
branches:
only:
- prod
- prod_preview
- staging
- staging_preview
- unit_test:
requires:
- checkout_code
- lint_code:
requires:
- checkout_code
- prep_env_vars:
requires:
- checkout_code
- build_code:
requires:
- unit_test
- lint_code
- prep_env_vars
- deploy:
requires:
- build_code
- publish_sentry:
requires:
- deploy
test_braches:
jobs:
- checkout_code:
filters:
branches:
ignore:
- master
- master_preview
- prod
- prod_preview
- staging
- staging_preview
- unit_test:
requires:
- checkout_code
- lint_code:
requires:
- checkout_code
deploy_integration:
jobs:
- checkout_code:
filters:
branches:
only:
- master
- master_preview
- prep_env_vars:
requires:
- checkout_code
- build_code:
requires:
- prep_env_vars
- deploy:
requires:
- build_code