Python Notes

my python import reference: https://chrisyeh96.github.io/2017/08/08/definitive-guide-python-imports.html#

• jobs vs workflows
• filtering
• commands
• caching

Jobs vs Workflows

First important to mention that workflows are only available for version 2.1. Plus you need to make sure to enable it on CircleCI settings page.

Workflows?

workflows can be used to group jobs together. For example you want to have: unit test and linting, then you want to bundle the code and then to deploy to production.

As you can see there are steps for this flow and some jobs depends on the success of other jobs.

The benefits of the above flow, is that you can run several jobs concurrently and save time for your builds.

Jobs?

This is available both for version 2 and 2.1. Its quite simple as you can define one job and have the whole build process in the steps of that job. For example:

jobs:
  build:
    docker:
      - image: someimage
    steps:
      - lint:
        implementation here
      - unittest:
        implementation here
      - prep env:
        implementation here
      - bundle code:
        implementation here
      - deploy:
        implementation here

The benefits here is the simplicity, one job, one process one point of failure for good and bad. But all the steps run synchronously one by one, so no time saving with concurrently running steps.

Filtering

Filtering can do real magic with the way you CI works. You can filter out branches that will get deployed to production. Filter branches that will skip testing and go directly to dev environment. I did used a lot of bash code to manipulate what should run and when. But as you get familiar with filters that can be super powerful and easy to implement.

Example:

workflows:
  deploy_staging_prod:
    jobs:
      - checkout_code:
          filters:
            branches:
              only:
                - branch1
                - branch2
              ignore:
                - branch3

Just to clarify "only" and "ignore" can't live together, but just wanted to show how it looks in the example. You can define which branches will start the workflow and which branches won't. Thats great if you have different workflows purposes.

Commands

Commands are a great way to reuse you actions. It could be a slack notification you want to send. It could be some bash snippet you want to run or whatever.

I got to use it only once and it mostly cleaned my config file so that's surely a big plus. It's quite easy to implement and use as you can see in the example below.

jobs:
  my-job:
    executor: my-executor
    steps:
      - first step
      - sayhello #this will use the below command

commands:
  sayhello:
    description: "A very simple command for demonstration purposes"
    parameters:
      to:
        type: string
        default: "Hello World"
    steps:
      - step-1
      - ...
      - step-n

Caching

Another great tool to your arsenal. This is mostly used to speed up your builds by caching long process that can get skipped.

For example you need to fetch your project dependencies. So you can store the dependencies in cache for the next job and it will get pulled from there and not from the web. Just be sure you are saving the right files to cache and refreshing the cache when needed.

How you cach node_modules and npm

• pulling the repository
• checking if cache exists for my checksum for yarn.lock file
• that means if we already npm installed this lock file and there were no changes we should have cache stored
• run install
• if cache was found so all the files from cache are in the workspace
• save cache for our yarn.lock checksum
• for the next build, we could use the cache

jobs:
  checkout_code:
    executor: my-executor
    steps:
      - checkout
      - restore_cache: # special step to restore the dependency cache
          key: dependency-cache-{{ checksum "yarn.lock" }}
      - run:
          name: yarn-install
          command: yarn install
      - save_cache:
          key: dependency-cache-{{ checksum "yarn.lock" }}
          paths:
            - ./node_modules

Cache Trick to skip job

I had a project where I need to skip a job in case cache existed. Specifically it was for skipping linting and unittests in case those jobs already ran successfully.

So if they ran successfully, I'll just skip the job and won't perform the tests or the linting.

unit_test:
    executor: my-executor
    steps:
      - skip_if_cache_exists:
          skiptype: "unittests"
      - run: 
          name: Unit Test
          command: yarn test --maxWorkers=4
      - save_cache_flag:
          skiptype: "unittests"

commands:
  skip_if_cache_exists:
    description: |
      a command to exit the job for selected branch
    parameters:
      skiptype:
        description: type of job to skip
        type: string
    steps:
      - restore_cache:
          key: skipcheck-<<parameters.skiptype>>-{{ .Environment.CIRCLE_BRANCH }}-{{ .Environment.CIRCLE_SHA1 }}
      - run: 
          name: if cache exists exit
          command: |
            FILE=~/cachedflags/job.<<parameters.skiptype>>.flag
            if test -f "$FILE"; then
                echo "$FILE exist"
                circleci step halt
            else
                echo "$FILE doesnt exist"
            fi
  save_cache_flag:
    description: |
      a command that will create the cache
    parameters:
      skiptype:
        description: type of job to skip
        type: string
    steps:
      - run:
          name: create job flag file
          command: mkdir -p ~/cachedflags/ && touch ~/cachedflags/job.<<parameters.skiptype>>.flag
      - save_cache:
          key: skipcheck-<<parameters.skiptype>>-{{ .Environment.CIRCLE_BRANCH }}-{{ .Environment.CIRCLE_SHA1 }}
          paths:
            - ~/cachedflags/job.<<parameters.skiptype>>.flag