CNLearn FastAPI Continuous Integration

• Sun, Jul 24, 2022
• 3-minute read

Like I promised in the previous post, this post will be about adding a few things to our continuous integration pipeline. We will accomplish the following in this post:

• adding a https://pre-commit.com/ to our repo
• adding a GitHub Action to run our tests

Pre-commit

Let’s start with pre-commit. https://pre-commit.com/ is a nice and simple way of adding certain hooks to happen before committing: could be linting, reformatting, blank line checker, whatever you want really. In this case, we will run a few hooks they can provide. First, let’s install pre-commit and add it to our dev.txt (newly created) requirements file.

pip install pre-commit

Then, let’s add a .pre-commit-config.yaml config file.

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.3.0
    hooks:
      - id: check-yaml
      - id: end-of-file-fixer
      - id: trailing-whitespace
      - id: flake8
  - repo: https://github.com/psf/black
    rev: 22.3.0
    hooks:
      - id: black
  - repo: https://github.com/pycqa/isort
    rev: 5.10.1
    hooks:
      - id: isort
        name: isort (python)

Then, we run pre-commit install to install the hooks.

Finally, let’s do a run over all the files outside a commit -> this actually showed me a looooot of errors and inconsistencies which I have since fixed.

pre-commit run --all-files

It’s worth noting I also made some changes to the black config by settings line-length to 120 in pyproject.toml. I also made the same change and added some flake8 settings in tox.ini.

After running the command above and making the changes, let’s create a workflow file in the .github/workflows directory that will set up our Continuous Integration.

Github Actions Continuous Integration

For now, I will only run the tests on Python 3.9, but I am adding the action in such a way as to make it easy to add other versions later on. In terms of the workflow we have:

• Set up Python 3.9
• Install testing dependencies as well as flake8 and black
• Run flake8 and black in case our pre-commit missed something
• Set up PostgreSQL
• Wait a few seconds for it to be ready
• Check the running containers
• Run the tests

Keep in mind that usernames/passwords were read by our settings from .dev.env. We must therefore add the necessary variables to the environment. We are doing that in the environment file as well.

name: Run flake8, black and pytest

on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.9"]

    steps:
      - uses: actions/checkout@v3
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v4
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install wheel
          pip install flake8 black
          pip install -r requirements/testing.txt          
      - name: Lint with flake8
        run: |
          # stop the build if there are Python syntax errors or undefined names
          flake8 . --count --show-source --statistics          
      - name: Run black --check .
        run: black --check .
      - name: Setup PostgreSQL
        uses: Harmon758/postgresql-action@v1.0.0
        with:
          postgresql version: "14"
          postgresql db: "cnlearn_testing"
          postgresql user: "postgres"
          postgresql password: "postgres"
      - name: Wait / Sleep
        uses: jakejarvis/wait-action@v0.1.0
        with:
          time: "10s"
      - name: Collect Docker Logs
        uses: jwalton/gh-docker-logs@v1.0.0
      - name: Check running containers
        run: docker ps -a
      - name: Test with pytest
        env:
          APP_NAME: "CNLearn Testing"
          VERSION: 0.0.5
          SERVER_NAME: localhost
          SERVER_HOST: http://127.0.0.1:8000
          POSTGRES_SERVER: localhost
          POSTGRES_PORT: 5432
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: cnlearn
          TESTING: 1
        run: |
          pytest          

Ok let’s add all the files we changed, push and hopefully the tests run :)

The commit for this post is here.