CNLearn FastAPI - Adding Poetry for Dependency Managements and Packaging

  • Sat, Apr 15, 2023
  • 4-minute read

Today we’ll be looking at introducing dependency management and packaging to our project. Specifically, we’ll be using poetry. Why? Because I like it and it’s quite easy to use and I like having everything in one place in pyproject.toml.

How do we get started with it? First we need to install it to our system. I will not go in detail on the installation as it can be found here. Instead, I will discuss about how I changed from a python .venv based project with pip install -r requirements/whateverfile.txt to using poetry.

Initialising in an existing project

First, I changed Poetry’s default location for virtual environments with the virtualenvs.in-project config. I like having my .venv in the same folder as the project as I often look at the code in there when editing.

Then, I initialised Poetry in the existing project by doing

cd backend
poetry init

I also

It then asked me to go over each of the dependencies and got added to the correct group. There are 3 sections in the pyproject.toml file regarding dependencies:

  • [tool.poetry.dependencies]
  • [tool.poetry.group.dev.dependencies]
  • [tool.poetry.group.test.dependencies]

The first one represents the dependencies to run the application (in a production environment). To get something in that section, it’s as simple as doing:

poetry add fastapi

If we want to add to the test group or dev group, we’d have to do:

poetry add pytest --group test
poetry add black --group dev

Once we have the dependencies in our pyproject.toml file in the correct sections, let’s run:

poetry install

It also creates a poetry.lock file that we add to our version control system in order to have reproducible environments.

Scripts section

I also added a scripts section to the pyproject.toml file, specifically:

[tool.poetry.scripts]
dev = "app.server:development"
prod = "app.server:production"

What do these do? Well, I want to have a easy way to run the dev server instead of doing

uvicorn app.server:app

Please note that these things will change as the project gets closer to being “complete”. It’s more of a ease of use change for now during development. The actual production server won’t be run from poetry :)

In server.py we have:

import uvicorn

from app.app import app


def development() -> None:
    """
    This runs the development server.
    """
    uvicorn.run("app.app:app", host="127.0.0.1", port=8000, log_level="debug", reload=True)


def production() -> None:
    """
    This runs the production server but it will probably be run differently.
    """
    uvicorn.run(app, host="127.0.0.1", port=8000, log_level="info")

When we do poetry run dev it essentially runs the development function in that file. You can think of it as similar to an npm run something script.

Updating CI

We also had to update the CI workflow file to install poetry and use it. At the same time, let’s also add caching to it.

Before we install dependencies, let’s do the following:

- name: Install Poetry
  uses: snok/install-poetry@v1
  with:
    virtualenvs-create: true
    virtualenvs-in-project: true
    installer-parallel: true

- name: Load cached venv
  id: cached-poetry-dependencies
  uses: actions/cache@v3
  with:
    path: .venv
    key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}

What does this do? Installs poetry, and caches the dependencies. If they are not cached, the following also runs:

if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
run: poetry install --no-interaction --no-root

That way we save a little time from the workflow duration. I also changed some of the checks:

  • mypy to poetry run mypy
  • black --check . to poetry run black --check . Oh and we also changed to using ruff for linting instead of flake8 :)

And that’s basically it. It’s a short post but needed as we make improvements to the project outside of just the code. The commit for this post is here. In the next post or two, we will be making some changes to the startup/shutdown events to use a lifespan state instead as well as making some improvments to the CI flow (we need to simplify it, speed it up). Finally, we will also add some structured logging soon. Once we have these in place, we’ll go back to the code and continue adding features.

pythonfastapi fastapipythonpoetry