Developer’s guide

Setup

Follow these steps to setup the project for development:

  1. Clone the project from git and go the the main folder.

  2. Create an .env file with your git credentials and personal GPG key data (if you need separate values for this project). You can use .env.sample for reference.

  3. If you’re using pyenv: set a local python version, for example: pyenv local 3.11.1.

  4. Create a new virtual environment python -m venv venv && . ./venv/bin/activate.

  5. Run the setup script to setup the project for development: tools/init.sh.

  6. Optional: build the dev documentation locally with: tools/docs.sh.

Project structure

You should stick to the standard layout when developing the project.

Files:

  • kaiju_tools/ - python project files

  • docs/ - sphinx documentation files, see Docs

  • etc/ - other non-executable files (db configs, docker configs, system package dependencies, etc.)

  • tests/ - pytest tests, see Testing

  • tools/ - additional executable not used by the project code (sh and py scripts, helpers, hooks)

Making changes

This project uses gitlab flow.

It means that there is a single master branch and there are no other branches except feature branches. However when there is a need to support an older version, a version branch may be created.

Open a new issue before making code changes, reference this issue in your branch by its number. All branch names must follow the simple rule:

<issue>-<change_type>-<description>

Where <change_type> must be:

  • fix - bugfix, doesn’t change any functionality

  • tweak - behavior / config tweak, shouldn’t change interface but may adjusts the behavior

  • docs - documentation related: docstrings, rst doc files, project description etc.

  • test - added tests or test related stuff

  • maint - project maintenance: package versions update, packaging config updates, distribution

  • feat - feature request, something that changes the behavior and interface

  • misc - other changes which don’t fit into provided categories

  • removal - discontinued code removal

Use - instead of _ in description.

Example:

112-feat-cache-reinitialization

You should use a specific commit message format for the project. It’s very simple.

<change type>: <message>

Where <change type> type must be similar to the mentioned above and the message should be less than 72 symbols (use ‘commit -a’ if you need more symbols and add a separate description from a new row). There’s no need to try to write down everything you did. These messages are intended for better navigation, keep them simple.

Example:

git commit -am 'feat: added request validation in search api'
git commit -am 'maint: removed unnecessary test deps'

Attention

Do not use meaningless messages:

  • “fixed data.py”

  • “many changes”

How to describe changes

The project uses towncrier to incorporate changelog into CHANGES.rst.

Significant changes (features, major bug fixes) should be described in separate CHANGES/<issue>.<change_type> files. Do not copy entire git tree or issue tracker in here: you should only describe changes significant from the user (not developer) perspective. The text should be in .rst format.

Example:

file CHANGES/342.feat

Added retry feature for RPC requests via `Retries=<num_of_retries>` header.

Releases

This project uses twine and setup.cfg for packaging. It uses wheel packages by default.

Since this project uses CI hooks there is no need to manually build packages. You just need to create a version bump.

Follow these steps to create a new version:

  1. Commit all uncommitted code.

  2. Run tools/release.sh patch (minor, major).

  3. Use git log to ensure that a version commit has been created.

  4. Ensure that all project files in ./build/ directory have been collected properly. If you can’t find subpackages, ensure that you have added subpackages into setup.cfg “[options] packages”.

  5. Push your commits including tags to the remote repo. The package will be built automatically.

If the remote hasn’t created a new version of the package, check that the release branch and tag are protected (in gitlab: Settings/Repository/Protected branches, Protected tags). You should also verify that TWINE_USERNAME and TWINE_PASSWORD are present in the CI (in gitlab: Settings/CI/Variables).

However you can build and publish it manually (just make sure that your .pypirc is properly configured):

Attention

  • Do not use requirements files. Write all dependencies into setup.cfg.

  • Do not change setup.py.

Files:

  • setup.cfg - package config

  • setup.py - additional setup instructions, should not be edited

  • MANIFEST.in - tells the package manager which files to include, should not be edited

  • requirements/*.txt - auto-generated frozen requirements files, should not be edited

Testing

This project uses pytest for testing.

Follow these recommendations for better testing:

  • Write all your tests inside tests/ but try to keep subdirectories consistent with the package structure.

  • Use doctests whenever possible.

  • Use pytest markers to distinguish between test types. New markers can be added in pyproject.toml pytest section.

  • Use tox more or less frequently to test different python versions.

Note

Some tests may require a running docker engine.

Files:

  • pyproject.toml - contains pytest configs

  • tests/* - all tests (except doctests) should be here

  • conftest.py - pytest hooks and global fixtures

  • tox.ini - tox configuration

QA

Q/A scripts use git hooks and pre-commit library which is installed automatically by the tools/init.sh script. The hooks include:

  • flake8, pydocstyle and other code style checks

  • pytest

  • pyupgrade - automatically upgrades python syntax to the modern standards

  • black - python code formatter

  • rst-lint - checks RST documentation files

  • pip-compile - resolves dependencies and auto-generates requirements/ files

Note

Pre-committ may abort a commit and may require some actions from the dev to fix or review changes made by the hooks. Simply repeat the commit command if everything is ok.

Files:

  • .editorconfig - standard text editor configuration

  • pyproject.toml - tools configuration

  • .pre-commit-config.yaml - pre-commit for local development

  • .pre-commit-config-ci.yaml - pre-commit for CI jobs on the server

CI

Use CI configs to store deployment and test hooks to increase your code quality.

Files:

Variables:

There are two pipelines configured: for gitlab and for github. These pipelines use a several env variables which should be put in CI secrets section:

  • TWINE_USERNAME and TWINE_PASSWORD - are required for PyPI package builds. The best way is to use __token__ for the username and a new PyPI token for this particular project.

  • DOCKER_REGISTRY_ADDR, DOCKER_REGISTRY_USERNAME, DOCKER_REGISTRY_PASSWORD - may be used to automatically create and push docker images.

Docs

CI should automatically build the latest version of documentation. If you need to build the docs locally, you should use tools/docs.sh tool.

To write documentation for the project:

  1. Create an .rst file inside docs/source directory.

  2. Write nice documentation using RST format.

  3. Add your file into the docs/index.rst.

  4. Once finished: try to test build it locally using tools/docs.sh and fix all errors and warnings.

  5. When everything is OK, make a commit with “docs: “ prefix.

Attention

Do not commit docs/build files and also source files created by autodoc. You should only commit manually created documentation.

Files:

  • LICENSE - project license file

  • README.rst - basic readme file

  • CHANGES.rst - minor and major version changes should be listed here

  • docs/ - Sphinx files