Developer's guide
=================

Setup
_____

Follow these steps to setup the project for development:

1. Clone the project from git and go the the main folder.
2. Create an `.env` file with your git credentials and personal GPG key data (if you need separate values for this project). You can use `.env.sample` for reference.
3. If you're using pyenv: set a local python version, for example: `pyenv local 3.11.1`.
4. Create a new virtual environment `python -m venv venv && . ./venv/bin/activate`.
5. Run the setup script to setup the project for development: `tools/init.sh`.
6. Optional: build the dev documentation locally with: `tools/docs.sh`.

Project structure
_________________

You should stick to the standard layout when developing the project.

Files:

- `kaiju_tools/` - python project files
- `docs/` - sphinx documentation files, see `Docs`_
- `etc/` - other non-executable files (db configs, docker configs, system package dependencies, etc.)
- `tests/` - pytest tests, see `Testing`_
- `tools/` - additional executable not used by the project code (sh and py scripts, helpers, hooks)

Making changes
______________

This project uses `gitlab flow <https://docs.gitlab.com/ee/topics/gitlab_flow.html>`__.

It means that there is a single `master` branch and there are no other branches except feature branches. However
when there is a need to support an older version, a *version branch* may be created.

Open a `new issue <https://gitlab.com/kaiju-python/kaiju-tasks/-/issues>`__ before making code changes, reference this
issue in your branch by its number. All branch names **must** follow the simple rule:

.. code-block::

  <issue>-<change_type>-<description>

Where `<change_type>` must be:

- `fix` - bugfix, doesn't change any functionality
- `tweak` - behavior / config tweak, shouldn't change interface but may adjusts the behavior
- `docs` - documentation related: docstrings, rst doc files, project description etc.
- `test` - added tests or test related stuff
- `maint` - project maintenance: package versions update, packaging config updates, distribution
- `feat` - feature request, something that changes the behavior and interface
- `misc` - other changes which don't fit into provided categories
- `removal` - discontinued code removal

Use `-` instead of `_` in description.

Example:

.. code-block::

  112-feat-cache-reinitialization

You should use a specific commit message format for the project. It's very simple.

.. code-block::

  <change type>: <message>

Where `<change type>` type must be similar to the mentioned above and the message should be
less than 72 symbols (use 'commit -a' if you need more symbols and add a separate description from a new row).
There's no need to try to write down everything you did. These messages are intended for better navigation,
keep them simple.

Example:

.. code-block::

  git commit -am 'feat: added request validation in search api'
  git commit -am 'maint: removed unnecessary test deps'

.. attention::

  Do not use meaningless messages:

  - "fixed data.py"
  - "many changes"

How to describe changes
_______________________

The project uses `towncrier <https://github.com/twisted/towncrier>`__ to incorporate changelog into `CHANGES.rst`.

Significant changes (features, major bug fixes) should be described in separate `CHANGES/<issue>.<change_type>`
files. Do not copy entire git tree or issue tracker in here: you should only describe changes significant
from the user (not developer) perspective. The text should be in *.rst* format.

Example:

.. code-block::

  file CHANGES/342.feat

  Added retry feature for RPC requests via `Retries=<num_of_retries>` header.


Releases
________

This project uses `twine <https://twine.readthedocs.io/en/latest/>`__ and
`setup.cfg <https://docs.python.org/3/distutils/configfile.html>`__ for packaging.
It uses `wheel <https://www.python.org/dev/peps/pep-0427/>`__ packages by default.

Since this project uses CI hooks there is no need to manually build packages. You just need
to create a *version bump*.

Follow these steps to create a new version:

1. Commit all uncommitted code.
2. Run `tools/release.sh patch` (`minor`, `major`).
3. Use `git log` to ensure that a version commit has been created.
4. Ensure that all project files in `./build/` directory have been collected properly. If you can't find subpackages, ensure that you have added subpackages into `setup.cfg` "[options] packages".
5. Push your commits *including tags* to the remote repo. The package will be built automatically.

If the remote hasn't created a new version of the package, check that the release branch and tag are protected
(in gitlab: Settings/Repository/Protected branches, Protected tags). You should also verify
that `TWINE_USERNAME` and `TWINE_PASSWORD` are present in the CI (in gitlab: Settings/CI/Variables).

However you can build and publish it manually
(just make sure that your `.pypirc <https://packaging.python.org/specifications/pypirc/>`__ is
properly configured):

.. attention::

  - Do not use requirements files. Write all dependencies into `setup.cfg`.
  - Do not change setup.py.

Files:

- `setup.cfg` - package config
- `setup.py` - additional setup instructions, should not be edited
- `MANIFEST.in` - tells the package manager which files to include, should not be edited
- `requirements/*.txt` - auto-generated frozen requirements files, should not be edited

Testing
_______

This project uses `pytest <http://pytest.org>`__ for testing.

Follow these recommendations for better testing:

- Write all your tests inside `tests/` but try to keep subdirectories consistent with the package structure.
- Use `doctests <https://docs.python.org/3/library/doctest.html>`__ whenever possible.
- Use pytest `markers <https://docs.pytest.org/en/7.2.x/example/markers.html>`__ to distinguish between test types. New markers can be added in `pyproject.toml` pytest section.
- Use `tox <https://tox.wiki/en/latest/index.html>`__ more or less frequently to test different python versions.

.. note::

  Some tests may require a running docker engine.

Files:

- `pyproject.toml` - contains pytest configs
- `tests/*` - all tests (except doctests) should be here
- `conftest.py` - pytest hooks and global fixtures
- `tox.ini` - tox configuration

QA
__

Q/A scripts use git hooks and `pre-commit <http://pre-commit.com>`__ library which is installed
automatically by the `tools/init.sh` script. The hooks include:

- `flake8 <https://flake8.pycqa.org>`__, `pydocstyle <http://www.pydocstyle.org>`__ and other code style checks
- `pytest <https://pytest.org>`__
- `pyupgrade <https://github.com/asottile/pyupgrade>`__ - automatically upgrades python syntax to the modern standards
- `black <https://black.readthedocs.io/en/stable/>`__ - python code formatter
- `rst-lint <https://github.com/twolfson/restructuredtext-lint>`__ - checks RST documentation files
- `pip-compile <https://pip-tools.readthedocs.io/en/latest/>`__ - resolves dependencies and auto-generates requirements/ files

.. note::

  Pre-committ may abort a commit and may require some actions from the dev to fix or review changes made by the hooks.
  Simply repeat the commit command if everything is ok.

Files:

- `.editorconfig` - standard text editor configuration
- `pyproject.toml` - tools configuration
- `.pre-commit-config.yaml` - `pre-commit <http://pre-commit.com>`__ for local development
- `.pre-commit-config-ci.yaml` - `pre-commit <http://pre-commit.com>`__ for CI jobs on the server

CI
__

Use CI configs to store deployment and test hooks to increase your code quality.

Files:

- `.gitlab-ci.yml` - `Gitlab CI <https://docs.gitlab.com/ee/ci/yaml/>`__ pipelines
- `.readthedocs.yaml` - `ReadTheDocs <https://docs.readthedocs.io/en/stable/config-file/index.html>`__ - doc builder configuration

Variables:

There are two pipelines configured: for gitlab and for github. These pipelines
use a several env variables which should be put in CI secrets section:

- `TWINE_USERNAME` and `TWINE_PASSWORD` - are required for PyPI package builds. The best way is to use `__token__` for the username and a new `PyPI token <https://pypi.org/help/#apitoken>`__ for this particular project.
- `DOCKER_REGISTRY_ADDR`, `DOCKER_REGISTRY_USERNAME`, `DOCKER_REGISTRY_PASSWORD` - may be used to automatically create and push docker images.

Docs
____

CI should automatically build the latest version of documentation. If you need to build the docs locally, you should use `tools/docs.sh` tool.

To write documentation for the project:

1. Create an `.rst` file inside `docs/source` directory.
2. Write nice documentation using `RST format <https://github.com/ralsina/rst-cheatsheet/blob/master/rst-cheatsheet.rst>`__.
3. Add your file into the `docs/index.rst`.
4. Once finished: try to test build it locally using `tools/docs.sh` and fix all errors and warnings.
5. When everything is OK, make a commit with "docs: " prefix.

.. attention::

  Do not commit `docs/build` files and also *source files created by autodoc*.
  You should only commit manually created documentation.

Files:

- `LICENSE` - project license file
- `README.rst` - basic readme file
- `CHANGES.rst` - minor and major version changes should be listed here
- `docs/` - `Sphinx <https://www.sphinx-doc.org/en/master/>`__ files