.. _app-guide: Application guide ================= This library is supposed to provide tools for constructing your asynchronous server application. See it for yourself in :ref:`app-api`. Application class is currently based on `aiohttp app `_. Basically application start consists of two stages: loading `Config files`_ and initialization of the application `Services`_. To create an app you can use :py:func:`~kaiju_tools.app.init_app` function. You can also start an aiohttp server by using :py:func:`~kaiju_tools.app.run_server`. .. code-block:: python from kaiju_tools import run_server, init_app run_server( init_app, base_config_paths=['./settings/config.yml'], base_env_paths=['./settings/env.base.json'], default_env_paths=['./settings/env.local.json'], ) Check out `Basic application`_ on how to construct your application from scratch and also :ref:`rpc-guide` and :ref:`rpc-services-howto` on how to implement and run RPC services. Services -------- The main building block of the app is :py:class:`~kaiju_tools.app.Service` - a class which has specific functions in the app and can interact with other such classes. Services are quasi-independent: they may require each other, some of them may require initialization, but they can be arranged in different ways in the app configuration system. To create a service you must inherit it from :py:class:`~kaiju_tools.app.Service`, register your class in the service classes registry and write it in your app configuration file (see `Config files`_). .. code-block:: python from kaiju_tools import Service, SERVICE_CLASS_REGISTRY class MyService(Service): """My service does something I'm not sure.""" SERVICE_CLASS_REGISTRY.register(MyService) To create a service configuration you must create a :py:class:`~kaiju_tools.app.ServiceSettings` object in the `services` section of your app .yml config file. The simplest config would look like this: .. code-block:: yaml services: - cls: MyService However there are additional useful options for service configuration. .. code-block:: yaml services: - cls: MyService name: my_service_with_different_name enabled: True required: False loglevel: ERROR settings: some_value: 42 - **name** - a custom name for a service, imagine you have two services of the same class but with different settings. You can set custom names to prevent conflict and configure service discovery by name for the dependent services (see `Dependencies`_). - **enabled** - a disabled service won't be initialized. - **required** - this option is different. The app will try to initialize the service, but if it fails, the app will continue anyway. - **loglevel** - override application log level for this service - **settings** - you can put `__init__` parameters here Contextable services ____________________ Sometimes a service may require intricate init or close logic. Use :py:class:`~kaiju_tools.app.ContextableService` as a base to be able to implement :py:meth:`~kaiju_tools.app.ContextableService.init`. It is executed by the app after all the services have been created. There's also :py:meth:`~kaiju_tools.app.ContextableService.close` method executed on app termination. You can use it to close connection pool, cleanup local cache, send notifications, etc. .. code-block:: python from kaiju_tools import ContextableService, SERVICE_CLASS_REGISTRY class MyService(ContextableService): _pool = None async def init(self): self._pool = await self._initialize_pool() async def close(self): if self._pool: self._pool.close() SERVICE_CLASS_REGISTRY.register(MyService) Currently contextable services use `aiohttp cleanup context `_ to call `init` and `close` methods. Dependencies ____________ Where may be a case when your service need to access another service. There is a way to discover services across the app using :py:meth:`~kaiju_tools.app.Service.discover_service` method. .. code-block:: python # will discover a service of class `MyService` my_service = service.discover_service(None, cls=MyService) # will discover a service of class `MyService` but only if its name is 'my_service' my_service = service.discover_service('my_service', cls=MyService) # will discover a service of class `MyService` but if not then `None` will be returned my_service = service.discover_service(None, cls=MyService, required=False) You can use it to get dependencies in your service `__init__`, however it may cause a problem if your service is initialized *before* the dependency. To prevent this you can use :py:class:`~kaiju_tools.app.ContextableService` as base and discover dependencies in :py:meth:`~kaiju_tools.app.ContextableService.init`. .. code-block:: python class MyService(ContextableService): def __init__(self, app, dep: MyDependency = None, logger): ContextableService.__init__(self, app=app, logger=logger) self._dep = dep async def init(self): self._dep = self.discover_service(self._dep, cls=MyDependency) Internal scheduler ------------------ :py:meth:`~kaiju_tools.app.Scheduler.schedule_task` can be used to schedule in-app periodic tasks. You can customize task execution behavior using :py:obj:`~kaiju_tools.app.ExecPolicy`. Use `task.enabled = False` to stop the task from scheduling. .. code-block:: python from kaiju-tools import Scheduler, ContextableService class MyService(ContextableService): async def init(self): _scheduler = self.discover_service(None, cls=Scheduler) self._task = _scheduler.schedule_task( self._reload_local_cache, interval=60, name=f'{self.service_name}._reload_local_cache', policy=_scheduler.ExecPolicy.WAIT) async def close(self): self._task.enabled = False #: disable the task async def _reload_local_cache(self) -> None: ... The scheduler itself can be enabled in config in the `services` block. .. code-block:: yaml services: - cls: Scheduler RPC Server ---------- :py:meth:`~kaiju_tools.rpc.JSONRPCServer` is used to process requests. See :ref:`rpc-guide`. It can be enabled in config in the `services` section. .. code-block:: yaml services: - cls: JSONRPCServer name: rpc settings: default_request_time: 60 max_request_time: 600 enable_permissions: True request_logs: True response_logs: False Loggers ------- Library logger is a modified version of the standard logger. It has the same syntax but will automatically insert client request context (such as correlation id) in a log message. It also allows you to write extra data in plain keyword arguments. For better log aggregation it is recommended to not format the log message itself, providing contextual data as extras. .. code-block:: python self.logger.info('Log in completed.', user_id=user_id) # user_id will be available in extras :py:class:`~kaiju_tools.app.LoggingService` provides a service for all application logging. It allows logging setting to be configured in the project settings file. Example configuration: .. code-block:: yaml services: - cls: LoggingService settings: loggers: - name: "[main_name]" handlers: True handlers: - cls: JSONHandler name: default See :py:class:`~kaiju_tools.app.HandlerSettings` and :py:class:`~kaiju_tools.app.LoggerSettings` for a description of loggers and handlers config parameters. There are two standard formatter classes. :py:class:`~kaiju_tools.logging.TextFormatter` provides better readability and can be used when you need to read from the terminal. There's also a standard handler for this formatter called :py:class:`~kaiju_tools.logging.TextHandler`. Example of standard output: .. code-block:: (time | level | correlation id | service | message | extra data) 12:18:40 | INFO | 135125a5 | app.rpc | rpc accepted | {'request': {'method': 'rpc.api', 'id': 0}} :py:class:`~kaiju_tools.logging.DataFormatter` provides encoded output with extended data. It may be used when collecting application output from stdout and directing it to a log aggregator (logstash etc.). There's also a standard handler for this formatter called :py:class:`~kaiju_tools.logging.JSONHandler`. Example of output: .. code-block:: {"t":1682328051.3172748,"name":"app.rpc","lvl":"INFO","msg":"rpc accepted","cid":"20d35cfc","sid":"6f580b919f00704584a0453a883310be","t_max":1682328142,"data":{"request":{"method":"rpc.api","id":0}}} Config files ------------ Configuration is handled by :py:class:`~kaiju_tools.app.ConfigLoader`. There are two types of configuration files: `.yml` which contain a project structure with all main settings and a description of the services, and `.env.json` files containing variables for the yml files. YML files _________ `*.yml` files contain the basic data structure of a :py:class:`~kaiju_tools.app.ProjectSettings` class. An example of an application config file. Note that you can use templates here (see :ref:`templates-howto`). .. code-block:: yaml packages: - kaiju_db app: debug: "[app_debug]" run: host: "[run_host]" port: "[run_port]" main: name: "[main_name]" version: "1.0" env: "[main_env:'dev']" loglevel: "[main_loglevel:'INFO']" etc: additional_data: 123 services: - cls: MyService name: my_service settings: value: 42 - **packages** contains a list of imported service packages, all service classes from the specified package will be registered automatically. - **app** contains aiohttp `Application `_ parameters. - **run** contains aiohttp `run_app() `_ arguments such as host and port. - **main** must contain :py:class:`~kaiju_tools.interfaces.App` data such as name, version, environment, loglevel etc. - **etc** may contain any information and metadata, it's not directly used by the app - **services** should contain a list of services to be initialized Note that you can use multiple `.yml` files in your project: data of them will be merged. It's a good idea to split a big list of services into a few files. For example, if you have `['settings/config.yml', 'settings/services.yml']` listed in `base_config_paths` of :py:func:`~kaiju_tools.app.run_server` function and you specify additional config files in CLI: .. code-block:: console python -m app -c settings/dev.yml -c settings/local.yml` then the files will be merged in the same order as written: .. code-block:: ['settings/config.yml', 'settings/services.yml', 'settings/dev.yml', 'settings/local.yml'] JSON files __________ `*.env.json` files contain a flat map of config parameters. These parameters are inserted in `.yml` files as template values. .. code-block:: json { "app_debug": true, "main_name": "my_app", "run_host": "localhost", "run_port": 9999, "main_env": "prod", "main_loglevel": "DEBUG" } Note that you can use multiple `.json` files which will be merged, so you can have a main configuration and sub-configurations. For example, if you have `['settings/base.env.json']` listed in `base_env_paths` of :py:func:`~kaiju_tools.app.run_server` function and `['settings/local.env.json]` in `default_env_paths`, and you specify additional env files in CLI: .. code-block:: console python -m app -f settings/executor.env.json -f settings/local.executor.env.json then the files will be merged in the same order as written *and `default_env_paths` will be replaced.* .. code-block:: ['settings/base.env.json', 'settings/executor.env.json', 'settings/local.executor.env.json'] Note that you do not need to set all keys in each file. All existing keys will be merged into a single mapping. Environment variables _____________________ The config loader will auto-load environment variables with same keys as in the `.env` files. A vlue is evaluated using python `eval()` function. .. code-block:: console run_port=9999 python -m app Note that the key must be present in your `.env` files. CLI values __________ You can pass an environment value using `-e =` flag. A value is evaluated using python `eval()` function. .. code-block:: console python -m app -e main_env=test CLI values have maximum priority over all other config sources. Note that the key must be present in your `.env` files. CLI arguments _____________ You can see supported CLI arguments by running .. code-block:: console python -m app --help Loading order _____________ A configuration is gathered and resolved in the following order: - `.yml` files from `base_config_paths` value of :py:func:`~kaiju_tools.app.run_server` function - `.yml` files provided by CLI `-c` - `.env.json` files from `base_env_paths` value of :py:func:`~kaiju_tools.app.run_server` function - `.env.json` files provided by CLI `-f` or by `default_env_paths` value of :py:func:`~kaiju_tools.app.run_server` when no CLI arguments provided - local system environment variables - env values provided by CLI `-e` A :py:class:`~kaiju_tools.app.ProjectSettings` object is produced containing full app configuration. Basic application ----------------- See `kaiju base application `_ template as an example. .. code-block:: | / | ├── app/ | │ ├── services/ | │ │ ├── __init__.py | │ ├── application.py | │ ├── commands.py | │ ├── tables.py | │ ├── views.py | │ ├── __main__.py | │ └── __init__.py | ├── settings/ | │ ├── config.yml | │ ├── services.yml | │ ├── env.json | │ └── env.local.json | ├── docs/ | ├── fixtures/ | ├── tests/ | ├── README.rst | └── requirements.txt `views.py` may contain custom aiohttp web views. `commands.py` may contain custom CLI commands. Use `application.py` to customize the application class and initialize routes. This example is generally a good starting point. .. code-block:: python :caption: application.py from kaiju_tools import App from kaiju_tools.app import init_app as init_app_base from kaiju_tools.http import JSONRPCView from app.tables import METADATA from app.views import MyView __all__ = ['init_app', 'URLS'] URLS = [ ('*', '/public/rpc', JSONRPCView, 'json_rpc_view'), ('*', '/my/view', MyView, 'my_view') ] #: web app routes def init_app(settings, **kws) -> App: _dir = Path(__file__).parent app = init_app_base(settings, attrs={'db_meta': METADATA}, **kws) for method, reg, handler, name in URLS: app.router.add_route(method, reg, handler) app.router.add_route(method, reg + '/', handler, name=name) return app `services` sub-package should contain all of your app custom services. It's a good idea to import all your service classes into `services/__init__.py` to be able to register them later in the :py:class:`~kaiju_tools.app.SERVICE_CLASS_REGISTRY` by a single command. .. code-block:: python :caption: __init__.py :emphasize-lines: 8 from kaiju_tools.app import COMMANDS, SERVICE_CLASS_REGISTRY import app.commands from app import services __version__ = '1.0.0' SERVICE_CLASS_REGISTRY.register_from_module(services) COMMANDS.register_from_module(app.commands) In you main file you should run the server specifying which config and env files are to be used as base. .. code-block:: python :caption: __main__.py from kaiju_tools.app import run_server from app.application import init_app if __name__ == '__main__': run_server( init_app, base_config_paths=[ './settings/config.yml', './settings/services/services.yml' ], base_env_paths=['./settings/env.json'], default_env_paths=['./settings/env.local.json'], )