Application guide

This library is supposed to provide tools for constructing your asynchronous server application. See it for yourself in app - base application classes.

Application class is currently based on aiohttp app. Basically application start consists of two stages: loading Config files and initialization of the application Services.

To create an app you can use init_app() function. You can also start an aiohttp server by using run_server().

from kaiju_tools import run_server, init_app

run_server(
    init_app,
    base_config_paths=['./settings/config.yml'],
    base_env_paths=['./settings/env.base.json'],
    default_env_paths=['./settings/env.local.json'],
)

Check out Basic application on how to construct your application from scratch and also RPC guide and RPC services HOWTO on how to implement and run RPC services.

Services

The main building block of the app is Service - a class which has specific functions in the app and can interact with other such classes. Services are quasi-independent: they may require each other, some of them may require initialization, but they can be arranged in different ways in the app configuration system.

To create a service you must inherit it from Service, register your class in the service classes registry and write it in your app configuration file (see Config files).

from kaiju_tools import Service, SERVICE_CLASS_REGISTRY

class MyService(Service):
  """My service does something I'm not sure."""

SERVICE_CLASS_REGISTRY.register(MyService)

To create a service configuration you must create a ServiceSettings object in the services section of your app .yml config file. The simplest config would look like this:

services:
  - cls: MyService

However there are additional useful options for service configuration.

services:
  - cls: MyService
    name: my_service_with_different_name
    enabled: True
    required: False
    loglevel: ERROR
    settings:
      some_value: 42

  • name - a custom name for a service, imagine you have two services of the same class but with different settings. You can set custom names to prevent conflict and configure service discovery by name for the dependent services (see Dependencies).

  • enabled - a disabled service won’t be initialized.

  • required - this option is different. The app will try to initialize the service, but if it fails, the app will continue anyway.

  • loglevel - override application log level for this service

  • settings - you can put __init__ parameters here

Contextable services

Sometimes a service may require intricate init or close logic. Use ContextableService as a base to be able to implement init(). It is executed by the app after all the services have been created.

There’s also close() method executed on app termination. You can use it to close connection pool, cleanup local cache, send notifications, etc.

from kaiju_tools import ContextableService, SERVICE_CLASS_REGISTRY

class MyService(ContextableService):

  _pool = None

  async def init(self):
    self._pool = await self._initialize_pool()

  async def close(self):
    if self._pool:
      self._pool.close()

SERVICE_CLASS_REGISTRY.register(MyService)

Currently contextable services use aiohttp cleanup context to call init and close methods.

Dependencies

Where may be a case when your service need to access another service. There is a way to discover services across the app using discover_service() method.

# will discover a service of class `MyService`
my_service = service.discover_service(None, cls=MyService)

# will discover a service of class `MyService` but only if its name is 'my_service'
my_service = service.discover_service('my_service', cls=MyService)

# will discover a service of class `MyService` but if not then `None` will be returned
my_service = service.discover_service(None, cls=MyService, required=False)

You can use it to get dependencies in your service __init__, however it may cause a problem if your service is initialized before the dependency. To prevent this you can use ContextableService as base and discover dependencies in init().

class MyService(ContextableService):

  def __init__(self, app, dep: MyDependency = None, logger):
    ContextableService.__init__(self, app=app, logger=logger)
    self._dep = dep

  async def init(self):
    self._dep = self.discover_service(self._dep, cls=MyDependency)

Internal scheduler

schedule_task() can be used to schedule in-app periodic tasks.

You can customize task execution behavior using ExecPolicy. Use task.enabled = False to stop the task from scheduling.

from kaiju-tools import Scheduler, ContextableService

class MyService(ContextableService):

  async def init(self):
    _scheduler = self.discover_service(None, cls=Scheduler)
    self._task = _scheduler.schedule_task(
      self._reload_local_cache, interval=60, name=f'{self.service_name}._reload_local_cache',
      policy=_scheduler.ExecPolicy.WAIT)

  async def close(self):
    self._task.enabled = False  #: disable the task

  async def _reload_local_cache(self) -> None:
    ...

The scheduler itself can be enabled in config in the services block.

services:
  - cls: Scheduler

RPC Server

JSONRPCServer() is used to process requests. See RPC guide.

It can be enabled in config in the services section.

services:
  - cls: JSONRPCServer
    name: rpc
    settings:
      default_request_time: 60
      max_request_time: 600
      enable_permissions: True
      request_logs: True
      response_logs: False

Loggers

Library logger is a modified version of the standard logger. It has the same syntax but will automatically insert client request context (such as correlation id) in a log message. It also allows you to write extra data in plain keyword arguments.

For better log aggregation it is recommended to not format the log message itself, providing contextual data as extras.

self.logger.info('Log in completed.', user_id=user_id)  # user_id will be available in extras

LoggingService provides a service for all application logging. It allows logging setting to be configured in the project settings file.

Example configuration:

services:
  - cls: LoggingService
    settings:
      loggers:
        - name: "[main_name]"
          handlers: True
      handlers:
        - cls: JSONHandler
          name: default

See HandlerSettings and LoggerSettings for a description of loggers and handlers config parameters.

There are two standard formatter classes.

TextFormatter provides better readability and can be used when you need to read from the terminal. There’s also a standard handler for this formatter called TextHandler.

Example of standard output:

(time | level | correlation id | service | message | extra data)

12:18:40 |  INFO | 135125a5 | app.rpc | rpc accepted | {'request': {'method': 'rpc.api', 'id': 0}}

DataFormatter provides encoded output with extended data. It may be used when collecting application output from stdout and directing it to a log aggregator (logstash etc.). There’s also a standard handler for this formatter called JSONHandler.

Example of output:

{"t":1682328051.3172748,"name":"app.rpc","lvl":"INFO","msg":"rpc accepted","cid":"20d35cfc","sid":"6f580b919f00704584a0453a883310be","t_max":1682328142,"data":{"request":{"method":"rpc.api","id":0}}}

Config files

Configuration is handled by ConfigLoader. There are two types of configuration files: .yml which contain a project structure with all main settings and a description of the services, and .env.json files containing variables for the yml files.

YML files

*.yml files contain the basic data structure of a ProjectSettings class.

An example of an application config file. Note that you can use templates here (see Templates guide).

packages:
  - kaiju_db
app:
  debug: "[app_debug]"
run:
  host: "[run_host]"
  port: "[run_port]"
main:
  name: "[main_name]"
  version: "1.0"
  env: "[main_env:'dev']"
  loglevel: "[main_loglevel:'INFO']"
etc:
  additional_data: 123
services:
  - cls: MyService
    name: my_service
    settings:
      value: 42

  • packages contains a list of imported service packages, all service classes from the specified package will be registered automatically.

  • app contains aiohttp Application parameters.

  • run contains aiohttp run_app() arguments such as host and port.

  • main must contain App data such as name, version, environment, loglevel etc.

  • etc may contain any information and metadata, it’s not directly used by the app

  • services should contain a list of services to be initialized

Note that you can use multiple .yml files in your project: data of them will be merged. It’s a good idea to split a big list of services into a few files.

For example, if you have [‘settings/config.yml’, ‘settings/services.yml’] listed in base_config_paths of run_server() function and you specify additional config files in CLI:

python -m app -c settings/dev.yml -c settings/local.yml` then the files

will be merged in the same order as written:

['settings/config.yml', 'settings/services.yml', 'settings/dev.yml', 'settings/local.yml']

JSON files

*.env.json files contain a flat map of config parameters. These parameters are inserted in .yml files as template values.

{
  "app_debug": true,
  "main_name": "my_app",
  "run_host": "localhost",
  "run_port": 9999,
  "main_env": "prod",
  "main_loglevel": "DEBUG"
}

Note that you can use multiple .json files which will be merged, so you can have a main configuration and sub-configurations.

For example, if you have [‘settings/base.env.json’] listed in base_env_paths of run_server() function and [‘settings/local.env.json] in default_env_paths, and you specify additional env files in CLI:

python -m app -f settings/executor.env.json -f settings/local.executor.env.json

then the files will be merged in the same order as written and `default_env_paths` will be replaced.

['settings/base.env.json', 'settings/executor.env.json', 'settings/local.executor.env.json']

Note that you do not need to set all keys in each file. All existing keys will be merged into a single mapping.

Environment variables

The config loader will auto-load environment variables with same keys as in the .env files. A vlue is evaluated using python eval() function.

run_port=9999
python -m app

Note that the key must be present in your .env files.

CLI values

You can pass an environment value using -e <key>=<value> flag. A value is evaluated using python eval() function.

python -m app -e main_env=test

CLI values have maximum priority over all other config sources. Note that the key must be present in your .env files.

CLI arguments

You can see supported CLI arguments by running

python -m app --help

Loading order

A configuration is gathered and resolved in the following order:

  • .yml files from base_config_paths value of run_server() function

  • .yml files provided by CLI -c

  • .env.json files from base_env_paths value of run_server() function

  • .env.json files provided by CLI -f or by default_env_paths value of run_server() when no CLI arguments provided

  • local system environment variables

  • env values provided by CLI -e

A ProjectSettings object is produced containing full app configuration.

Basic application

See kaiju base application template as an example.

| /
| ├── app/
| │   ├── services/
| │   │   ├── __init__.py
| │   ├── application.py
| │   ├── commands.py
| │   ├── tables.py
| │   ├── views.py
| │   ├── __main__.py
| │   └── __init__.py
| ├── settings/
| │   ├── config.yml
| │   ├── services.yml
| │   ├── env.json
| │   └── env.local.json
| ├── docs/
| ├── fixtures/
| ├── tests/
| ├── README.rst
| └── requirements.txt

views.py may contain custom aiohttp web views.

commands.py may contain custom CLI commands.

Use application.py to customize the application class and initialize routes. This example is generally a good starting point.

application.py
  from kaiju_tools import App
  from kaiju_tools.app import init_app as init_app_base
  from kaiju_tools.http import JSONRPCView

  from app.tables import METADATA
  from app.views import MyView

  __all__ = ['init_app', 'URLS']

  URLS = [
    ('*', '/public/rpc', JSONRPCView, 'json_rpc_view'),
    ('*', '/my/view', MyView, 'my_view')
  ]  #: web app routes

  def init_app(settings, **kws) -> App:
      _dir = Path(__file__).parent
      app = init_app_base(settings, attrs={'db_meta': METADATA}, **kws)
      for method, reg, handler, name in URLS:
          app.router.add_route(method, reg, handler)
          app.router.add_route(method, reg + '/', handler, name=name)
      return app

services sub-package should contain all of your app custom services. It’s a good idea to import all your service classes into services/__init__.py to be able to register them later in the SERVICE_CLASS_REGISTRY by a single command.

__init__.py
 from kaiju_tools.app import COMMANDS, SERVICE_CLASS_REGISTRY

 import app.commands
 from app import services

 __version__ = '1.0.0'

 SERVICE_CLASS_REGISTRY.register_from_module(services)
 COMMANDS.register_from_module(app.commands)

In you main file you should run the server specifying which config and env files are to be used as base.

__main__.py
 from kaiju_tools.app import run_server
 from app.application import init_app

 if __name__ == '__main__':
     run_server(
         init_app,
         base_config_paths=[
             './settings/config.yml',
             './settings/services/services.yml'
         ],
         base_env_paths=['./settings/env.json'],
         default_env_paths=['./settings/env.local.json'],
     )