2. Development#

This section is for software developers who want to setup an environment to develop UCS@school ID Connector. It provides an overview of the architecture, the components and their interactions.

_images/id-connector-containers-simplified.svg — Fig. 2.1 ID Connector - Containers (C4 Style)#

Legend

Fig. 2.1 shows the C4 style of Fig. 1.1 from the last chapter Administration:

The school management software that runs on the state level, and exports user data in a file format, for example CSV.
UCS@school import which is a Python script to import users into a DC Primary UCS system.
The DC Primary UCS system passes on the user and group data to the actual ID Connector running in a Docker container.
The ID Connector finally writes user and group data to the school authorities.

Fig. 2.3 is a simplification. It’s on a container level, in the sense used by the C4 model.

Note

Arrows in C4 model diagrams are in the direction of data flow. It should be apparent from the source and target nodes what the label on the arrow refers to.

2.1. Development prerequisites#

This section assumes that you are already familiar with the section Administration, and the Definitions described therein.

You also need the following knowledge to follow this manual and to develop for ID Connector:

HTTP

The foundation of data communication for the internet. The ID Connector APIs use HTTP. You need to understand:

HTTP messages
authentication concepts
error codes

→ https://developer.mozilla.org/en-US/docs/Web/HTTP

Python and pytest

The Python programming language and its testing module. You need to:

program and debug Python modules
test Python source code, ideally using pytest

→ https://www.python.org
→ https://pytest.org

FastAPI

ID Connector uses the FastAPI framework for the APIs. You need to understand:

FastAPI
dependency injection
Pydantic models

→ https://fastapi.tiangolo.com/
→ https://docs.pydantic.dev/latest/

Docker

Software to isolate software and run them in containers. You need to:

understand Dockerfile basics
run containers
understand mounts

→ https://www.docker.com/

Pluggy

Pluggy is the pytest plugin system. You need to understand basic concepts of hook specifications, hook implementation and hook calling.

→ https://pluggy.readthedocs.io/en/latest/

UDM REST API (optional)

A HTTP REST API which you can use to inspect, modify, create, and delete UDM objects through HTTP requests.

You only need to know about the UDM REST API if you want to access extra information about objects within your custom plugin. You need to understand:

the structure of UDM objects.
how to read and maybe write UDM objects, according to your needs.

→ https://docs.software-univention.de/developer-reference/5.0/en/udm/rest-api.html

pre-commit (optional)

A framework for managing and maintaining multi-language pre-commit hooks. You only need pre-commit, if you commit to the Univention ID Connector repository. You need to understand:

how to install pre-commit definitions.
how to run pre-commit checks.
be aware of using different virtual environments for writing code and running pre-commit.

→ https://pre-commit.com/

2.2. Interactions and components#

This section describes the interactions between the components.

2.2.1. Overview, less simplified#

Fig. 2.3 shows the containers for the ID Connector.

_images/id-connector-containers.svg — Fig. 2.3 ID Connector containers#

Legend

Compared to Fig. 2.1, the additional element is Large in-queue on the left in the middle. It’s a folder which interacts as the interface between the Primary Directory Node and the ID Connector. JSON files are written to the folder, and then read out.

The get extra data arrow is an interaction from the ID Connector when it might need extra data that isn’t contained in the JSON files.

2.2.2. Primary Directory Node#

Fig. 2.5 gives a detailed view on the UCS DC component located between UCS@school import and Large in-queue. This section describes the elements in detail.

_images/id-connector-container-ucs.svg — Fig. 2.5 ID Connector Primary Directory Node components#

Legend

The Univention Corporate Server import is a Python script, that reads data such as CSV data, and writes the contained user and group data to the LDAP. As mentioned in the diagram, there are other mechanisms that modify the LDAP, the UMC being one of them. The point is that user and group data somehow arrives.

The LDAP machinery then calls the ID Connector ID Connector listener Python script is. The ID Connector listener handles the write events that are of interest for the ID Connector.

In a first step, the ID Connector listener writes this data to the small in-queue, a folder containing minimal information in JSON format, namely the type of change, such as add, update, delete, and the entryUUID of the concerned object.

The ID Connector listener doesn’t write the data directly to the Converter for the following reasons:

Speed by decoupling - the LDAP listeners should be able to do their job as fast as possible, and shouldn’t have to wait for the next processing steps. Hence, the folder acts as a queue, and only writes minimal data.
The folder can also act as an entry point for debugging and manual insertion of user data. For example, you want to reschedule a user without import the user again.
- To write some JSON data into this folder, use the schedule_user script.
- Or for groups, use the schedule_group script.
- To add JSON data into the folder for all school users and school groups, use the schedule_school script.

The Converter runs as a daemon, picks up the JSON files from the small in-queue, and fetches the actual data from the LDAP using the python-ldap. It then puts a JSON representation of the UDM Object into the Large in-queue.

In turn, the ID Connector running in a Docker container reads the Large in-queue.

2.2.3. ID Connector#

Fig. 2.7 shows the ID Connector component between Large in-queue and the School authority. This section describes the elements in detail.

_images/id-connector-container-docker.svg — Fig. 2.7 ID Connector components#

Legend

As described in Primary Directory Node, the Primary Directory Node writes data to the Large in-queue. The host UCS system and the ID Connector Docker container can access the Large in-queue folder. The Docker container actually mounts the folder.

In Queue is a Python process, that reads the Large in-queue. It may need extra data from the LDAP on the Primary Directory Node, which it fetches using python-ldap. For caching purposes, it uses an SQLite database as a caching mechanism, called the UUID record cache.

The In Queue decides, what user and group data to send where. It uses the School to authority mapping example for decision in the process. For each potential recipient there is a separate Out queue. It writes user and group data in JSON format into the respective folder.

The plugin processes pick up the JSON data, for example Out A. Usually there is only the Kelvin ID Connector plugin, which helps ID Connector to talk to Kelvin REST APIs. The Kelvin plugin process then talks to Kelvin API on the School authority A, doing the final transmission of the user and group data.

The Management REST API orchestrates the processes and manages the outgoing queues.

Hint

To read more about Management REST API, see UCS@school ID Connector HTTP API.

2.2.4. Complete picture#

The complete picture is a bit crowded. If you want see it anyway, here are your choices:

Complete overview, C4 style

_images/id-connector-unified.svg — Fig. 2.9 The ID Connector overview in C4 style#

Legend

Overview, manually drawn, with file locations

_images/ucsschool-id-connector_details2.svg — Fig. 2.11 The ID Connector, *not* simplified.#

2.3. Setup for development#

This section describes the setup of ID Connector for development.

Running the ID Connector requires an LDAP, listeners etc., so you really need a complete UCS installation. Hence, you rather have a local checkout on the development machine, and then synchronize the code changes into an ID Connector container that runs on a virtual machine. Fig. 2.12 shows a C4 model for the relationship between the developer’s local system writing changes to the UCS system used for development.

_images/dev_setup.svg — Fig. 2.12 Setup for development#

Legend

The following sections describe the setup for development for the ID Connector:

You have a git checkout of the ucsschool-id-connector repository on your development machine.
To synchronize the changes, you use the script devsync to synchronize changes on your development machine.
You synchronize the changes to the corresponding installation folder of the ID Connector Docker container.

Hint

If you don’t have devsync, a Univention internal script from the toolshed repository, you might as well use scp, rsync, or any other transfer mechanism of your liking to copy changes to a remote system.

2.3.1. Machine#

To set up the local development environment, run the following commands. They create the directory venv with a Python virtual environment with the app and all its dependencies in it.

# clone ucsschool-id-connector
$ cd ucsschool-id-connector
$ make setup_devel_env
$ . venv/bin/activate
$ make install
$ pre-commit run -a

To activate the Python virtual environment in the venv directory, run the following command:

$ . venv/bin/activate

Warning

All commands in the Makefile assume that you activated the Python virtual environment.

To see the commands, run make without argument:

$ make

 clean                 remove all build, test, coverage and Python artifacts
 clean-build           remove build artifacts
 clean-pyc             remove Python file artifacts
 clean-test            remove test and coverage artifacts
 setup_devel_env       setup development environment (virtualenv)
 lint                  check style (requires Python interpreter activated from venv)
 format                format source code (requires Python interpreter activated from venv)
 test                  run tests with the Python interpreter from 'venv'
 coverage              check code coverage with the Python interpreter from 'venv'
 coverage-html         generate HTML coverage report
 install               install the package to the active Python's site-packages

2.3.2. Virtual machine#

You need to install the ID Connector app through the Univention App Center on your UCS system for development in a virtual machine.

After installation, the Univention App Center starts the ID Connector container. To enter the container of the app, use the following command:

$ univention-app shell ucsschool-id-connector

Inside the container, you can use the Python provided by the container’s system. Run the following commands from inside the directory ucsschool-id-connector.

$ python3
Python 3.8.2 (default, Feb 29 2020, 17:03:31)
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ucsschool_id_connector import models

To synchronize your local working copy into the running ID Connector container on the remote development virtual machine, use the following steps:

Stop the ID Connector in it’s container on the remote development virtual machine:

$ univention-app shell ucsschool-id-connector \
  /etc/init.d/ucsschool-id-connector stop

Find out container’s ID on the remote development virtual machine:

$ docker inspect --format='{{.GraphDriver.Data.MergedDir}}' \
    "$(ucr get appcenter/apps/ucsschool-id-connector/container)"

→  /var/lib/docker/overlay2/8dc...387/merged

On your local developer machine, use container ID to synchronize the local files into the remote container:

$ devsync -v src/ \
  "$IP_OF_DEV_VM":/var/lib/docker/overlay2/8dc...387/merged/ucsschool-id-connector/

Restart and prepare the container for development. The commands perform the following steps:

On the remote development system:

Enter the ID Connector container.

In the ID Connector container:

Install the requirements for development.
Install the ID Connector from the source files in development mode.
Start the ID Connector.
Stop the HTTP REST API.
Start the HTTP REST API in auto-reloading development mode.
Schedule a user.
Schedule a group.
Schedule a school with a number of parallel tasks.

$ univention-app shell ucsschool-id-connector

$ python3 -m pip install --no-cache-dir -r src/requirements.txt -r src/requirements-dev.txt

$ python3 -m pip install -e src/

$ /etc/init.d/ucsschool-id-connector restart

$ /etc/init.d/ucsschool-id-connector-rest-api stop

$ /etc/init.d/ucsschool-id-connector-rest-api-dev start

$ src/schedule_user demo_teacher

$ src/schedule_group demo_class

$ src/schedule_school DEMOSCHOOL 32

# DEBUG: Searching LDAP for user with username 'demo_teacher'...
# INFO : Adding user to in-queue: 'uid=demo_teacher,cn=lehrer,cn=users,ou=DEMOSCHOOL,dc=uni,dc=dtr'.
# DEBUG: Done.

You find logging information in /var/log/univention/ucsschool-id-connector/queues.log.

2.3.3. Run unit tests#

Unit tests run as part of the build process. To start the units tests manually inside the ID Connector Docker container, run the following commands:

root@ucs-host:# univention-app shell ucsschool-id-connector
$ cd src/
$ python3 -m pytest -l -v tests/unittests
$ exit

2.4. Plugin development#

This section describes how to develop a plugin for ID Connector.

2.4.1. How does the plugin system work?#

You can enhance the UCS@school ID Connector through plugins. ID Connector uses the pluggy plugin system to define, implement, and call plugins.

2.4.2. A simple custom plugin#

The following demonstrates a simple example of a custom plugin for ID Connector.

You find the directory structure for your custom plugins and packages on the host system in /var/lib/univention-appcenter/apps/ucsschool-id-connector/conf/. For packages, see the Advanced example below:

/var/lib/univention-appcenter/apps/ucsschool-id-connector/conf/plugins/packages/
/var/lib/univention-appcenter/apps/ucsschool-id-connector/conf/plugins/plugins/

You can put a file containing a plugin class into the plugins/plugins directory. For example, you can save the following content into a file called myplugin.py:

from ucsschool_id_connector.utils import ConsoleAndFileLogging
from ucsschool_id_connector.plugins import hook_impl, plugin_manager
logger = ConsoleAndFileLogging.get_logger(__name__)

class MyPlugin:

    @hook_impl
    def get_listener_object(self, obj_dict):
        logger.info("Myplugin runs get_listener_obj with %r", obj_dict)

plugin_manager.register(MyPlugin())

Restart the ID Connector:

$ univention-app restart ucsschool-id-connector

Validate the queues log file in the directory /var/log/univention/ucsschool-id-connector/queues.log and find entries like this:

2021-12-13 14:32:52 INFO  [ucsschool_id_connector.plugin_loader.load_plugins:79] Loaded plugins:
[...]
2021-12-13 14:32:52 INFO  [ucsschool_id_connector.plugin_loader.load_plugins:81]     'myplugin.MyPlugin': ['get_listener_object']

The entries tell you that the ID Connector found your plugin MyPlugin and the hook implementation for get_listener_object().

2.4.3. Advanced example#

In this example, you learn how to additionally:

define your own hook specifications.
use an extra package for shared code across plugins.

The directory structure for a custom plugin dummy and custom package example_package inside /var/lib/univention-appcenter/apps/ucsschool-id-connector/conf/ looks as the following:

.../plugins/
.../plugins/packages
.../plugins/packages/example_package
.../plugins/packages/example_package/__init__.py
.../plugins/packages/example_package/example_module.py
.../plugins/plugins
.../plugins/plugins/dummy.py

Note

Putting the example_package into the packages directory solves an import problem. The module loader in the plugin_loader.py file appends the packages directory to the Python sys.path. The ID Connector imports packages herein without being properly installed.

Listing 2.1 Content of plugins/packages/example_package/example_module.py#

#
# An example Python module that will be loadable as "example_package.example_module"
# if stored in 'plugins/packages/example_package/example_module.py'.
# Do not forget to create 'plugins/packages/example_package/__init__.py'.
#

from ucsschool_id_connector.utils import ConsoleAndFileLogging

logger = ConsoleAndFileLogging.get_logger(__name__)


class ExampleClass:
    def add(self, arg1, arg2):
        logger.info("Running ExampleClass.add() with arg1=%r arg2=%r.", arg1, arg2)
        return arg1 + arg2

Listing 2.2 Content of plugins/plugins/dummy.py#

#
# An example plugin that will be usable as "plugin_manager.hook.dummy_func()".
# It uses a class from a module in a custom package:
# plugins/packages/example_package/example_module.py
#

from ucsschool_id_connector.utils import ConsoleAndFileLogging
from ucsschool_id_connector.plugins import hook_impl, hook_spec, plugin_manager
from example_package.example_module import ExampleClass

logger = ConsoleAndFileLogging.get_logger(__name__)

class DummyPluginSpec:
    @hook_spec(firstresult=True)
    def dummy_func(self, arg1, arg2):
        """An example hook."""

class DummyPlugin:
    @hook_impl
    def dummy_func(self, arg1, arg2):  # <-- this must match the specification!
        """
        Example plugin function.

        Returns the sum of its arguments.
        Uses a class from a custom package.
        """
        logger.info("Running DummyPlugin.dummy_func() with arg1=%r arg2=%r.", arg1, arg2)
        example_obj = ExampleClass()
        res = example_obj.add(arg1, arg2)
        assert res == arg1 + arg2
        return res


# register plugins
plugin_manager.register(DummyPlugin())

Upon app startup, the ID Connector discovers all plugins and logs them in the log file. You find successful messages like this in the queues log file in the /var/log/univention/ucsschool-id-connector/queues.log directory:

...
INFO  [ucsschool_id_connector.plugins.load_plugins:83] Loaded plugins: {.., <dummy.DummyPlugin object at 0x7fa5284a9240>}
INFO  [ucsschool_id_connector.plugins.load_plugins:84] Installed hooks: [.., 'dummy_func']
...

2.5. Build artifacts#

This section describes how to build the ID Connector artifacts, such as the Docker image and the release image.

2.5.1. Build Docker image#

The repository contains a Dockerfile that you can use to build a Docker image.

Warning

Don’t use the image for production. It’s suitable for testing and development purposes.

Listing 2.3 Manually start the ID Connector Docker container#

$ docker run -p 127.0.0.1:8911:8911/tcp --name ucsschool_id_connector \
  docker-test-upload.software-univention.de/ucsschool-id-connector:$(cat VERSION.txt)

Note

When you start the ID Connector Docker container manually as shown in Listing 2.3, and not through the Univention App Center, you need to stop the local firewall with service univention-firewall stop and can then access the container through the URL https://FQDN:8911/ucsschool-id-connector/api/v1/docs.

You can also:

# let it run in the background.
$ docker run -d ...

# see the stdout
$ docker logs ucsschool_id_connector

# stop the running container
$ docker stop ucsschool_id_connector

# remove the container
$ docker rm ucsschool_id_connector

To enter the running container run:

$ docker exec -it ucsschool_id_connector /bin/ash

2.5.2. Build release image#

Warning

You need to be a software developer at Univention to use this section.

To build a release image, use the following steps:

Update the app version in VERSION.txt.
Add an entry to the changelog in src/HISTORY.rst.
Adjust the app ini file, if needed.
The repository pipeline builds the Docker image automatically.
Use the dedicated Jenkins job. The job tags the image and also updates the Docker image in the App Provider Portal.
Verify that the Jenkins job correctly set the tag for the Docker image of the app in the App Provider Portal.

2.6. Integration tests#

Univention has automated integration tests using Jenkins. The Jenkins configuration file locates at univention/univention-corporate-server. If you want to manually set up integration tests, you need to look there for hints on how to do it.

UCS@school ID Connector 2.3.3 documentation

Development

Contents

2. Development#

2.1. Development prerequisites#

2.2. Interactions and components#

2.2.1. Overview, less simplified#

2.2.2. Primary Directory Node#

2.2.3. ID Connector#

2.2.4. Complete picture#

2.3. Setup for development#

2.3.1. Machine#

2.3.2. Virtual machine#

2.3.3. Run unit tests#

2.4. Plugin development#

2.4.1. How does the plugin system work?#

2.4.2. A simple custom plugin#

2.4.3. Advanced example#

2.5. Build artifacts#

2.5.1. Build Docker image#

2.5.2. Build release image#

2.6. Integration tests#