6.5. Technical Details#

6.5.1. User-ID and Credentials#

The listener runs with the effective permissions of the user listener. If root-privileges are required, listener.SetUID() can be used as a context manager or method wrapper to switch the effective UID.

from listener import SetUID

@SetUID()
def prerun() -> None:
            pass

def postrun() -> None:
    with SetUID(0):
            pass

6.5.2. Internal Cache#

The directory /var/lib/univention-directory-listener/ contains several files:

cache/cache.mdb, cache/lock.mdb

Starting with UCS 4.2, the LMDB cache database contains a copy of all objects and their attributes. It is used to supply the old values supplied through the old parameter, when the function handler() is called.

The cache is also used to keep track, for which object which module was called. This is required when a new module is added, which is invoked for all already existing objects when the Univention Directory Listener is restarted.

On domain controllers the cache could be replaced by doing a query to the local LDAP server, before the new values are written into it. But Managed Node doesn’t have a local LDAP server, so there the cache is needed. Also note that the cache keeps track of the associated listener modules, which is not available from the LDAP.

It also contains the KB 13149 - CacheMasterEntry, which stores the notifier and schema ID.

cache.lock

Starting with UCS 4.2, this file is used to detect if a listener opened the cache database.

cache.db, cache.db.lock

Before UCS 4.2, the BDB cache file contained a copy of all objects and their attributes. With the update to UCS 4.2, it gets converted into an LMDB database.

notifier_id

This legacy file contains the last notifier ID read from the Univention Directory Notifier.

handlers/

For each module the directory contains a text file consisting of a single number. The name of the file is derived from the values of the variable name as defined in each listener module. The number is to be interpreted as a bit-field of HANDLER_INITIALIZED=0x1 and HANDLER_READY=0x2. If both bits are set, it indicates that the module was successfully initialized by running the function initialize(). Otherwise both bits are unset.

The package univention-directory-listener contains several commands useful for controlling and debugging problems with the Univention Directory Listener. This can be useful for debugging listener cache inconsistencies.

6.5.2.1. univention-directory-listener-ctrl#

The command univention-directory-listener-ctrl status shows the status of the Listener. This includes the transaction from the Primary Directory Node in comparison to the last processes transaction. It also shows a list of all installed modules and their status.

The command univention-directory-listener-ctrl resync $name can be used to reset and re-initialize a module. It stops any currently running listener process, removes the state file for the specified module and starts the listener process again. This forces the functions clean() and initialize() to be called one after the other.

6.5.2.2. univention-directory-listener-dump#

The command univention-directory-listener-dump can be used to dump the cache file /var/lib/univention-directory-listener/cache.db. The Univention Directory Listener must be stopped first by invoking systemctl stop univention-directory-listener. It outputs the cache in format compatible to the LDAP Data Interchange Format (LDIF).

6.5.2.3. univention-directory-listener-verify#

The command univention-directory-listener-verify can be used to compare the content of the cache file /var/lib/univention-directory-listener/cache.db to the content of an LDAP server. The Univention Directory Listener must be stopped first by invoking systemctl stop univention-directory-listener. LDAP credentials must be supplied at the command line. For example, the following command would use the machine password:

$ univention-directory-listener-verify \
  -b "$(ucr get ldap/base)" \
  -D "$(ucr get ldap/hostdn)" \
  -y /etc/machine.secret

6.5.2.4. get_notifier_id.py#

The command /usr/share/univention-directory-listener/get_notifier_id.py can be used to get the latest ID from the notifier. This is done by querying the Univention Directory Notifier running on the LDAP server configured through the Univention Configuration Registry Variable ldap/master. The returned value should be equal to the value currently stored in the file /var/lib/univention-directory-listener/notifier_id. Otherwise, the Univention Directory Listener might still be processing a transaction or it might indicate a problem with the Univention Directory Listener

6.5.3. Internal working#

The Listener/Notifier mechanism is used to trigger arbitrary actions when changes occur in the LDAP directory service. In addition to the LDAP server slapd it consists of two other services: The Univention Directory Notifier service runs next to the LDAP server and broadcasts change information to interested parties. The Univention Directory Listener service listens for those notifications, downloads the changes and runs listener modules performing arbitrary local actions like storing the data in a local LDAP server for replication or generating configuration files for non-LDAP-aware local services.

On startup the listener connects to the notifier and opens a persistent TCP connection to port 6669. The host can be configured through several Univention Configuration Registry Variables:

If notifier/server is explicitly set, only that named host is used. In addition, the Univention Configuration Registry Variable notifier/server/port can be used to explicitly configure a different TCP port other then 6669.
Otherwise, on the Primary Directory Node and on all Backup Directory Nodes, only the host named in ldap/master is used.
Otherwise, on all other system roles a host is chosen randomly from the combined list of names in ldap/master and ldap/backup.

This list of Backup Directory Nodes stored in the Univention Configuration Registry Variable ldap/backup is automatically updated by the listener module ldap_server.py.

The following steps occur on changes:

An LDAP object is modified on the Primary Directory Node. Changes initiated on all other system roles are re-directed to the Primary Directory Node.
The UCS-specific overlay-module translog assigns the next transaction number. It uses the file /var/lib/univention-ldap/last_id to keep track of the last transaction number.

As a fallback the transaction number of the last entry from the file /var/lib/univention-ldap/listener/listener or /var/lib/univention-ldap/notify/transaction is used. The module appends the transaction ID, DN and change type to the file /var/lib/univention-ldap/listener/listener.

Referred to as FILE_NAME_LISTENER, TRANSACTION_FILE in the source code.
The Univention Directory Notifier watches that file and waits until it becomes non empty. The file is then renamed to /var/lib/univention-ldap/listener/listener.priv (referred to as FILE_NAME_NOTIFIER_PRIV) and the original files is re-created empty. The transactions from the renamed file are processed line-by-line and are appended to the file /var/lib/univention-ldap/notify/transaction (referred to as FILE_NAME_TF in the source code), including the DN. Since protocol version 3 the notifier also stores the same information within the LDAP server by creating the entry reqSession=ID,cn=translog. After successful processing the renamed file is deleted. For efficient access by transaction ID the index transaction.index is updated.
All listeners get notified of the new transaction. Before UCS 4.3 erratum 427 the information already included the latest transaction ID, DN and the change type. With protocol version 3 only the transaction ID is included.
Each listener opens a connection to the LDAP server running on the UCS system which was used to query the Notifier. With protocol version 3 the listener first queries the LDAP server for the missing DN and change type information by retrieving the entry reqSession=ID,cn=translog. With that it retrieves the latest state of the object identified through the DN. If access is blocked, for example, by selective replication, the change is handled as a delete operation instead.
The old state of the object is fetched from the local Internal Cache located in /var/lib/univention-directory-listener/cache/.
For each module it is checked, if either the old or new state of the object matches the filter and attributes specified in the corresponding Python variables. If not, the module is skipped. By default replication.py is always called first to guarantee that the data is available from the local LDAP server for all subsequent modules. Since UCS 5.0 erratum 164 the order of how modules are called can be configured using the per module property priority.
If the function prerun() of module was not called yet, this is done to signal the start of changes.
The function handler() specified in the module is called, passing in the DN and the old and new state.
The main listener process updates its cache with the new values, including the names of the modules which successfully handled that object. This guarantees that the module is still called, even when the filter criteria would no longer match the object after modification.
On a Backup Directory Node the Univention Directory Listener writes the transaction data to the file /var/lib/univention-ldap/listener/listener (referred to as FILE_NAME_LISTENER, TRANSACTION_FILE in the source code) to allow the Univention Directory Notifier to be cascaded. This is configured internally with the option -o of univention-directory-listener and is done for load balancing and failover reasons.
The transaction ID is written into the legacy local file /var/lib/univention-directory-listener/notifier_id. It also is written into the master record of the listener cache.

After 15 seconds of inactivity the function postrun() is invoked for all prepared modules. This signals a break in the stream of changes and requests the module to release its resources and/or start pending operations.

6.5.4. LDAP Schema handling#

The LDAP Schema is managed on the Primary Directory Node. Extensions must be made available there first. All other systems running LDAP replica download it from there using the Univention Directory Notifier / Univention Directory Listener mechanism.

On the Primary Directory Node the LDAP Schema is extracted by the script /etc/init.d/slapd on each start. The MD5 hash is stored in /var/lib/univention-ldap/schema/md5.
On each change the counter in file /var/lib/univention-ldap/schema/id/id is incremented.
Univention Directory Notifier monitors that file and makes the value available over the network. It can be queried by running /usr/share/univention-directory-listener/get_notifier_id.py -s.
Univention Directory Listener retrieves the value during each transaction. It is stored in the local file /var/lib/univention-ldap/schema/id/id and in the CacheMasterEntry of the Internal Cache.
On change the Listener downloads the current Schema from the LDAP server of the Primary Directory Node, saves it to the local schema file /var/lib/univention-ldap/schema.conf and restarts the local service slapd.
The Listener then continues processing transactions.

6.5.5. Python 3 migration#

Since UCS 5.0 the Univention Directory Listener uses Python 3 to execute listener modules.

For a successful migration all functions must be migrated to work with Python 3. There is no change in the module variables (name, description, filter, …) necessary.

The data structure of the arguments new and old given to the handler() function now explicitly differentiates between byte strings (bytes) and unicode strings (str). The dictionary keys are strings while the LDAP attribute values are list of byte strings:

{
  'associatedDomain': [b'example.net'],
  'krb5RealmName': [b'EXAMPLE.NET'],
  'dc': [b'example'],
  'nisDomain': [b'example.net'],
  'objectClass': [
    b'top',
    b'krb5Realm',
    b'univentionPolicyReference',
    b'nisDomainObject',
    b'domainRelatedObject',
    b'domain',
    b'univentionBase',
    b'univentionObject'
  ],
  'univentionObjectType': [b'container/dc'],
}

While in UCS 4 handler() typically looked like:

def handler(
    dn:  # type: str,
    new,  # type: Dict[str, List[str]]
    old,  # type: Dict[str, List[str]]
):  # type: (...) -> None
    if new and 'myObjectClass' in new.get('objectClass', []):
        value = new['myAttribute'][0]
        ...

In UCS 5 it would look like:

from typing import Dict, List


def handler(
    dn: str,
    new: Dict[str, List[bytes]],
    old: Dict[str, List[bytes]],
) -> None:
    if new and b'myObjectClass' in new.get('objectClass', []):
        value = new['myAttribute'][0].decode('UTF-8')
        ...

Univention Corporate Server - Manual for developers

Technical Details

Contents