6. Troubleshooting#
When you encounter problems with the operation of the OX Connector app, this section provides information where you can look closer into and to get an impression about what’s going wrong.
6.1. Log files#
The OX Connector app produces different logging information in different places.
- Provisioning Consumer: standard output of the OX Connector container
Contains log information from the Provisioning Consumer about create, update, and delete actions of objects.
It also shows warnings and errors when the OX Connector configuration isn’t correct, or the connector can’t establish a connection to the SOAP API.
$ univention-app logs ox-connector
- Provisioning Service:
/var/log/univention/listener_modules/nubus-provisioning.log Contains log information from the Provisioning Service about the changes it detected in the LDAP directory and delivered to subscribed services. The Provisioning Service containers write additional log information to
/var/log/syslog.
- Database management script:
/var/lib/univention-appcenter/apps/ox-connector/data/univention-ox-connector-task-management.log Contains log information from the Database management script that is described below.
- App Center:
/var/log/univention/appcenter.log Contains log information around activities in the App Center.
The App Center writes OX Connector relevant information to this file, when you run app lifecycle tasks like install, update and uninstall or when you change the app settings.
- Domain join:
/var/log/univention/join.log Contains log information from the join processes. When the App Center install OX Connector, the app also joins the domain.
6.2. Check the Provisioning Consumer#
To troubleshoot the OX Connector, inspect the queue of tasks that the Provisioning Consumer receives from the Provisioning Service.
$ /usr/sbin/univention-ox-connector-task-management summarize-tasks
If the number of pending tasks keeps growing after a change in the LDAP directory, this indicates a problem in the Provisioning Consumer or in the Provisioning Service. For more information, see Log files.
6.3. CLI to monitor the current state#
The OX Connector ships a command-line interface that you can use to query and manipulate the database it uses to keep track of current tasks, objects already synced and errors it may have found.
$ /usr/sbin/univention-ox-connector-task-management --help
The tool operates on the SQLite database
/var/lib/univention-appcenter/apps/ox-connector/data/ox-connector.db.
The terminology of the tool is as follows:
- Tasks#
A database table managed by the OX Connector. A row represents an active task. The OX Connector iterates over all tasks and synchronizes them to the OX App Suite.
- Old#
A database table managed by the OX Connector. A row represents the state of an item at the moment it was successfully synchronized. It is more or less a copy of a former task. Needed when certain items are synchronized and reference other items (e.g., when synchronizing a group that contains users). Also used for faster look-ups by storing the database ID given by OX.
- Morgue#
A database table managed by the OX Connector. A row represents a failed task. It was moved automatically or manually to this table and is not actively processed by the OX Connector. Administrators can examine the items in the morgue and decide how to proceed with them (see below).
6.3.1. Health check#
First, have a look at the log output of the Provisioning Consumer and look for warnings and errors, see Log files.
Second you can get a brief summary of current tasks. This can indicate if the OX Connector can process the items fast enough or at all.
$ /usr/sbin/univention-ox-connector-task-management summarize-tasks
$ /usr/sbin/univention-ox-connector-task-management search-tasks
Third, you can get a brief summary of past errors. Every item is an object not synchronized. Note that this only makes sense should you have chosen OX Connector continues after faulty items.
$ /usr/sbin/univention-ox-connector-task-management search-morgue
6.3.2. Handling errors#
You can decide what to do with the items that have been moved to the morgue.
All commands assume that you have the UniventionObjectIdentifier of that
object. For each item you have the option to
Delete it from the list: It is as if this item never hit the OX Connector. The underlying object can of course be synchronized again if it is modified in the LDAP directory (creating a completely new item in the OX Connector’s tasks).
$ /usr/sbin/univention-ox-connector-task-management remove-from-morgue --obj-id=...
Retry the very same item: The erroneous item in the list is again copied to the list of tasks, assuming that the problem is now fixed (e.g., a validation on the OX App Suite’s side has been disabled).
$ /usr/sbin/univention-ox-connector-task-management retry-from-morgue --obj-id=...
Fresh synchronization of the object: The object is again put into the list of tasks but not with the attributes it had when the synchronization happened (and failed). Instead, it is freshly fetched from the LDAP database. This only works for the first object found, so asterisks may not do what you expect.
$ /usr/sbin/univention-ox-connector-task-management resync-item --obj-id=...
6.4. Provisioning stops working#
When the provisioning stopped working, a previous change in UDM is a probable reason and the OX Connector doesn’t know how to proceed. The connector retries the action over and over again until an administrator repairs the cause manually.
First, see the Log files and look for warnings and errors. If it’s not a temporary problem like for example network connectivity, the fix requires manual action.
As a last resort, the administrator can move the task aside. The log file
reveals the Database ID of that object (e.g. uid=...; $object_identifier;
tasks:$database_id).
$ /usr/sbin/univention-ox-connector-task-management move-task-to-morgue --task-id=$database_id --error-msg="Manual intervention after careful consideration"
6.4.1. Re-provision all data#
Warning
Depending on the number of users and groups in the UCS LDAP directory, this task may take a lot of time.
Reprovisioning all data isn’t recommended.
To re-provision all data, you recreate the subscription of the OX Connector with prefill. The Provisioning Service then sends all existing UDM objects of the subscribed modules to the OX Connector, and the Provisioning Consumer adds them to the provisioning queue.
Run the commands in Listing 6.10 on the Primary Directory Node. The Provisioning Service doesn’t add deleted UDM objects to the queue. Therefore, the OX Connector doesn’t run delete operations during re-provisioning.
$ export BASE_URL="https://$(ucr get ldap/master)/univention/provisioning"
$ export ADMIN_PASSWORD="$(python3 -c 'import json; print(json.load(open("/etc/provisioning-secrets.json"))["PROVISIONING_API_ADMIN_PASSWORD"])')"
$ export SUBSCRIPTION_PASSWORD="$(openssl rand -hex 32)"
$ curl --user "admin:$ADMIN_PASSWORD" \
-X DELETE "$BASE_URL/v1/subscriptions/ox-connector" || true
$ umask 077
$ cat > /tmp/ox-connector-subscription.json <<EOF
{
"name": "ox-connector",
"realms_topics": [
{"realm":"udm", "topic":"users/user"},
{"realm":"udm", "topic":"groups/group"},
{"realm":"udm", "topic":"oxmail/oxcontext"},
{"realm":"udm", "topic":"oxmail/accessprofile"},
{"realm":"udm", "topic":"oxresources/oxresources"},
{"realm":"udm", "topic":"oxmail/functional_account"},
{"realm":"udm", "topic":"oxmail/shared_account"},
{"realm":"udm", "topic":"oxmail/shared_account_permission"}
],
"request_prefill": true,
"password": "$SUBSCRIPTION_PASSWORD"
}
EOF
$ curl --fail --user "admin:$ADMIN_PASSWORD" \
-H "Content-Type: application/json" \
-X POST "$BASE_URL/v1/subscriptions" \
--data @/tmp/ox-connector-subscription.json \
|| { rm -f /tmp/ox-connector-subscription.json; exit 1; }
$ rm -f /tmp/ox-connector-subscription.json
$ printf 'export PROVISIONING_API_USERNAME=ox-connector\nexport PROVISIONING_API_PASSWORD=%s\n' \
"$SUBSCRIPTION_PASSWORD" \
> /var/lib/univention-appcenter/apps/ox-connector/conf/provisioning.env
$ chmod 640 /var/lib/univention-appcenter/apps/ox-connector/conf/provisioning.env
$ univention-app restart ox-connector
Caution
The OX Connector can delete objects
based on the data it receives.
For example, isOxGroup = False in a group object.
6.5. Ensuring the OX database ID integrity#
The internal ID of objects in the database of OX App Suite can become corrupted, for example after a backup restore of the database. For more information about the cache, see Database of old entries.
To rewrite that cache, run the following commands:
$ /usr/sbin/univention-ox-connector-task-management rewrite-ox-db-id
Tip
- Retrieve all users per context in one request
Rebuilding the cache may take a long time and depends on the amount of users in the OX App Suite database.
/usr/sbin/univention-ox-connector-task-management rewrite-ox-db-id --build-cache-size=1000 can speed up the rebuild, because it retrieves up to 1000 users of one context with one request.
Warning
- Memory consumption
On the UCS system with the OX Connector, the rebuild process may use up to 1 GB memory per 10,000 users in the database for OX App Suite.
- System load
Furthermore, the process may generate a lot of load on the OX App Suite system and the OX Connector app.
6.6. Duplicated displaynames#
In OX Connector version 2.2.0 the UDM property oxDisplayName does not have a unique constraint anymore.
If duplicate values are used, but OX is not prepared for that, the SOAP API calls will fail with the following exception.
2023-05-30 11:59:31 WARNING Traceback (most recent call last):
2023-05-30 11:59:31 WARNING File "/tmp/univention-ox-connector.listener_trigger", line 324, in run_on_files
2023-05-30 11:59:31 WARNING f(obj)
2023-05-30 11:59:31 WARNING File "/usr/lib/python3.9/site-packages/univention/ox/provisioning/__init__.py", line 86, in run
2023-05-30 11:59:31 WARNING modify_user(obj)
2023-05-30 11:59:31 WARNING File "/usr/lib/python3.9/site-packages/univention/ox/provisioning/users.py", line 420, in modify_user
2023-05-30 11:59:31 WARNING user.modify()
2023-05-30 11:59:31 WARNING File "/usr/lib/python3.9/site-packages/univention/ox/soap/backend.py", line 477, in modify
2023-05-30 11:59:31 WARNING super(SoapUser, self).modify()
2023-05-30 11:59:31 WARNING File "/usr/lib/python3.9/site-packages/univention/ox/soap/backend.py", line 180, in modify
2023-05-30 11:59:31 WARNING self.service(self.context_id).change(obj)
2023-05-30 11:59:31 WARNING File "/usr/lib/python3.9/site-packages/univention/ox/soap/services.py", line 536, in change
2023-05-30 11:59:31 WARNING return self._call_ox('change', usrdata=user)
2023-05-30 11:59:31 WARNING File "/usr/lib/python3.9/site-packages/univention/ox/soap/services.py", line 163, in _call_ox
2023-05-30 11:59:31 WARNING return getattr(service, func)(**kwargs)
2023-05-30 11:59:31 WARNING File "/usr/lib/python3.9/site-packages/zeep/proxy.py", line 46, in __call__
2023-05-30 11:59:31 WARNING return self._proxy._binding.send(
2023-05-30 11:59:31 WARNING File "/usr/lib/python3.9/site-packages/zeep/wsdl/bindings/soap.py", line 135, in send
2023-05-30 11:59:31 WARNING return self.process_reply(client, operation_obj, response)
2023-05-30 11:59:31 WARNING File "/usr/lib/python3.9/site-packages/zeep/wsdl/bindings/soap.py", line 229, in process_reply
2023-05-30 11:59:31 WARNING return self.process_error(doc, operation)
2023-05-30 11:59:31 WARNING File "/usr/lib/python3.9/site-packages/zeep/wsdl/bindings/soap.py", line 329, in process_error
2023-05-30 11:59:31 WARNING raise Fault(
2023-05-30 11:59:31 WARNING zeep.exceptions.Fault: The displayname is already used; exceptionId 1170523631-4
To fix this issue, a change in the OX App Suite configuration is required.
Add the following lines to the user.properties file.
com.openexchange.user.enforceUniqueDisplayName=false
com.openexchange.folderstorage.database.preferDisplayName=false
Note
This is configured by default in the OX App Suite installation from the App center.
6.7. Missing group members#
When the OX Connector synchronizes a group, it needs the internal ID of all its members, see Database of old entries. It looks the members up in its database of old entries. If a user belongs to a group but isn’t in the database of old entries, the OX Connector doesn’t fail. It skips that user and logs a message as shown in Listing 6.12.
You need to re-provision the user object manually,
in the example uid=oxuser1,cn=users,dc=example,dc=com.
Follow the instructions in Handling errors to synchronize the missing users.
The next time the OX Connector processes the group object,
the Provisioning Consumer takes the user up as group member again.
2024-11-15 16:06:33 INFO Group will be OX Group
2024-11-15 16:06:33 INFO Group wants user as member. But the user is unknown. Ignoring...
6.8. Collect information for support ticket#
Before you open a support ticket, make sure to collect and provide relevant details about your case, so that the Univention Support team can help you:
Provide the relevant messages and tracebacks from Log files, specifically the Provisioning Consumer.
Describe the steps that can reproduce the faulty behavior.
Describe the expected behavior.
Provide data from the provisioning that causes the error.
6.9. Invalid values for OX_USER_IDENTIFIER or OX_GROUP_IDENTIFIER#
Only a UDM user property (or UDM group property in case of OX_GROUP_IDENTIFIER) that contains a single value which is not None is a valid option. In case a UDM property that contains an empty value or a list of values is specified, the OX Connector will enter an error state which needs to be resolved manually by simply setting a valid value.
Setting invalid values for the app settings OX_USER_IDENTIFIER or OX_GROUP_IDENTIFIER will lead to the following errors:
2024-01-11 13:57:39 WARNING Traceback (most recent call last):
2024-01-11 13:57:39 WARNING File "/tmp/univention-ox-connector.listener_trigger", line 351, in run_on_files
2024-01-11 13:57:39 WARNING function(obj)
2024-01-11 13:57:39 WARNING File "/usr/lib/python3.9/site-packages/univention/ox/provisioning/__init__.py", line 86, in run
2024-01-11 13:57:39 WARNING modify_user(obj)
2024-01-11 13:57:39 WARNING File "/usr/lib/python3.9/site-packages/univention/ox/provisioning/users.py", line 454, in modify_user
2024-01-11 13:57:39 WARNING user.modify()
2024-01-11 13:57:39 WARNING File "/usr/lib/python3.9/site-packages/univention/ox/soap/backend.py", line 475, in modify
2024-01-11 13:57:39 WARNING super(SoapUser, self).modify()
2024-01-11 13:57:39 WARNING File "/usr/lib/python3.9/site-packages/univention/ox/soap/backend.py", line 176, in modify
2024-01-11 13:57:39 WARNING assert self.name is not None
2024-01-11 13:57:39 WARNING No name for this attribute. Missing or misconfigured identifier app settings
2024-01-11 13:57:39 WARNING (OX_USER_IDENTIFIER or OX_GROUP_IDENTIFIER) might be the reason, see
2024-01-11 13:57:39 WARNING https://docs.software-univention.de/ox-connector-app/latest/troubleshooting.html#invalid-values-for-ox-user-identifier-or-ox-group-identifier
2024-01-11 13:57:39 WARNING for more information.
setting "users" udm property for groups
2024-01-11 13:59:36 WARNING Traceback (most recent call last):
2024-01-11 13:59:36 WARNING File "/tmp/univention-ox-connector.listener_trigger", line 351, in run_on_files
2024-01-11 13:59:36 WARNING function(obj)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/univention/ox/provisioning/__init__.py", line 108, in run
2024-01-11 13:59:36 WARNING modify_group(new_obj)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/univention/ox/provisioning/groups.py", line 146, in modify_group
2024-01-11 13:59:36 WARNING group.modify()
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/univention/ox/soap/backend.py", line 180, in modify
2024-01-11 13:59:36 WARNING self.service(self.context_id).change(obj)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/univention/ox/soap/services.py", line 607, in change
2024-01-11 13:59:36 WARNING return self._call_ox('change', grp=grp)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/univention/ox/soap/services.py", line 194, in _call_ox
2024-01-11 13:59:36 WARNING return getattr(service, func)(**kwargs)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/proxy.py", line 46, in __call__
2024-01-11 13:59:36 WARNING return self._proxy._binding.send(
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/wsdl/bindings/soap.py", line 123, in send
2024-01-11 13:59:36 WARNING envelope, http_headers = self._create(
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/wsdl/bindings/soap.py", line 73, in _create
2024-01-11 13:59:36 WARNING serialized = operation_obj.create(*args, **kwargs)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/wsdl/definitions.py", line 224, in create
2024-01-11 13:59:36 WARNING return self.input.serialize(*args, **kwargs)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/wsdl/messages/soap.py", line 79, in serialize
2024-01-11 13:59:36 WARNING self.body.render(body, body_value)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/xsd/elements/element.py", line 232, in render
2024-01-11 13:59:36 WARNING self._render_value_item(parent, value, render_path)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/xsd/elements/element.py", line 256, in _render_value_item
2024-01-11 13:59:36 WARNING return self.type.render(node, value, None, render_path)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/xsd/types/complex.py", line 307, in render
2024-01-11 13:59:36 WARNING element.render(node, element_value, child_path)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/xsd/elements/indicators.py", line 256, in render
2024-01-11 13:59:36 WARNING element.render(parent, element_value, child_path)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/xsd/elements/element.py", line 232, in render
2024-01-11 13:59:36 WARNING self._render_value_item(parent, value, render_path)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/xsd/elements/element.py", line 255, in _render_value_item
2024-01-11 13:59:36 WARNING return value._xsd_type.render(node, value, xsd_type, render_path)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/xsd/types/complex.py", line 307, in render
2024-01-11 13:59:36 WARNING element.render(node, element_value, child_path)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/xsd/elements/indicators.py", line 256, in render
2024-01-11 13:59:36 WARNING element.render(parent, element_value, child_path)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/xsd/elements/element.py", line 232, in render
2024-01-11 13:59:36 WARNING self._render_value_item(parent, value, render_path)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/xsd/elements/element.py", line 256, in _render_value_item
2024-01-11 13:59:36 WARNING return self.type.render(node, value, None, render_path)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/xsd/types/simple.py", line 96, in render
2024-01-11 13:59:36 WARNING node.text = value if isinstance(value, etree.CDATA) else self.xmlvalue(value)
2024-01-11 13:59:36 WARNING File "/usr/lib/python3.9/site-packages/zeep/xsd/types/builtins.py", line 27, in _wrapper
2024-01-11 13:59:36 WARNING raise ValueError(
2024-01-11 13:59:36 WARNING ValueError: The String type doesn't accept collections as value