Hey!
Long time no see 
I’ve a topic to discuss, and I think here is the best place to do so.
Context
As you may know or not, we offer RocketChat and Nextcloud behind Keycloak to our users.
For RocketChat, we use oauth/oidc and saml for Nextcloud.
(Yes we know that RocketChat is abandonning oidc in community edition, let’s not discuss this here, open a new thread if you want to collaborate on keeping the feature in the community for future versions, we plan to work on this as well, unfortunately…)
Problem to solve
With this approach,
There are mainly 3 problems
- group membership and user attributes are only synced at login time
- group creation and user creation are only synced at login time
- group and user deletion are never garbage collected on RocketChat and Nextcloud
And the main problems with these problems are:
- gdpr compliance about user deleted data
- security with user deleted (if a user is deleted and somebody creates the same username account…)
- UX, as an admin, in keycloak, I create a user, a group and assign this user to the group, I expect to see this replicated in Nextcloud instantly
Solution space
are we alone?
A good question to ask, is “are we alone to have this issue?”
Of course not
So basically during this investigation, I realized that actually ldap was used to solve this use case.
And it is already implemented in RocketChat and Nextcloud.
Ldap can be synced ~ 1h in Nextcloud and RocketChat.
And yeah, this is really nice, ldap is indeed a standard for user and group membership directories. So this is really nice solution.
Then you get the ldap for the user sync part, and oidc or saml for user login which is a lot nicer than ldap, especially with 2FA
Problems with ldap
But, yeah ldap is not rainbows and unicorn.
- one more database to store state about your user directory (ldap, keycloak, Nextcloud)
- ldap is another database, and if you want High Availability, it is another layer of complexity
- sync once an hour, can still be problematic in term of UX
- sync once an hour, is not ecological for an organisation that update its directory once a month
- we don’t want to expose user password to the app (might be possible throught conf)
- we depend on the app implementation of ldap and oidc (or saml)
a solution without ldap?
So the requirements for a solution could be this:
- not another database with state to store/sync
- nice in term of UX (change is done, and replicated almost immediately)
- nicer in term of environment
- not deploying ldap is probably nicer in term of eco impact
- not syncing once an hour is probably also nicer
Our proposal that we plan on working
So after some discussions internally, we came up with the following plan.
And we post it here to gather early feedback from orgs like you that can be interested.
(we’ll develop it in go, if you plan to develop it, choose your language, but I’m not here to discuss about the language that we’ll work with)
Phase 1 - entire sync - cron based
The phase 1 of the project is to have a generic sync tool, like ldap, but more generic.
The idea is to pull from a source of truth (keycloak) and push to targeted apps (nextcloud & rocketchat).
This tool can be deployed easily by any org, to sync the entire directory at the frequence they like.
And during phase 1, we’ll develop the following connectors:
- from keycloak
- to Nextcloud
- to RocketChat
We’ll make it modular so if you want to develop a from ldap connector, or a to discourse connector, it shouldn’t be that much work on your side.
(Here I say from, to, but in the end, I’m not sure it is relevant, we’ll see during implementation, but indeed, we just need this)
At the end of phase 1, we have the same as ldap without having to have ldap, it will already be a nice improvement to our current situation.
Phase 2 - granular sync - event based
On a second phase, we can go further.
Keyclaok does have a way to emit events, and there is a plugin to make webhook calls from these events.
Based on the phase 1, we can develop an api around the libs used in phase1, that would sync only the user or the group that is concerned by the event.
The problem with webhooks, is that you can miss some. We could circumvent this by having a persitent queue like nats or rabbitMQ, but this adds complexity. We can also just keep the entire, cron based, sync at a much lower frequency, and it keeps complexity low, and is as resilient.
This phase 2 would be more involving for orgs that want to deploy it, but we think it is a lot nicer in term of environment and UX.
Concept
Overview
┌───────────────────┐
│ │ Directory
│ Keycloak Provider ├─────────────────┐
│ │ │ ┌────────┐
└───────────────────┘ └──►│ │
│ Core ├─────┐
┌────────────────────┐ ┌──►│ │ │
│ │ Directory │ └────────┘ │
│ Nextcloud Provider ├────────────────┘ │
│ │ │
└────────────────────┘ │
▲ │
│ │
│ Changelog │
└──────────────────────────────────────────────┘
Flow
- Get directories from providers
- Optionnaly perfom pre processing
- Generate a changelog of the diff betwen directories
- Optionnaly perform post processing
- Patch provider with the changelog
Feedback ?
We know some librehosters that would be interested in that, and we’d love your feedback
