Identity provider backup and upgrade order

For identity provider backup, back up the database and also the secret keys that are kept outside it, because a database backup on its own is not enough. This page, for the person who runs SSOGEN, explains what SSOGEN consists of, what must be backed up (and why a database backup on its own is not enough), what happens when you upgrade and which order to follow.

Note

The installer sets up SSOGEN as the ssogen service (ssogen-platform still works as its alias) and takes a database backup before every upgrade. SSOGEN does not provide point-in-time recovery or a container image. Follow your organization’s standard practice for those parts.

1. What runs

Part	What it is
SSOGEN	One Java application. The Admin Console, My Apps, My Account, sign-in and every API are in it. The user interface ships inside it. There is no separate worker, scheduler or queue to run.
PostgreSQL	The only datastore. PostgreSQL 15 or later; no other database is supported.
A web server or load balancer in front	Presents SSOGEN’s public address (for example https://sso.acme.example) to users.
SSOGEN Web Agent (optional)	An Apache module on your application servers. Separate component.
SSOGEN Gateway (optional)	The gateway component (App Proxy, Remote Access, Directory Connector). Separate component.

The installer sets up one SSOGEN server. A new installation listens on port 9941 (an existing installation keeps the port it has); behind a load balancer, the balancer connects to that port and shows users the public address.

2. Where configuration lives

SSOGEN keeps configuration in four places, and only two of them are files:

Place	What goes there	Changed how
Environment file (/etc/ssogen/ssogen-platform.env, read at start)	Where the database is and how to reach it; the secret encryption key; the audit digest key; the public address of SSOGEN; what network SSOGEN trusts.	Edit the file, restart. Set at install and rarely changed.
Mounted key files	The signing keystore (the key SSOGEN signs tokens and assertions with), and TLS stores if SSOGEN terminates TLS itself.	Replace the file, restart.
Database: settings	Mail server, retention periods, rate limits, security toggles.	The Admin Console (Security settings; the mail server is set under Settings, SMTP, see Multi-factor authentication). No restart.
Database: stored secrets	Directory bind passwords, the mail password, provisioning tokens. Encrypted.	The Admin Console (Secrets). No restart.

The signing key is not inside the database or the application package.

The environment variables that matter to an operator:

Variable	Purpose
SSOGEN_DATABASE_URL, SSOGEN_DATABASE_USER, SSOGEN_DATABASE_PASSWORD	Where the database is and how to sign in to it. The password can be kept encrypted in the file (a value starting ENC:v1:); an installation made before that option keeps working with a plain one, and sudo /opt/ssogen/install.sh --encrypt-db-password encrypts it (see section 6).
SSOGEN_SECRETS_MASTER_KEY	The key that encrypts every stored secret. Without it the Secrets page cannot store anything.
SSOGEN_SECRETS_PREVIOUS_MASTER_KEY	Only while you are changing the master key. See Secrets.
SSOGEN_AUDIT_DIGEST_KEY	The key that signs the daily audit checkpoints (at least 32 characters). Without it checkpoints cannot be sealed or verified.
SSOGEN_CANONICAL_ORIGIN	The one public address of this SSOGEN installation, such as https://sso.acme.example. Other systems are configured with it. It cannot change once the installation exists.
SSOGEN_TRUSTED_PROXY_CIDRS	The network addresses of your load balancer, whose forwarded client addresses SSOGEN may believe. Empty means forwarded headers are ignored.
SSOGEN_SECRETS_ENV_ALLOWED	Which environment variable names a secret://env/NAME reference may use. Empty (the default) allows none.
SSOGEN_SIGNING_PROVIDER, SSOGEN_SIGNING_LOCAL_KEYSTORE_PATH, SSOGEN_SIGNING_LOCAL_PASSWORD_FILE, SSOGEN_SIGNING_LOCAL_ALIAS	Where the signing key is.
SSOGEN_SIGNING_LOCAL_NHI_ALIAS	The second key in the same keystore that signs AI agent tokens. Needed before AI agents are turned on.
SSOGEN_DPOP_NONCE_KEY	A key (at least 32 characters, identical on every instance) used for AI agent proofs. Unset, AI agents get no tokens.

The System health page lists which of the variables named in SSOGEN’s standard configuration are set or not set, without values (see System health). A variable it does not name, such as the secret encryption key, is not listed there.

3. Starting and checking

For a load balancer or monitoring tool:

Address	Meaning
/health/live	Answers 200 when the process is running.
/health/ready	Answers 200 when SSOGEN is ready for traffic and 503 when it is not (for example, the database cannot be reached).

Inside the console, System health shows the same state with reasons, and the signing key’s expiry date.

Start, stop and restart SSOGEN with sudo systemctl start|stop|restart ssogen; the installation guide in the release shows the checks to run.

If the only administrator loses the password or the phone, the install guide’s section “Break-glass recovery of an administrator” gives a command you run on the server as root. It needs no web server, prints a temporary password and new recovery codes, and revokes every authenticator and sign-in of that administrator, so have a phone with the SSOGEN Authenticator ready to set up again.

4. What to back up, and how to restore

A restorable backup has several parts. Keep them apart on purpose:

Part	Why
The PostgreSQL database	Everything SSOGEN knows: people, applications, policies, the audit log, settings and the encrypted secrets.
The environment file (/etc/ssogen/ssogen-platform.env)	Rebuilds the same settings, including the four keys below.
The four keys (SSOGEN_SECRETS_MASTER_KEY, SSOGEN_TOTP_MASTER_KEY, SSOGEN_AUDIT_DIGEST_KEY, SSOGEN_DPOP_NONCE_KEY)	Without the secret encryption key every stored secret is unreadable and you must type each directory and provisioning password again. Without the authenticator key (SSOGEN_TOTP_MASTER_KEY) every enrolled authenticator stops working, and it cannot be replaced; people enroll again. Without the audit digest key the audit integrity checks cannot verify old digests, and it cannot be rotated. The fourth key (SSOGEN_DPOP_NONCE_KEY) must be identical on every instance. Keep the previous secret encryption key too while one exists.
The signing keystore and its password file (/etc/ssogen/certs/signing.p12, signing.password)	The key applications and agents already trust. Losing it means re-establishing trust everywhere.
The workload CA store and its password file (/etc/ssogen/certs/workload-ca.p12, workload-ca.password)	Every Gateway certificate was issued by it; a new one means enrolling every Gateway again.
The TLS keystore	Replaceable by issuing a new certificate, but easier to restore.
The audit archive folder (/var/lib/ssogen/audit-archive)	Once old audit events are removed from the database, these files are their only copy. A database backup does not hold them.

Caution

The secret keys are kept outside the database on purpose, so that a copy of the database is not a set of working credentials for your directories. Store these keys in your organization’s secret store or a safe place that is not the same place as the database backups.

Practical guidance:

1. Take regular PostgreSQL backups with your usual database tools (for example pg_dump, or snapshots by your backup product). The frequency is your decision: the database holds sign-in sessions and the audit log, so more frequent backups lose less.
2. Back up the key files and the environment file whenever they change, to a separate protected place.
3. Before each upgrade, take a database backup (section 6).
4. Test a restore on a spare machine before you need one.

Because backups can be copied and shared, remember that a backup contains personal data, sign-in history and the audit trail. Protect and dispose of it as you would the live database.

Restoring

1. Stop SSOGEN (sudo systemctl stop ssogen). Restore the dump (pg_restore) into an empty PostgreSQL database.
2. Put the same environment file, the same four keys, the same signing keystore and password file in place (and the same canonical address: it is stored in the database and cannot change).
3. Install the same or a newer release and start SSOGEN. It checks the database schema on start.
4. Open the Admin Console. System health should show Healthy, and on Audit choose Run full verification: it should report the whole chain verified. If it shows Chain broken, see the audit log.

Note

SSOGEN does not provide point-in-time recovery or a disaster-recovery plan; use your database’s own tools and your organization’s plan.

5. Scheduled work that runs by itself

You do not set these up. They run inside SSOGEN:

Work	When
Seal the daily audit checkpoint	Every day at 00:05 UTC.
Verify the whole audit trail	Every day at 02:35 UTC.
Archive and remove old audit events	Every day at 03:30 UTC, when a Super admin has turned retention on (new installations start with it on, keeping 400 days). See The audit log.
Delete expired short-lived records (half-finished sign-ins, expired sessions and similar)	Regularly, according to the retention settings in Security settings.
Directory synchronization	Each LDAP directory runs on its own schedule (15 minutes to 24 hours).
App Proxy certificate expiry check	Regularly. It writes an audit entry when a certificate has 30, 14, 7 or 1 days left. It does not send an email. Watch the audit log or the App Proxy certificates screen.
Webhook delivery	Continuously, when there is something to send. See Webhooks.

6. Upgrading

The installer does the work and rolls back by itself when the new version does not start. The steps are the install guide’s, shown here in the order you do them.

1. Get the new release and check it. Download the archive and its checksum, then in the folder that holds both run sha256sum -c <release>.tar.gz.sha256, tar -xzf <release>.tar.gz and cd <release>.
2. Read the release notes for the new version, and the notes below.
3. Run sudo ./install.sh --check and read the findings.
4. Back up first. The upgrade takes a database dump and a copy of /etc/ssogen for you, in /var/backups/ssogen/pre-upgrade-<old>-to-<new>-<time>/; take your own backup as well and keep it somewhere else.
5. Check the App Proxy hostnames. If you use the App Proxy, list each route’s hosts in the App Proxy console before upgrading and correct any host that is not a valid DNS name, any route with more than 64 hosts on it, and any route id containing ~. The upgrade cannot represent such a host and SSOGEN then refuses sign-in on that host (it fails closed).
6. Run sudo ./install.sh --upgrade, inside tmux or screen. It refuses to start unless the bundle verifies, Java 21 and the PostgreSQL client are present, the database is PostgreSQL 15 or later and there is disk for the backup. It shows its plan and asks, stops the service, backs up, installs the new version and starts it. The database schema is updated automatically when SSOGEN starts; there is no separate migration step. If the new version does not become healthy, the command rolls back by itself: it puts the previous version and the database dump back. If the upgrade is interrupted, run the command again and it finishes or rolls back. The installer upgrades one server. With more than one instance over one database, stop every instance and back up first, then start one and wait until it is healthy before the others; rolling upgrades are not supported.
7. Check. Open System health. The badge should read Healthy, Failed migrations should be 0 and Migrations applied should have increased. Read Setup health for anything the new version asks you to set up.
8. Upgrade the SSOGEN Gateways and Web Agents afterwards. See below. Remove /opt/ssogen.previous and old backups when you are satisfied.

Caution

There is no downgrade command. Schema updates only go forward, and the installer refuses to install an older version over a newer one. If an upgrade fails the installer rolls back for you. If you find a problem later, restore the database backup you took in step 4 and the previous version. Do not run the old version against the upgraded database.

Upgrading from 4.2.1 to 4.3

Follow the steps above, then check these points, which come from the install guide’s upgrade section for 4.3:

• There are no down migrations. Take the database backup first; the way back is that backup and the previous release.
• The service is renamed. The service is now ssogen, with ssogen-platform as its alias. The upgrade renames the service and moves your own override files for you; use sudo systemctl restart ssogen from then on.
• What an application sees does not change. Every App Proxy route saved before 4.3 keeps its earlier Host header choice (external); new routes start with internal.
• Gateways older than 4.3 keep working but report no tunnels, so the fleet page shows none for them. A 4.3 Gateway refuses a configuration file that still has the old relay block or private routes: remove them and set the Outbound Connector address on the Gateway group in the console.
• A new SSH Remote Access resource without a stored host key starts in pairing: nobody can open it until an administrator tests and trusts the host.
• The upgrade switches none of the new features on. No relay link, connector address or automatic certificate exists until an administrator sets it. The firewall changes they need are in the install guide.

Upgrading from 4.1.5 to 4.1.6

Follow the steps above. Then check these points, which come from the install guide’s upgrade section for 4.1.6 and its release notes:

• The port does not change. Version 4.1.5 made 9941 the default for new installations; an upgrade keeps the port in the settings file, and adds SSOGEN_SERVER_PORT="8443" to a file laid out by hand that has no such line. The upgrade plan says so first.
• No sign-in policy changes. A new installation gets a separate console policy; an upgrade to 4.1.6 creates none and changes none. Look at SSO policies > Defaults and consider giving the console its own policy (Defaults).
• Every administrator needs the SSOGEN Authenticator or a passkey. The console accepts, of what its policy lists, an approval in the SSOGEN Authenticator, its code, a recovery code or a passkey as the second step; an email code and a text message code never open it. Before you upgrade, make sure every administrator has one set up under My Account. See Admin lockout protection.
• Webhook receivers must be public https addresses. The address is checked on every delivery, so a receiver at a private or local address stops receiving after the upgrade. Move it to a public address (Webhooks).
• The help desk role (Directory admin) no longer reads Gateway tokens, routes or webhooks.
• The canonical address never changes. It is fixed when the installation is made; an upgrade cannot change it.
• Encrypt the database password afterwards if the settings file still holds it in plain text: run sudo /opt/ssogen/install.sh --encrypt-db-password. The upgrade rewrites no value in the file, so older installations keep a plain password until you do; the command shows what it will do, asks, replaces only that line and puts the old file back if SSOGEN is not healthy afterwards. Delete the old copies of the file it names (they hold the plain password) or keep them as safe as the keys.
• Disk space for the audit trail. An installation upgraded from before the audit trail was stored by month copies it once while the service is stopped, so it is held twice for a while. Check the free space on the database disk against the size of the audit trail first (the install guide gives the query).
• People who sign in through an identity provider are asked for SSOGEN’s second factor after the provider unless the provider’s setting accepts the provider’s own multi-factor sign-in. Existing providers are set to trust a provider that reports its own multi-factor sign-in; check each provider’s Second factor after sign-in with this provider setting and tell people who sign in through a provider that does none to set up the SSOGEN Authenticator under My Account before the upgrade.

Order: SSOGEN first, then SSOGEN Gateways and Web Agents

Upgrade the SSOGEN before the Gateways. An SSOGEN Gateway that is newer than SSOGEN may report things SSOGEN does not yet understand, which makes its status look stale in the console while it is still enforcing. Older Gateways and Web Agents keep working with newer SSOGEN.

Some newer SSOGEN features only reach a Gateway or Web Agent that is up to date. Until you upgrade them, the related routes are not delivered (they are “withheld”) and the audit log records this:

Feature	Needs
App Proxy ingress: listeners, certificates and per-route affinity	Every Gateway in the group must be up to date.
App Proxy back-end sign-in	The Gateway must be up to date.
AI agent (non-human identity) support	The Gateway or Web Agent must be up to date.

If a route does not appear to be in effect after an upgrade, check that your Gateways are up to date under Secure access, Gateways, and the audit log for entries about withheld configuration.

Not provided

• Point-in-time recovery and a downgrade command. Follow your organization’s standard practice for these.