﻿# WorkSmartHub on Docker and Kubernetes

> This repository builds the WorkSmartHub container image and runs it with Docker or on Kubernetes, and this document is the operator guide for both. WorkSmartHub (by WorkWise Solutions) is one web application: the ASP.NET Core server and the Angular portal are served from the same container on…

Source: https://workwise-solutions.net/docs/installation/operator-guide/ · Language: en · Hungarian version: https://workwise-solutions.net/hu/docs/installation/operator-guide/

This repository builds the WorkSmartHub container image and runs it with Docker or on Kubernetes, and this document is
the operator guide for both. WorkSmartHub (by WorkWise Solutions) is one web application: the ASP.NET Core server and
the Angular portal are served from the same container on the same port.

## What this repository contains

| Path | What it is |
|------|------------|
| `build/build-image.sh`, `build/build-image.ps1` | Builds the image `worksmarthub:<version>` from the Backend and Frontend source folders (bash or PowerShell) |
| `docker/linux/` | `worksmarthub-run.sh` (start the container), `worksmarthub-service-install.sh` / `worksmarthub-service-uninstall.sh` (systemd service) |
| `docker/win/` | The same for Windows with Docker Desktop: `worksmarthub-run.ps1`, `worksmarthub-service-install.ps1` / `worksmarthub-service-uninstall.ps1` (Scheduled Task) |
| `kubernetes-helm-chart/` | The Helm chart `worksmarthub`: the application, a persistent data volume, an optional built-in PostgreSQL and Redis, Ingress, NetworkPolicies |
| `test-local/` | Test scripts for a local Docker daemon and minikube (see [test-local/README.md](https://github.com/WorkWise-Solutions/WorkSmartHub-Docker-Kubernetes/blob/main/test-local/README.md)) |

## Prerequisites

- **Docker:** Docker Engine 20.10 or newer on Linux, or Docker Desktop on Windows (details under [Docker prerequisites](#docker-prerequisites)).
- **Kubernetes:** a cluster, `kubectl` and Helm 3 (details under [Kubernetes prerequisites](#kubernetes-prerequisites)).
- **Building the image:** the .NET SDK of the Backend (`WorkSmartHub/global.json`), Node.js with npm, Docker, and bash
  (Git Bash on Windows) or PowerShell 5.1 or 7.
- The image carries the ASP.NET Core 10.0 runtime on Ubuntu 24.04 LTS and runs as the non-root user `appuser`
  (uid 1001). The hosts need no .NET runtime; a runtime update arrives with a new image.

## Building the image

There is no registry and no pipeline: the image is built locally from the two source folders.

```bash
# from this repository, with the sources in the sibling folders ../WorkSmartHub-Backend and ../WorkSmartHub-Frontend
bash build/build-image.sh --version 23
```

```powershell
.\build\build-image.ps1 -Version 23
```

The script

1. builds the Frontend (`npm ci && npm run build`; the bundle with its `.br` and `.gz` files lands in its `dist/`);
   `--skip-frontend-build` / `-SkipFrontendBuild` reuses an existing `dist/`;
2. copies the Backend sources into a temporary staging folder (without `.git`, `bin`, `obj`, local databases, logs and
   File Storage), puts the Frontend bundle into the copy's `WorkSmartHub/wwwroot/Frontend`, and writes the version as
   `ReleaseId` into the copy's `wwwroot/version.json` (the release number the application shows; only a positive
   integer is written, a tag such as `23-rc1` leaves the file as it is);
3. runs `dotnet publish -c Release` on the copy;
4. runs `docker build --pull` with the `Dockerfile` of the publish output and tags `worksmarthub:<version>` and
   `worksmarthub:latest`.

The source folders are never changed: the build regenerates files (the API client, `appsettings.template.json`), and
that happens in the staging copy, which is removed afterwards (`--keep-staging` / `-KeepStaging` keeps it). The image
gets the labels `org.opencontainers.image.version` and the Git revisions of both source folders
(`com.workwise.worksmarthub.backend.revision`, `...frontend.revision`, with `+dirty` for uncommitted changes).

| Option (bash / PowerShell) | Default | Meaning |
|----------------------------|---------|---------|
| `--backend <dir>` / `-Backend` | `../WorkSmartHub-Backend` | The Backend folder (with its `submodules/WorkWise.Common`) |
| `--frontend <dir>` / `-Frontend` | `../WorkSmartHub-Frontend` | The Frontend folder |
| `--skip-frontend-build` / `-SkipFrontendBuild` | | Reuse `<frontend>/dist` |
| `--changelog <file>` / `-Changelog <file>` | `../WorkSmartHub-Changelog/changelog.html` | Release notes shown in the portal, copied into the image as `wwwroot/changelog.html` |
| `--version <v>` / `-Version` | `ReleaseId` of `WorkSmartHub/wwwroot/version.json` if positive | The image tag. A working tree has `-1` there, so give the version |
| `--repository <name>` / `-Repository` | `worksmarthub` | The image name, e.g. `docker.io/yourorg/worksmarthub` for a registry |
| `--no-latest` / `-NoLatest` | | Do not move the `latest` tag |
| `--minikube` / `-Minikube` | | Build into minikube's Docker daemon; `--minikube-profile <name>` / `-MinikubeProfile` for another profile |
| `--save <file.tar>` / `-Save` | | Also write the image to a tar file |
| `--no-cache` / `-NoCache` | | Build without the layer cache, so the image gets the newest Ubuntu security updates |
| `--staging <dir>` / `-Staging` | a new temporary folder | Where the copy and the publish output go (must be empty) |
| `--keep-staging` / `-KeepStaging` | | Keep the staging folder |

**Another host.** Copy the image as a file: build with `--save worksmarthub-23.tar`, copy the file, then
`docker load -i worksmarthub-23.tar` on the target host.

**minikube.** `--minikube` builds the image directly in minikube's Docker daemon (`eval $(minikube docker-env)`), so the
cluster finds it without a registry. `minikube image load` is not an alternative for an image of this size on Windows:
it fails there.

```bash
minikube start --driver=docker
bash build/build-image.sh --version 23 --minikube
```

**A registry later.** Build with `--repository registry.example.com/worksmarthub`, then `docker push` both tags, and set
the same repository in the Docker scripts (`WSH_IMAGE` or `--image`) and in the chart (`app.image.repository`).

**What is in the image.** The publish output under `/app` (entry `dotnet WorkSmartHub.dll`, owned by `appuser`), the
listen port 8080, and the data folder `/app/data` with the links `/app/database.db`, `/app/FileStorage` and `/app/Logs`
into it (see [Data persistence](#data-persistence)). The image has `bash`, `sh`, `tar` and `openssl`, but no `curl` or
`wget`, and no `HEALTHCHECK` instruction (see [Health checks](#health-checks)).

---

## Docker

The `docker/linux` and `docker/win` folders contain scripts that run the application as a Docker container, in the
foreground, in the background, or as a service that starts at boot.

### Docker prerequisites

- **Linux:** Docker Engine 20.10 or newer (`docker version`), installed from Docker's repository for your distribution
  and enabled at boot (`sudo systemctl enable --now docker`). Run the scripts with `sh`; the service install and
  uninstall scripts need root (`sudo sh ...`).
- **Windows:** Docker Desktop with the WSL 2 backend, running Linux containers. The run script needs no administrator
  rights; the service install and uninstall scripts do.
- The image `worksmarthub:<version>` on the host: built there ([Building the image](#building-the-image)) or loaded with
  `docker load -i`.
- Optional: an `appsettings.json` and a `worksmarthub.env` next to the run script (see [Configuration](#configuration)).
- For a service install, a script folder that other users of the host cannot write to (see
  [Protecting the script folder](#protecting-the-script-folder)).

### Configuration

The application reads `/app/appsettings.json` and environment variables; **an environment variable overrides the
file**. The variable name is the settings path with `__` between the sections and `__0`, `__1` for array elements:
`ConnectionStrings__InternalDb`, `CacheSettings__Type`, `HttpSettings__Urls__0`, `PublicBaseUrl`.

Without any file the container runs with the `appsettings.json` shipped in the image: SQLite in the data volume, the
in-memory cache, local users. The run script sets the listen address `HttpSettings__Urls__0=http://+:8080` in every
case. Two ways to configure more:

- **`worksmarthub.env` next to the run script:** environment variables, one `NAME=value` per line, no quotes and no
  `export`. The script passes it with `--env-file`. It suits secrets; keep it readable by administrators only
  (`sudo chown root:root worksmarthub.env && sudo chmod 600 worksmarthub.env`).
- **`appsettings.json` next to the run script:** the whole configuration file, mounted at `/app/appsettings.json`.
  Start from the shipped file of your image:
  ```bash
  docker run --rm --entrypoint cat worksmarthub:23 /app/appsettings.json > appsettings.json
  ```
  At every start the application merges the settings keys of a newer release into the file and keeps your values
  (first it writes a backup `appsettings.json.backup.<time>` next to it inside the container's `/app`). For that the
  application (uid 1001) must be able to write the file: on Linux
  `sudo chown 1001:root appsettings.json && sudo chmod 600 appsettings.json`. A root-owned file with mode 644 works too:
  the application then logs `appsettings.json is read-only; new settings keep their defaults until the file is updated`
  and uses the new keys with their defaults in memory. A root-owned file with mode 600 or 640 cannot be read and the
  start fails. Never make the file world-writable.

The file is JSON with comments; every setting is explained in it. The sections:

| Section | Purpose |
|---------|---------|
| `ConnectionStrings` | The internal database: `InternalDbType` (0 SQLite, 1 SQL Server, 2 PostgreSQL, 3 MySQL), `InternalDb` (connection string), `Schema` (SQL Server and PostgreSQL only; must be empty for SQLite and MySQL), `MySqlServerVersion` (MySQL only) |
| `CacheSettings` | `Type`: `InMemory` (default), `None`, `Redis`, `SqlServer`, `Postgres`, `MySql`, with a settings block for each |
| `PublicBaseUrl` | The address users open the application at (see below) |
| `HttpSettings` | `Urls` (listen addresses; in the container `http://+:8080`), `Cors` |
| `Swagger` | `Enabled`: the interactive API description at `/swagger` |
| `Serilog` | Log sinks (console and the daily files `Logs/log-.txt`) and log levels |
| `UseDetailedExceptions` | Error details in API error responses |
| `UserSettings` | The user store: `Local` (default), `LDAP`, `AzureAD` (Microsoft Entra ID), `ActiveDirectory` |
| `LoginSettings` | `AuthType` (`Local`, `ActiveDirectory`, `AppleConnect`, `OAuth`), session length, OAuth providers (Okta, Google, Microsoft, Custom), the sign-in cookie, the JWT signing key |
| `Notifications` | The Microsoft Teams service URL |
| `EmailSettings` | The SMTP server for e-mails sent by the application |
| `TicketAttachment`, `ImageConfiguration` | Link signing keys, link lifetime and upload chunk sizes of ticket attachments and component images |
| `ConfigSync` | How often the nodes pick up configuration changes made in the Admin UI (default 10 seconds) |

Everything else (File Storages, notification channels, the Számlázz.hu connection, service user catalogues, security
headers, background jobs, look and feel) is configured in the Admin UI and stored in the database.

#### Settings to set before the first real use

| Setting | Why |
|---------|-----|
| `LoginSettings.JwtAuthenticationSettings.SymmetricSecurityKey` | Signs sign-in and personal access tokens. The shipped value is public: with it anyone could forge a token for any user. Changing it later signs everyone out and invalidates the access tokens. |
| `TicketAttachment.DownloadLink.SymmetricSecurityKey` | Signs ticket attachment download links. Changing it later invalidates the links of every attachment uploaded before. |
| `ImageConfiguration.ImageLink.SymmetricSecurityKey` | Signs component image links. Changing it later invalidates the links of every image uploaded before. |
| `PublicBaseUrl` | e.g. `https://wsh.example.com/`. Links in e-mails (password reset, unsubscribe, notification footers, also those sent by background jobs), the links stored for later e-mails and the mobile activation QR code are built from it. Empty: they are built from the address of each request, which the sender of the request can choose. |
| `EmailSettings` | The SMTP host, port, user, password and sender, if the application sends e-mails. |
| `UseDetailedExceptions` | `true` in the shipped file. Set `false` in production so API errors carry no internal details. |

Use a different long random value for each key, for example `openssl rand -base64 48`, and the same values on every
node. The application logs a `SECURITY:` warning at every start while a key still has the shipped value or
`PublicBaseUrl` is empty. In `worksmarthub.env`:

```
LoginSettings__JwtAuthenticationSettings__SymmetricSecurityKey=<long random value>
TicketAttachment__DownloadLink__SymmetricSecurityKey=<another long random value>
ImageConfiguration__ImageLink__SymmetricSecurityKey=<a third long random value>
PublicBaseUrl=https://wsh.example.com/
UseDetailedExceptions=false
```

> A mounted file that uses the newer `InternalDatabaseSettings` section instead of `ConnectionStrings` takes precedence
> over the `ConnectionStrings` section and over `ConnectionStrings__*` environment variables; use
> `InternalDatabaseSettings__*` variables then.

#### Log levels

| Channel | `appsettings.json` key | Environment variable |
|---------|------------------------|----------------------|
| Default (the application) | `Serilog:MinimumLevel:Default` | `Serilog__MinimumLevel__Default` |
| Microsoft (ASP.NET Core, Entity Framework) | `Serilog:MinimumLevel:Override:Microsoft` | `Serilog__MinimumLevel__Override__Microsoft` |
| System | `Serilog:MinimumLevel:Override:System` | `Serilog__MinimumLevel__Override__System` |

- Valid values: `Verbose`, `Debug`, `Information`, `Warning`, `Error`, `Fatal`, in any letter case. Without a setting the
  level is `Information`; `Warning` is a good choice for the Microsoft channel in production.
- Changes in the file or the environment apply at the next start.
- In the Admin UI, **App Konfiguráció > Log Szintek** has the switch **Szerver naplózási szintek felülírása**. Off: the
  server configuration above applies. On: the three levels chosen on the page apply on every node within the
  `ConfigSync` interval, without a restart. Every change is logged with the name of the administrator.
- Request logs never contain the values of credential query parameters (`access_token`, link tokens, OAuth `code` and
  `state`, and any parameter whose name contains token, secret, auth, key, password or session); they are written as `***`.
- **Never leave `Debug` or `Verbose` on in production:** they write much more data, which can include personal data,
  and fill the disk quickly.

#### Security settings

- **Signing keys and `PublicBaseUrl`:** see the table above.
- **Swagger:** `Swagger:Enabled` (`true` in the shipped file) serves the API description and test UI at `/swagger`; the
  API endpoints still require authentication. With `false` every `/swagger` path answers 404. Set
  `Swagger__Enabled=false` where nobody needs it.
- **Security headers:** set in the Admin UI, **App Konfiguráció > Biztonsági fejlécek**, and applied on every node.
  Defaults: `X-Frame-Options`, `X-Content-Type-Options: nosniff`, `Referrer-Policy: no-referrer`, `Permissions-Policy`
  and a `Content-Security-Policy` on; `Strict-Transport-Security` (HSTS) off. Turn HSTS on once the application is
  reached only over HTTPS (default max-age 30 days).
- **Sign-in cookie:** `LoginSettings.CookieAuthenticationSettings.SecureCookie` is `true`: browsers send the cookie over
  HTTPS only. Serve the application over HTTPS (TLS at the reverse proxy, the load balancer or the Ingress).
- **CORS:** `HttpSettings.Cors.AllowOrigin` is `http://localhost:4200` (the Frontend development server) in the shipped
  file. The portal is served from the same origin as the API and needs no CORS; do not open it to `*`.
- **Secrets entered in the Admin UI are write-only:** the UI and the API never return them, and an empty field keeps
  the stored value. Changing where a secret is sent (the SMTP server, a File Storage endpoint, the Vault URL) requires
  entering the secret again in the same save.
- **Sessions:** the keys that protect the sign-in cookies are stored in the database. Protect the database and its
  backups accordingly.

#### Reverse proxy and forwarded headers

The container speaks plain HTTP on port 8080; TLS ends at a reverse proxy, a load balancer or the Kubernetes Ingress.
The application takes the original client address, scheme and host from `X-Forwarded-For`, `X-Forwarded-Proto` and
`X-Forwarded-Host`, and accepts these headers from any source. Therefore:

- the proxy must **set and overwrite** these headers (never pass through values the client sent), for nginx:
  ```nginx
  proxy_set_header Host              $host;
  proxy_set_header X-Forwarded-Host  $host;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
  client_max_body_size 50m;
  proxy_read_timeout 600s;
  proxy_send_timeout 600s;
  ```
- publish the container port only to the proxy (`WSH_HTTP_PORT=127.0.0.1:8080` when the proxy runs on the same host),
  so nobody can reach the application around the proxy.

Attachment and image links and the OAuth `redirect_uri` are built from the request (the forwarded host and scheme);
links in e-mails come from `PublicBaseUrl`. The upload limit of 50 MB and the 600-second timeouts match the 25 MB upload
chunks (`TicketAttachment.UploadChunkSize`, `ImageConfiguration.ImageChunkSize`).

#### HTTPS in the container (optional)

If TLS must end in the container itself, add an HTTPS listen address and give Kestrel a certificate with its standard
variables. The run scripts publish port 8080 only, so use a manual `docker run` (see
[Running without the scripts](#running-without-the-scripts-manual-setup)):

```bash
docker run -d --name worksmarthub --restart unless-stopped --stop-timeout 60 \
  -p 8443:8443 \
  -e HttpSettings__Urls__0=http://+:8080 -e HttpSettings__Urls__1=https://+:8443 \
  -e ASPNETCORE_Kestrel__Certificates__Default__Path=/certs/wsh.pfx \
  --env-file /opt/worksmarthub/worksmarthub.env \
  --mount type=bind,source=/opt/worksmarthub/wsh.pfx,target=/certs/wsh.pfx,readonly \
  -v worksmarthub-data:/app/data worksmarthub:23
```

Put `ASPNETCORE_Kestrel__Certificates__Default__Password=<pfx password>` into `worksmarthub.env`, not onto the command
line. The certificate file must be readable by uid 1001.

#### Custom CA certificates (outbound TLS)

The application connects out over TLS: SMTP with STARTTLS, LDAPS, Microsoft Entra ID and OAuth providers, HashiCorp
Vault, Számlázz.hu, Slack, Teams, the object stores of the File Storages, and an external database with TLS. When one of
them presents a certificate of an internal or self-signed CA, the connection fails until the container trusts that CA.
The application uses the operating system trust store of the image (`/etc/ssl/certs/ca-certificates.crt`). Two ways to
add a CA:

**A derived image (works with the run scripts).** Put the CA file (PEM, `.crt`) next to a `Dockerfile`:

```dockerfile
FROM worksmarthub:23
USER root
COPY internal-ca.crt /usr/local/share/ca-certificates/internal-ca.crt
RUN update-ca-certificates
USER appuser
```

```bash
docker build -t worksmarthub:23-ca .
sh worksmarthub-run.sh --version 23-ca --detach
```

Build it again for every new version.

**A mounted bundle (manual `docker run`).** Append your CA to the bundle of the image and mount the result over it,
read-only:

```bash
docker run --rm --entrypoint cat worksmarthub:23 /etc/ssl/certs/ca-certificates.crt > ca-bundle.crt
cat internal-ca.crt >> ca-bundle.crt
# add to the docker run command:
#   --mount type=bind,source=/opt/worksmarthub/ca-bundle.crt,target=/etc/ssl/certs/ca-certificates.crt,readonly
```

Recreate the bundle from the new image at every upgrade, so that updated public CAs are kept. On Kubernetes the chart
does the same merge for you (see [Step 6](#step-6-custom-ca-certificates)). Get the CA from your PKI team; to see the
chain a server sends: `openssl s_client -connect smtp.example.com:587 -starttls smtp -showcerts </dev/null`. For a
database connection you can alternatively relax the check in the connection string (PostgreSQL `SSL Mode=Require`,
SQL Server `TrustServerCertificate=true`); trusting the CA is the better way.

#### Outbound connectivity

The application is also a client. What it connects to depends on the configuration:

| Destination | Port | When |
|-------------|------|------|
| DNS | 53 | Always |
| Internal database | 5432 PostgreSQL, 1433 SQL Server, 3306 MySQL | Unless SQLite |
| Redis, or the database cache | 6379, or the database port | `CacheSettings.Type` `Redis` / `SqlServer` / `Postgres` / `MySql` |
| SMTP server | 587 (STARTTLS), 25 | `EmailSettings` and e-mail notification channels. Implicit TLS on 465 is not supported for sending. |
| Slack (`slack.com`) | 443 | Slack notification channels |
| Microsoft Teams (Bot Framework, `smba.trafficmanager.net`, `login.microsoftonline.com`) | 443 | Teams notification channels |
| Számlázz.hu (`www.szamlazz.hu`) | 443 | Invoicing |
| File Storage backends: Amazon S3, Azure Blob, MinIO, Box, SharePoint, Google Cloud Storage | 443 (MinIO: its port) | File Storages created on these backends |
| HashiCorp Vault | its URL (often 8200) | A Vault service user catalogue |
| Microsoft Entra ID / Graph (`login.microsoftonline.com`, `graph.microsoft.com`) | 443 | `UserStore: AzureAD`, Microsoft OAuth |
| OAuth providers (Okta, Google, a custom provider) | 443 | `AuthType: OAuth` |
| LDAP / Active Directory domain controllers | 389, 636 | `UserStore: LDAP` |
| AppleConnect | its URLs | `AuthType: AppleConnect` |

`UserStore: ActiveDirectory` and the `ActiveDirectory` login need a domain-joined Windows server and do not work in a
Linux container; use `LDAP` (with the Active Directory flavour) or `AzureAD` there.

### Running the application

#### Linux

```bash
cd docker/linux
sh worksmarthub-run.sh --version 23 --detach     # or without --detach: in the foreground, Ctrl+C stops it
```

#### Windows

```powershell
cd docker\win
.\worksmarthub-run.ps1 -Version 23 -Detach
```

Open `http://<host>:8080/` and create the first user. Without `--version` / `-Version` the scripts run
`worksmarthub:latest`; `--image <reference>` / `-Image` runs any other image (e.g. from a registry).

What the run script does:

- checks the Docker engine and the image, and refuses to start (exit code 78) when a container of the same name is
  running, or when a stopped one exists that does not mount `/app/data` (it may hold data, see
  [Moving data out of a container](#moving-data-out-of-a-container)); a stopped container with the data volume is a
  leftover and is removed;
- runs the container named `worksmarthub` (one instance per host);
- mounts the named volume `worksmarthub-data` at `/app/data` (Docker creates it on the first start and copies the
  image's empty `FileStorage` and `Logs` folders into it);
- publishes port 8080 and sets `HttpSettings__Urls__0=http://+:8080`;
- mounts `appsettings.json` and passes `worksmarthub.env` when they are next to the script;
- gives the application 60 seconds to stop (`--stop-timeout 60`): it allows its background jobs 45 seconds and then
  closes the database;
- in the foreground the container is removed when it stops (`--rm`) and the script waits for it; with `--detach` /
  `-Detach` it runs with the restart policy `unless-stopped`: Docker restarts it after a crash and when the Docker daemon
  starts, unless you stopped it.

| Setting (environment variable) | Default | Meaning |
|--------------------------------|---------|---------|
| `WSH_IMAGE` | `worksmarthub:latest` | The image (`--version` and `--image` override it) |
| `WSH_CONTAINER_NAME` | `worksmarthub` | The container name |
| `WSH_DATA_VOLUME` | `worksmarthub-data` | The named volume; on Linux also an absolute host directory |
| `WSH_HTTP_PORT` | `8080` | The host port, or `address:port`, e.g. `127.0.0.1:8080` for the host's loopback only |
| `WSH_DATA_MOUNT_OPTS` | (empty) | Linux only: extra mount options, e.g. `:Z` on SELinux hosts |

`--preflight-only` / `-PreflightOnly` runs the checks without starting anything. The container's clock is UTC, so the
times of the background jobs (Admin UI, **App Konfiguráció > Háttérfolyamatok**) are UTC.

### Installing as a service

#### Linux (systemd service)

Keep the scripts in a folder only root can write to, for example `/opt/worksmarthub` (see
[Protecting the script folder](#protecting-the-script-folder)), with your `appsettings.json` and `worksmarthub.env`:

```bash
sudo sh worksmarthub-service-install.sh --version 23
```

The script checks the folder permissions, stops a running service, runs the preflight checks, writes
`/etc/systemd/system/worksmarthub.service`, and enables and starts it. The `WSH_*` settings that are set when the
install script runs (`sudo WSH_HTTP_PORT=8081 sh worksmarthub-service-install.sh ...`) are written into the unit. The
service:

- runs `worksmarthub-run.sh` in the foreground from the script folder (keep the folder in place), after the Docker
  service (`docker.service`, or `snap.docker.dockerd.service` for the snap package);
- restarts it 10 seconds after it exits, except after exit code 78 (a failed preflight check, which a restart does not
  fix: `RestartPreventExitStatus=78`);
- stops it with `docker stop -t 60` (`TimeoutStopSec=90`).

```bash
systemctl status worksmarthub      # status
systemctl restart worksmarthub     # restart (also applies appsettings.json and worksmarthub.env changes)
systemctl stop worksmarthub        # stop
journalctl -u worksmarthub -f      # the service log (the container's console output)
```

`WSH_SERVICE_NAME` changes the service name. Re-run the install script with a new `--version` to upgrade.

#### Windows (Scheduled Task)

As Administrator, in the script folder:

```powershell
.\worksmarthub-service-install.ps1 -Version 23
```

The script checks the folder permissions, stops an existing task and its container, runs the preflight checks, asks
for the account that runs the task (the password goes to Task Scheduler and is stored nowhere else), registers the task
`WorkSmartHub` and starts it. The task:

- runs `worksmarthub-run.ps1 -Image <image>` non-interactively with the highest privileges, at startup (with a random
  delay of up to a minute) and at every sign-in, because Docker Desktop only runs after a user signed in;
- waits for the Docker engine as long as it takes, and writes its start-up messages to `worksmarthub-task.log` next to
  the script;
- is restarted up to 10 times at one-minute intervals by Task Scheduler if it fails.

Docker Desktop must start automatically: **Settings > General > Start Docker Desktop when you sign in**.

```powershell
Get-ScheduledTask -TaskName WorkSmartHub | Get-ScheduledTaskInfo     # last run and result
Stop-ScheduledTask -TaskName WorkSmartHub; docker stop worksmarthub  # stop
Start-ScheduledTask -TaskName WorkSmartHub                           # start
docker logs -f worksmarthub                                          # the application log
```

**Docker restart policy only.** `--detach` / `-Detach` is enough when the Docker daemon starts at boot. Do not combine
it with the service or the task.

### Uninstalling the service

```bash
cd docker/linux
sudo sh worksmarthub-service-uninstall.sh
sudo sh worksmarthub-service-uninstall.sh --remove-image worksmarthub:23    # also remove that image
```

```powershell
cd docker\win
.\worksmarthub-service-uninstall.ps1                           # as Administrator
.\worksmarthub-service-uninstall.ps1 -RemoveImage worksmarthub:23
```

The service or task and the container are removed. **The data volume is kept** and is used again by the next
installation. Images are kept unless you name one, because a locally built image cannot be pulled again. Delete the
data with `docker volume rm worksmarthub-data` only when you want to remove all of it.

### Data persistence

#### What is persisted

All runtime state is in the volume `worksmarthub-data`, mounted at `/app/data`:

| Content | In the volume | Notes |
|---------|---------------|-------|
| SQLite database (the default `InternalDbType: 0`, `Data Source=database.db;`) | `database.db`, `database.db-wal`, `database.db-shm` | `/app/database.db` links here. The `-wal` and `-shm` files belong to the database; copy them together. |
| Physical File Storage (default root path `FileStorage`) | `FileStorage/` | `/app/FileStorage` links here. Ticket attachments and component images of the File Storages created on the Physical backend (Admin UI, **App Fájltárolók**), and the invoice PDFs of tickets without a File Storage (the automatic **Számlák (automatikus)** storage). |
| Log files | `Logs/` | `/app/Logs` links here. Daily files, shown in the Admin UI (**Naplófájlok**). `docker logs` shows the console output. |
| Live e-mail and notification templates | `LiveEmailTemplates/`, `LiveNotificationTemplates/` | Created at start; missing defaults are added, edited files are never overwritten. |

Relative paths resolve inside the volume: `Data Source=database.db`, the File Storage root path `FileStorage` and
`Logs/log-.txt`. Keep the Physical File Storage root path relative (`FileStorage`, or another path below `data/`): an
absolute path elsewhere in the container is not persisted. With an external database and an external File Storage
backend the volume holds only the logs and the templates.

Not in the volume: `appsettings.json` and `worksmarthub.env` (next to the scripts) and the licence and the sign-in
cookie keys (in the database). The container itself is disposable: it can be removed and recreated at any time, and a
restart, a reboot, a crash or an upgrade never loses data.

**Editing a template.** Copy it out, edit it, copy it back. The next e-mail uses it, no restart needed:

```bash
docker cp worksmarthub:/app/data/LiveEmailTemplates/ResetPassword.cshtml .
docker cp ResetPassword.cshtml worksmarthub:/app/data/LiveEmailTemplates/
```

`docker cp` leaves the file owned by root; the application only reads it. Delete a file from the folder and restart to
get the default back.

#### Protecting the script folder

The service makes the operating system run the run script as root (Linux) or with the highest privileges (Windows) at
every start, and the folder holds `appsettings.json` and `worksmarthub.env` with their secrets. Whoever can change these
files controls the application and the host, so the install scripts **refuse to install when other users can write to
the folder**:

- **Linux:** the folder, the run script, `appsettings.json` and `worksmarthub.env` must not be group- or world-writable,
  and no parent directory may be world-writable without the sticky bit. Use a root-owned folder such as
  `/opt/worksmarthub` (`sudo chown -R root:root /opt/worksmarthub && sudo chmod -R go-w /opt/worksmarthub`); give
  `appsettings.json` to uid 1001 afterwards if the application should merge new keys into it. Never install from `/tmp`.
- **Windows:** only Administrators, SYSTEM and the installing user may have write rights on the folder and the files.
  A folder created directly under `C:\` inherits *Modify* for *Authenticated Users* and is refused; remove the inherited
  rights first: `icacls "C:\WorkSmartHub" /inheritance:r /grant:r "*S-1-5-32-544:(OI)(CI)F" "*S-1-5-18:(OI)(CI)F"`.

uid 1001 may be a regular local account on a Linux host (`getent passwd 1001`); that account can then read the files
the application owns, exactly like the application.

#### Upgrading

1. Build or load the new image (`worksmarthub:24`) and take a backup (below).
2. Start the new image on the same volume:
   - Linux service: `sudo sh worksmarthub-service-install.sh --version 24`;
   - Windows task: `.\worksmarthub-service-install.ps1 -Version 24`;
   - manual run: `docker stop worksmarthub`, then `sh worksmarthub-run.sh --version 24 --detach`.
3. The new version migrates the database at start (`docker logs -f worksmarthub` shows `Migrating database` and
   `Migration complete`) and merges new settings keys into a writable `appsettings.json`. Users stay signed in.

Going back to an earlier version means restoring the backup taken before the upgrade: the database is never migrated
back. Read the changelog of the new version for steps it needs. With `worksmarthub:latest` (no `--version`) a new build
moves the tag, and a restart of the service starts the new image.

#### Backup and restore

Stop the application first: copying a SQLite database in write-ahead-log mode while it runs can give an inconsistent
copy. The helper reuses the application image and runs without network. The archive contains credentials, so it is
created readable by you only.

```bash
# Linux: stop first (the one that applies)
sudo systemctl stop worksmarthub      # service
docker stop worksmarthub              # manual run
docker run --rm --network none -v worksmarthub-data:/app/data -v "$PWD":/backup --user "$(id -u):$(id -g)" \
  --entrypoint sh worksmarthub:23 -c 'umask 077; tar czf "/backup/worksmarthub-data-$(date +%F).tgz" -C /app/data .'
sudo systemctl start worksmarthub     # or run the run script again
```

```powershell
# Windows: stop first (the one that applies), in a folder only administrators can read
Stop-ScheduledTask -TaskName WorkSmartHub; docker stop worksmarthub     # task
docker stop worksmarthub                                                # manual run
docker run --rm --network none -v worksmarthub-data:/app/data -v "${PWD}:/backup" `
  --entrypoint tar worksmarthub:23 czf "/backup/worksmarthub-data-$(Get-Date -Format yyyy-MM-dd).tgz" -C /app/data .
Start-ScheduledTask -TaskName WorkSmartHub                              # or run the run script again
```

**Restore** (application stopped; replaces the whole volume; the archive is read completely before anything is
deleted). On Windows use `-v "${PWD}:/backup:ro"` and a backtick for the line breaks:

```bash
docker run --rm --network none -u 0 --cap-drop ALL --cap-add CHOWN --cap-add DAC_OVERRIDE --cap-add FOWNER \
  -v worksmarthub-data:/app/data -v "$PWD":/backup:ro --entrypoint sh worksmarthub:23 \
  -c 'tar tzf /backup/worksmarthub-data-YYYY-MM-DD.tgz >/dev/null && find /app/data -mindepth 1 -delete && tar xzf /backup/worksmarthub-data-YYYY-MM-DD.tgz -C /app/data && chown -R 1001:1001 /app/data'
```

The helper runs as root inside the container only to read your archive and to give the files to the application user,
with no network and only the three capabilities it needs. Only restore archives you created yourself. Back up
`appsettings.json` and `worksmarthub.env` separately; with an external database, back it up with its own tools.

> **Backups contain credentials.** The database stores the File Storage keys, the e-mail, notification and Számlázz.hu
> credentials and the sign-in cookie keys in a form the application can use without anything else; `appsettings.json`
> holds the signing keys. Store backups with restricted permissions and encryption at rest, and delete temporary copies.

#### Keeping the volume safe

- Named volumes survive `docker stop`, `docker rm`, `docker rmi`, image upgrades and the uninstall scripts.
- While the container is stopped and removed, the volume counts as unused: on Docker Engine before 23
  `docker volume prune` and `docker system prune --volumes` delete unused named volumes, on 23 and later
  `docker volume prune -a` still does. Docker Desktop's **Clean / Purge data** and **Reset to factory defaults** delete
  all volumes. Take a backup before any of these.
- To delete all application data on purpose: `docker volume rm worksmarthub-data`, with the application stopped.

#### Alternative: a host directory instead of the named volume (Linux)

To keep the data in a directory you manage (a dedicated disk, an existing backup job), create it on a **local
filesystem** (not NFS or SMB: the SQLite write-ahead log needs local locks and shared memory) with both subfolders, and
give it to uid 1001:

```bash
sudo mkdir -p /srv/worksmarthub/FileStorage /srv/worksmarthub/Logs
sudo chown -R 1001:1001 /srv/worksmarthub
sudo chmod 700 /srv/worksmarthub
sudo WSH_DATA_VOLUME=/srv/worksmarthub sh worksmarthub-service-install.sh --version 23
```

An absolute `WSH_DATA_VOLUME` is mounted as a bind mount. The run script checks that the directory and both subfolders
exist and that uid 1001 owns it, and refuses to start otherwise (without the subfolders the File Storage and the log
files would fail). With mode 700 the start must run as root (the service does). On SELinux hosts also set
`WSH_DATA_MOUNT_OPTS=:Z`. Not recommended on Windows: bind mounts from NTFS are slow and SQLite locking on them is
unreliable.

#### Multi-node and high availability

The data volume is local to one Docker host. Several application nodes (load-balanced hosts, Kubernetes replicas) can
share one database when all of these hold:

| Part | For several nodes | Not for several nodes |
|------|-------------------|-----------------------|
| Database | SQL Server, PostgreSQL, MySQL | SQLite |
| Cache | `Redis`, or the database cache (`SqlServer`, `Postgres`, `MySql`) | `InMemory` (permissions, the licence status, master data lists and the OAuth sign-in state would differ between nodes; the application logs this advice at start) |
| File Storage | S3, Azure Blob, MinIO, Box, SharePoint, Google Cloud Storage, or one shared network volume | Physical on a local disk |
| Settings | the same `appsettings.json` values on every node, including the signing keys and `PublicBaseUrl` | |

With that:

- **Database migrations** run at start under a cross-node lock (PostgreSQL advisory lock, SQL Server application lock,
  MySQL named lock): one node migrates, the others wait and find nothing to do, also on a fresh database.
- **Sessions** are valid on every node (the keys are in the database).
- **Scheduled background jobs run once per occurrence across all nodes:** each node claims the occurrence in the
  database before it runs, so e-mails, Teams and Slack messages and Számlázz.hu calls are not duplicated. The
  **Run-once claim cleanup** job deletes old claims (on by default, daily at 04:15, keeps 7 days). On several nodes use
  job intervals of 1 day or less, and keep the clocks synchronised.
- **Configuration changes** made in the Admin UI reach every node within the `ConfigSync` interval (10 seconds).
- Object storage credentials need permission to **list and delete** objects as well as read and write, and every
  installation needs its own bucket or prefix: a copy of the database (a restored backup, a test environment) keeps the
  File Storage definitions with their credentials, so point them elsewhere before such a copy goes live.
- **Live templates** are per node unless the data folder is shared; copy an edited template to every node.

#### Moving data out of a container

A container started by hand without the volume keeps its database and files inside the container, and removing that
container deletes them. The run script refuses to start while a stopped container named `worksmarthub` without a
`/app/data` volume exists, and it never removes such a container. Copy the data into the volume first:

```bash
docker stop worksmarthub                        # if it runs; a container started with --rm is deleted by this, see below
mkdir -m 700 wsh-migrate
docker cp worksmarthub:/app/data/. wsh-migrate/
docker run --rm --network none -u 0 --cap-drop ALL --cap-add CHOWN --cap-add DAC_OVERRIDE --cap-add FOWNER \
  -v worksmarthub-data:/app/data -v "$PWD/wsh-migrate":/migrate:ro --entrypoint sh worksmarthub:23 \
  -c 'cp -a /migrate/. /app/data/ && mkdir -p /app/data/FileStorage /app/data/Logs && chown -R 1001:1001 /app/data'
```

- The image keeps everything under `/app/data`, so the `docker cp` takes it all. Older images without that folder kept
  the files at `/app/database.db` (with `-wal`), `/app/FileStorage`, `/app/Logs` and
  `/app/wwwroot/LiveEmailTemplates`, `/app/wwwroot/LiveNotificationTemplates`: copy those into `wsh-migrate` instead
  (`docker cp -L` for the database files).
- **A running container that was started with `--rm`** is deleted when it stops. Copy from it while it runs, after a
  consistent snapshot: `docker commit worksmarthub wsh-rescue` (pauses it briefly), `docker create --name wsh-rescue-c
  wsh-rescue`, and run the `docker cp` against `wsh-rescue-c`.
- Start the new installation, check the data, then delete `wsh-migrate` (it holds credentials), the rescue container
  and image, and finally the old container (`docker rm worksmarthub`).

To start with an empty database instead, remove the old container yourself.

### Running without the scripts (manual setup)

If your policies do not allow our scripts, plain Docker commands do the same. The essentials are a fixed container
name, the data volume, the configuration from the host and a restart mechanism.

#### Step 1: Prepare the host

1. Install Docker (Docker Engine on Linux, Docker Desktop on Windows) and build or load the image.
2. Create a folder for the configuration that only administrators can write to, for example `/opt/worksmarthub`
   (root-owned, mode 755). Put your `appsettings.json` (optional; owned by uid 1001 with mode 600 if the application
   should merge new keys into it) and `worksmarthub.env` (mode 600) into it.
3. Create the data volume once: `docker volume create worksmarthub-data`.

#### Step 2: Run the container

Linux:

```bash
docker run -d --name worksmarthub --restart unless-stopped --stop-timeout 60 \
  -p 8080:8080 -e HttpSettings__Urls__0=http://+:8080 \
  --mount type=bind,source=/opt/worksmarthub/appsettings.json,target=/app/appsettings.json \
  --env-file /opt/worksmarthub/worksmarthub.env \
  -v worksmarthub-data:/app/data \
  worksmarthub:23
```

Windows (PowerShell):

```powershell
docker run -d --name worksmarthub --restart unless-stopped --stop-timeout 60 `
  -p 8080:8080 -e HttpSettings__Urls__0=http://+:8080 `
  --mount "type=bind,source=C:\Users\<admin>\WorkSmartHub\appsettings.json,target=/app/appsettings.json" `
  --env-file "C:\Users\<admin>\WorkSmartHub\worksmarthub.env" `
  -v worksmarthub-data:/app/data `
  worksmarthub:23
```

| Option | Purpose |
|--------|---------|
| `-d` | Runs in the background. |
| `--name worksmarthub` | Fixed name: a second copy cannot start by accident, and the commands below address it. |
| `--restart unless-stopped` | Docker restarts it after a crash and when the daemon starts (host reboot), unless you stopped it. |
| `--stop-timeout 60` | `docker stop` waits up to 60 seconds: the application gives its background jobs 45 seconds and then closes the database. |
| `-p 8080:8080` | Publishes the web UI on host port 8080 (the first number; `127.0.0.1:8080:8080` for the loopback only). |
| `-e HttpSettings__Urls__0=http://+:8080` | The listen address inside the container; the port must be the container side of `-p`. |
| `--mount type=bind,...appsettings.json` | Optional: your configuration file from the host. Without it the shipped file is used. |
| `--env-file ...` | Optional: settings and secrets as environment variables; they override the file. |
| `-v worksmarthub-data:/app/data` | The persistent data volume. |

#### Step 3: Start at boot

**Linux, option A (simplest): the Docker restart policy.** With `--restart unless-stopped` and the Docker service
enabled at boot (`sudo systemctl enable docker`), nothing else is needed.

**Linux, option B: a systemd unit.** Do not combine it with `--restart unless-stopped`; systemd owns the lifecycle.
Create `/etc/systemd/system/worksmarthub.service`:

```ini
[Unit]
Description=WorkSmartHub
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
TimeoutStopSec=90
Restart=always
RestartSec=10
ExecStartPre=-/usr/bin/docker rm worksmarthub
ExecStart=/usr/bin/docker run --rm --name worksmarthub --stop-timeout 60 -p 8080:8080 -e HttpSettings__Urls__0=http://+:8080 --mount type=bind,source=/opt/worksmarthub/appsettings.json,target=/app/appsettings.json --env-file /opt/worksmarthub/worksmarthub.env -v worksmarthub-data:/app/data worksmarthub:23
ExecStop=/usr/bin/docker stop -t 60 worksmarthub

[Install]
WantedBy=multi-user.target
```

`ExecStart` runs the container in the foreground (no `-d`) so systemd tracks it; `--rm` removes the finished container
so the next start never hits a name conflict, and `ExecStartPre` clears a leftover after an unclean shutdown (it fails
harmlessly and never stops a running container). Use the real path of the `docker` binary (`command -v docker`); for
the snap package use `snap.docker.dockerd.service` in `After=` and `Requires=`. Then:

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now worksmarthub
journalctl -u worksmarthub -f
```

**Windows, option A (recommended): the Docker restart policy.** Enable **Start Docker Desktop when you sign in**; when
the daemon starts it restarts every container with `--restart unless-stopped`. Docker Desktop does not run before a
user signs in.

**Windows, option B: a Scheduled Task** that waits for the engine and starts the existing container (Task Scheduler
does not retry a program that fails, and Docker Desktop needs a minute or two after sign-in):

```powershell
$command = 'while ($true) { docker version *> $null; if ($LASTEXITCODE -eq 0) { break }; Start-Sleep -Seconds 15 }; docker start worksmarthub'
$action   = New-ScheduledTaskAction -Execute "powershell.exe" -Argument "-NoProfile -NonInteractive -Command `"$command`""
$triggers = @((New-ScheduledTaskTrigger -AtStartup), (New-ScheduledTaskTrigger -AtLogOn))
$settings = New-ScheduledTaskSettingsSet -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Hours 1)
Register-ScheduledTask -TaskName "WorkSmartHub" -Action $action -Trigger $triggers -Settings $settings
```

The task runs as the registering user, who must be in the `docker-users` group. `docker start` does nothing when the
container already runs.

#### Step 4: Day-to-day commands

| Task | Command |
|------|---------|
| Logs | `docker logs -f worksmarthub` |
| Stop / start | `docker stop worksmarthub` (waits up to 60 s) / `docker start worksmarthub` |
| Apply an `appsettings.json` change | `docker restart worksmarthub` |
| Apply a `worksmarthub.env` change | remove and run again (`docker stop worksmarthub && docker rm worksmarthub`, then Step 2): environment variables are fixed when a container is created |
| Upgrade | `docker stop worksmarthub && docker rm worksmarthub`, then Step 2 with the new image. The volume is reused. |
| Backup / restore | See [Backup and restore](#backup-and-restore). |
| Uninstall | `docker stop worksmarthub && docker rm worksmarthub`; `docker rmi worksmarthub:23` if the image should go. The volume stays. |

---

## Kubernetes

The `kubernetes-helm-chart` folder contains the Helm chart `worksmarthub` (chart version 1.0.0). Every setting is
documented in [`kubernetes-helm-chart/values.yaml`](https://github.com/WorkWise-Solutions/WorkSmartHub-Docker-Kubernetes/blob/main/kubernetes-helm-chart/values.yaml).

### Kubernetes prerequisites

- A running cluster (EKS, AKS, GKE, self-managed; minikube for a local test) and `kubectl` configured for it.
- Helm 3.
- The application image where the nodes can get it: built into the node's Docker daemon (minikube), loaded onto the
  nodes, or pushed to a registry the cluster can pull from.
- A default StorageClass (or one you name) for the PersistentVolumeClaims.
- The **ingress-nginx** controller for the chart's Ingress; with another controller or a service mesh, disable the
  Ingress ([Step 8](#step-8-service-type-and-ingress)).
- Outbound access to Docker Hub for `postgres:16-alpine`, `redis:7-alpine` and `busybox:1.38.0`, or copies in your
  registry ([Air-gapped and restricted networks](#air-gapped-and-restricted-networks)).

### Quick start

The chart's defaults start a complete installation without editing a file:

```bash
bash build/build-image.sh --version 23 --minikube      # or push the image to a registry the cluster can pull from
helm install worksmarthub ./kubernetes-helm-chart --set app.image.tag=23
kubectl get pods -n worksmarthub -w
kubectl port-forward -n worksmarthub svc/worksmarthub 8080:80      # then http://localhost:8080/
```

It deploys into the namespace `worksmarthub`:

| Component | What it is |
|-----------|------------|
| Application | One pod of `worksmarthub:<tag>` with probes on `GET /health`, `appsettings.json` from a ConfigMap, and the data volume at `/app/data` |
| Data volume | A PersistentVolumeClaim `worksmarthub-data-claim` (5 Gi, ReadWriteOnce, default StorageClass): Physical File Storage, logs, live templates, and the SQLite file when SQLite is selected |
| PostgreSQL | `postgres:16-alpine`, one replica, PVC `postgres-data-claim` (5 Gi). The chart's `appsettings.json` selects it (`InternalDbType: 2`, `Server=db`) |
| Redis | `redis:7-alpine`, one replica, cache only. The chart's `appsettings.json` selects it (`CacheSettings.Type: Redis`, `cache:6379`) |
| Ingress | For ingress-nginx, with upload limits ([Step 8](#step-8-service-type-and-ingress)) |
| NetworkPolicies | Only pods of the namespace may connect to the built-in PostgreSQL and Redis |

> **Change the database password.** The built-in PostgreSQL starts with the public default user `user` and password
> `password`. Before the first install of anything but a throw-away evaluation, set your own in
> `internalDatabase.settings` and the same in the connection string of `kubernetes-helm-chart/appsettings.json` (or in a
> Secret), see [Database password](#database-password). Changing it later needs a step inside the database.

This setup suits a first install and evaluation. Before going live also set the signing keys and `PublicBaseUrl`
([Step 7](#step-7-secure-credentials)) and work through the [Production readiness checklist](#production-readiness-checklist).

Keep your changes in a values file of your own (`-f my-values.yaml`) and keep that file for upgrades; the steps below
show the keys.

### Network and connectivity requirements

**Inbound** is handled by the Ingress (or the Service): once it exposes the application on 80/443, users can reach it.
**Outbound** connections are made by the application; each needs two things: *where to connect* (application
configuration: `appsettings.json`, Secrets, or the Admin UI) and *whether the pod can reach it* (platform setup, needed
only when egress is restricted or the target is on a network the cluster cannot route to, such as an on-premises SMTP
relay or LDAP server).

**Inbound:**

| Port | Source | Purpose |
|------|--------|---------|
| 80 / 443 | Users' browsers, API clients, the load balancer | The portal and the REST API, served from the same container and port |

TLS ends at the Ingress or load balancer; the container listens on 8080 (`HttpSettings.Urls`), the Service on 80.

**Outbound:** the table under [Outbound connectivity](#outbound-connectivity) applies, plus the container registry (443)
at image pull time. In-cluster services (the built-in database and cache) are reached over Service DNS (`db:5432`,
`cache:6379`) and never leave the cluster. A pod's traffic to outside hosts is source-NAT'd to its **node's** address,
so external firewalls see the node IP: allow the nodes (or a dedicated node pool, [Step 9](#step-9-replicas-and-scaling))
to reach each destination.

**Restricting egress (optional).** With a NetworkPolicy-capable CNI (Calico, Cilium) you can add a default-deny egress
policy for the namespace and an allow list for the application pods. The allow list must include **everything the
application uses**, or it stops working:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: worksmarthub
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress: []
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: worksmarthub-app-egress
  namespace: worksmarthub
spec:
  podSelector:
    matchLabels:
      app: worksmarthub-pod
  policyTypes: [Egress]
  egress:
  - ports:                                   # DNS
    - { protocol: UDP, port: 53 }
    - { protocol: TCP, port: 53 }
  - to:                                      # built-in PostgreSQL
    - podSelector: { matchLabels: { app: db, wsh-internal-service: db } }
    ports:
    - { protocol: TCP, port: 5432 }
  - to:                                      # built-in Redis
    - podSelector: { matchLabels: { app: cache, wsh-internal-service: cache } }
    ports:
    - { protocol: TCP, port: 6379 }
  - ports:                                   # Számlázz.hu, Slack, Teams, Entra ID, OAuth, object stores
    - { protocol: TCP, port: 443 }
  - ports:                                   # SMTP (STARTTLS)
    - { protocol: TCP, port: 587 }
  # Add what your configuration needs: an external database (1433/3306/5432) or cache (6379) as an ipBlock,
  # LDAP (389/636), HashiCorp Vault (8200), SMTP on 25, a MinIO port.
```

The default-deny policy also applies to the built-in PostgreSQL and Redis pods, which need no egress. A NetworkPolicy
is namespace-scoped; to keep every other workload of the cluster away from a target, scope the firewall rule to a
dedicated node pool.

### Step 1: Namespace

```yaml
nameSpace: "worksmarthub"
installNameSpace: True
```

With `installNameSpace: True` the chart creates the namespace, and `helm uninstall` deletes it with everything in it.
To keep the namespace (and the Secrets you create in it) independent of the release, create it yourself and set
`installNameSpace: False`:

```bash
kubectl create namespace worksmarthub
helm install worksmarthub ./kubernetes-helm-chart --set app.image.tag=23 --set installNameSpace=False
```

The commands in this guide use `worksmarthub`; substitute your namespace if you change it.

### Step 2: Application image

```yaml
app:
  image:
    repository: "worksmarthub"
    tag: "23"               # required
  imagePullSecrets: []
global:
  imagePullPolicy: "IfNotPresent"
```

- `app.image.tag` is **required**; the chart refuses to render without it. A tag of the form `sha256:...` is used as a
  digest (`repository@sha256:...`).
- `imagePullPolicy: IfNotPresent` (for every container of the chart) uses an image the node already has. A locally
  built image needs it: with `Always` the kubelet asks a registry and the pod fails with `ErrImagePull`. The other side
  of it: **a node never fetches a tag again that it already has**, so every build needs a new tag (the build script
  tags the version), and an upgrade sets the new tag. Do not deploy `latest`.
- **Getting the image to the nodes:** on minikube build into its daemon (`build-image.sh --minikube`); on other
  clusters without a registry, load the tar file on every node with the node's container runtime; with a registry, push
  the image and set `app.image.repository` (e.g. `registry.example.com/worksmarthub`).
- **A private registry:** create a pull secret and reference it:
  ```bash
  read -rs REGISTRY_PASSWORD      # type it; nothing is shown or stored
  kubectl create secret docker-registry regcred -n worksmarthub --docker-server=registry.example.com \
    --docker-username=<user> --docker-password="$REGISTRY_PASSWORD"
  ```
  ```yaml
  app:
    imagePullSecrets:
    - name: regcred
  ```

### Step 3: Internal database

#### The built-in PostgreSQL (default)

`postgres:16-alpine`, one replica, on the PVC `postgres-data-claim` (`internalDatabase.persistence`: 5 Gi, ReadWriteOnce,
default StorageClass), as the non-root user 70 with `fsGroup: 70`. Its Service is `db:5432`, and the chart's
`appsettings.json` connects to it with the values of `internalDatabase.settings`. Durable across reschedules, but a
single copy without failover, and briefly unavailable on every restart of its pod (`Recreate` strategy).

- `internalDatabase.persistence.storageClass`: a durable block class (`gp3`, `managed-csi`, `premium-rwo`); `existingClaim`
  for a PVC you manage; `enabled: false` for an `emptyDir` (data lost with the pod; throw-away tests only).
- `internalDatabase.fixDataPermissions: True` adds a one-shot root init container that `chown`s the data directory.
  Only for storage that ignores `fsGroup` and creates the volume root-owned with restrictive permissions; leave it off
  on policy-restricted clusters.
- `internalDatabase.image`: a copy in your registry or a newer tag. The containers run as uid 70, the Alpine variant's
  postgres user; a Debian-based image (uid 999) needs `runAsUser` changed in `templates/internaldb.yaml` and in the
  `wait-for-db` init container. A **major version change** of an existing database is not an in-place upgrade:
  PostgreSQL refuses a data directory of another major version, so dump and restore.

#### Database password

PostgreSQL creates its user, password and database from `internalDatabase.settings` (`user` / `password` /
`worksmarthub`) on its **first** start (an empty data directory), and the chart's `appsettings.json` connects with the
same values:

```json
"InternalDbType": 2,
"InternalDb": "Server=db;Port=5432;Database=worksmarthub;User id=user;Password=password;",
```

These defaults are public. Before the first install set your own in both places:

```yaml
internalDatabase:
  settings:
    userName: wsh
    password: <a strong password>
    dbName: worksmarthub
```

and `"InternalDb": "Server=db;Port=5432;Database=worksmarthub;User id=wsh;Password=<the same password>;"` in
`kubernetes-helm-chart/appsettings.json`, or rather a Secret with the key `ConnectionStrings__InternalDb`
([Step 7](#step-7-secure-credentials)); `values.yaml` and the ConfigMap hold the values in plain text.

Later changes of `internalDatabase.settings` do not change an existing database. To change the password of an
installation, change it in the database first, then in your values and in the connection string, and `helm upgrade`:

```bash
kubectl exec -n worksmarthub deploy/db-deployment -- psql -U user -d worksmarthub -c "ALTER USER \"user\" PASSWORD '<new password>';"
```

The application pod waits for PostgreSQL in the `wait-for-db` init container. It logs in with the `values.yaml`
credentials and writes into its log when PostgreSQL refuses them (`kubectl logs -n worksmarthub <app pod> -c wait-for-db`);
the application starts anyway, with the connection string of `appsettings.json` or a Secret.

#### An external database (recommended for production)

Set `internalDatabase.skipDeployment: True` and give the connection in a Secret or in the chart's `appsettings.json`.
Managed services (Amazon RDS, Azure Database, Google Cloud SQL) or your own server:

| Database | `InternalDbType` | `InternalDb` example |
|----------|------------------|----------------------|
| SQL Server | 1 | `Data Source=sql.example.com;Initial Catalog=wsh;User id=wsh;Password=...;Connection Timeout=60;Encrypt=True;TrustServerCertificate=False;` |
| PostgreSQL | 2 | `Server=pg.example.com;Port=5432;Database=wsh;User id=wsh;Password=...;SSL Mode=Require;` |
| MySQL | 3 | `Server=mysql.example.com;Port=3306;Database=wsh;Uid=wsh;Pwd=...;SslMode=Required;` (and `MySqlServerVersion`, e.g. `8.0.28`) |

- `Schema` places the tables in a schema on SQL Server and PostgreSQL; it must stay empty for MySQL (and SQLite).
- The application's database user needs the right to create and alter tables: the migrations run at every start.
- Moving between database engines is not built in; an existing installation keeps its engine.

#### SQLite, single replica

For a small installation without PostgreSQL and Redis, SQLite lives on the data volume (the same as
[`test-local/values-sqlite.yaml`](https://github.com/WorkWise-Solutions/WorkSmartHub-Docker-Kubernetes/blob/main/test-local/values-sqlite.yaml)):

```yaml
app:
  replicationFactor: 1
  env:
  - name: ConnectionStrings__InternalDbType
    value: "0"
  - name: ConnectionStrings__InternalDb
    value: "Data Source=database.db;"
  - name: CacheSettings__Type
    value: "InMemory"
  dataVolume:
    persistence:
      enabled: true
internalDatabase:
  skipDeployment: True
cacheSystem:
  skipDeployment: True
```

The chart refuses SQLite with more than one replica, and SQLite without the data volume (unless
`app.allowNonPersistentSqlite: true`, for throw-away tests). Use a block StorageClass, not NFS.

### Step 4: Cache

The chart deploys `redis:7-alpine` (cache only: no persistence, no password; its NetworkPolicy admits only pods of the
namespace), and the chart's `appsettings.json` selects it:

```json
"CacheSettings": {
  "Type": "Redis",
  "RedisCacheSettings": { "Configuration": "cache:6379", "InstanceName": "worksmarthub" }
}
```

- **More than one replica needs a shared cache:** `Redis` or a database cache. The chart refuses `InMemory` with more
  than one replica. The in-memory cache is local to each pod, so permissions, the licence status, master data lists and
  the OAuth sign-in state would differ between pods; the application logs a warning when it runs with `InMemory` on
  Kubernetes.
- **External Redis** (Amazon ElastiCache, Azure Cache for Redis): `cacheSystem.skipDeployment: True` and
  `CacheSettings__RedisCacheSettings__Configuration=redis.example.com:6380,password=...,ssl=true` in a Secret.
- **Database cache:** `CacheSettings__Type` `SqlServer`, `Postgres` or `MySql` with the matching
  `...CacheSettings__ConnectionString`, schema and table name (see `appsettings.json`).
- **One replica without Redis:** `cacheSystem.skipDeployment: True` and `CacheSettings__Type=InMemory`.

The application pod waits up to 5 minutes for the built-in Redis in the `wait-for-cache` init container, so a cache pod
that starts later does not restart the application; after 5 minutes it starts anyway.

#### Container images and security policy

The chart uses two kinds of images:

- **The application image**, built by you from the sources on a supported Microsoft base image (ASP.NET Core 10.0 on
  Ubuntu 24.04 LTS) with the current OS security updates (`--no-cache` for a rebuild of an unchanged release).
- **Infrastructure images:** `postgres:16-alpine` (`internalDatabase.image`), `redis:7-alpine` (`cacheSystem.image`),
  `busybox:1.38.0` (`busyboxImage`, the `wait-for-cache` and `fix-db-permissions` init containers). They are upstream
  images, not part of the product; keeping them current is an operator task. Point them at your registry or at a newer
  tag that passes your scanner; changing the reference forces a fresh pull even with `IfNotPresent`.

For a scanner finding, check which image it belongs to: an infrastructure image is fixed by a newer upstream tag
without a product release; the application image by a rebuild (OS packages) or a new version (application packages).

### Step 5: Application settings

**`kubernetes-helm-chart/appsettings.json`** is the shipped file with four changes for the chart: the built-in
PostgreSQL (`ConnectionStrings`), the built-in Redis (`CacheSettings`), and the listen address `http://+:8080`. The chart
puts it into the ConfigMap `worksmarthub-config` and mounts it read-only at `/app/appsettings.json`; a change rolls the
application pod at the next `helm upgrade` (a checksum annotation). Because the file is read-only, settings keys of a
newer release are used with their defaults in memory, and the log says `appsettings.json is read-only`.

**`app.env`** sets plain environment variables, which override the file. Values must be strings. Never put secrets
here (they appear in the Deployment in plain text); use [Secrets](#step-7-secure-credentials).

```yaml
app:
  env:
  - name: PublicBaseUrl
    value: "https://wsh.example.com/"
  - name: Serilog__MinimumLevel__Override__Microsoft
    value: "Warning"
  - name: Swagger__Enabled
    value: "false"
  - name: UseDetailedExceptions
    value: "false"
```

What else to review: `UserSettings` and `LoginSettings` (on Kubernetes use `Local`, `LDAP` or `AzureAD`;
`ActiveDirectory` needs a Windows host), `EmailSettings`, the log levels (see [Log levels](#log-levels)), and the
security settings (see [Security settings](#security-settings)). The signing keys belong in a Secret.

#### Data volume and File Storage

`app.dataVolume.persistence` (enabled by default) mounts the PVC `worksmarthub-data-claim` at `/app/data`, the same
layout as the Docker volume ([What is persisted](#what-is-persisted)):

```yaml
app:
  dataVolume:
    persistence:
      enabled: true
      size: 5Gi
      storageClass: ""          # "" = default; a durable class with encryption at rest
      accessMode: ReadWriteOnce
      existingClaim: ""
```

- An init container (`prepare-data`, the application image as uid 1001) creates `FileStorage` and `Logs` on a new
  volume; the pod's `fsGroup: 1001` gives the application write access.
- A ReadWriteOnce volume switches the Deployment to the `Recreate` strategy: the old pod stops before the new one
  starts, so an upgrade has a short downtime.
- Without the volume (`enabled: false`) logs, live templates and a Physical File Storage are lost with every pod.
- The PVC follows the StorageClass reclaim policy on `helm uninstall`; use `Retain` or `existingClaim` to keep it.

**File Storages** are created in the Admin UI (**App Fájltárolók**, and per workspace), not in `appsettings.json`.
The **Physical** backend writes below the data volume (root path `FileStorage`) and suits a single replica. For several
replicas use an external backend (S3, Azure Blob, MinIO, Box, SharePoint, Google Cloud Storage), or a ReadWriteMany
data volume. Object storage credentials need list and delete permission, and every installation its own bucket or
prefix.

**Editing a live template:**

```bash
POD=$(kubectl get pod -n worksmarthub -l app=worksmarthub-pod -o jsonpath='{.items[0].metadata.name}')
kubectl cp -n worksmarthub -c worksmarthub-container $POD:/app/data/LiveEmailTemplates/ResetPassword.cshtml ./ResetPassword.cshtml
# edit the file, then:
kubectl cp -n worksmarthub -c worksmarthub-container ./ResetPassword.cshtml $POD:/app/data/LiveEmailTemplates/ResetPassword.cshtml
```

The next e-mail uses it. Delete the file (`kubectl exec ... -- rm <file>`) and the pod to get the default back.

### Step 6: Custom CA certificates

For outbound TLS to endpoints with an internal or self-signed CA (see
[Custom CA certificates](#custom-ca-certificates-outbound-tls)), the chart merges your CA certificates into the trust
store of the application container:

```bash
kubectl create configmap wsh-ca-certs -n worksmarthub --from-file=internal-ca.crt=/path/to/internal-ca.crt
```

```yaml
customCaCerts:
  enabled: true
  configMapName: wsh-ca-certs     # or secretName: wsh-ca-certs
```

The init container `merge-ca-certs` (the application image, uid 65534) concatenates the system bundle with every
`*.crt` file (PEM) of the ConfigMap or Secret into an `emptyDir`, which is mounted read-only over
`/etc/ssl/certs/ca-certificates.crt`. No application setting is needed; it applies at the next pod start
(`kubectl rollout restart -n worksmarthub deploy/worksmarthub-deployment` after a change of the ConfigMap). This is
for outbound trust only; inbound TLS ends at the Ingress.

### Step 7: Secure credentials

Do not keep passwords and keys in `appsettings.json` or `values.yaml`: they end up in a ConfigMap and in the Helm
release, unencrypted. Put them into Kubernetes Secrets; the chart injects them as environment variables, which override
`appsettings.json`. The chart never creates a Secret; it only references them.

**Method 1: all keys of a Secret** (`secrets.envFromSecret`). Each key is an environment variable name:

```bash
kubectl create namespace worksmarthub        # once, if the chart does not create it (installNameSpace: False)
kubectl create secret generic wsh-secrets -n worksmarthub \
  --from-literal=ConnectionStrings__InternalDb='Server=pg.example.com;Port=5432;Database=wsh;User id=wsh;Password=<password>;SSL Mode=Require;' \
  --from-literal=LoginSettings__JwtAuthenticationSettings__SymmetricSecurityKey="$(openssl rand -base64 48)" \
  --from-literal=TicketAttachment__DownloadLink__SymmetricSecurityKey="$(openssl rand -base64 48)" \
  --from-literal=ImageConfiguration__ImageLink__SymmetricSecurityKey="$(openssl rand -base64 48)" \
  --from-literal=EmailSettings__SmtpPassword='<smtp password>'
```

```yaml
secrets:
  envFromSecret: "wsh-secrets"
```

Keys that are not valid environment variable names are skipped silently. With `envFromSecret` the chart cannot see
which database and cache the Secret selects, so its SQLite and InMemory replica checks are skipped.

**Method 2: single keys** (`secrets.env`), for Secrets with other key names:

```yaml
secrets:
  env:
  - name: ConnectionStrings__InternalDb
    secretName: wsh-db
    key: connection-string
  - name: LoginSettings__JwtAuthenticationSettings__SymmetricSecurityKey
    secretName: wsh-keys
    key: jwt
```

| `appsettings.json` path | Environment variable |
|-------------------------|----------------------|
| `ConnectionStrings.InternalDb` | `ConnectionStrings__InternalDb` |
| `CacheSettings.RedisCacheSettings.Configuration` | `CacheSettings__RedisCacheSettings__Configuration` |
| `LoginSettings.JwtAuthenticationSettings.SymmetricSecurityKey` | `LoginSettings__JwtAuthenticationSettings__SymmetricSecurityKey` |
| `TicketAttachment.DownloadLink.SymmetricSecurityKey` | `TicketAttachment__DownloadLink__SymmetricSecurityKey` |
| `ImageConfiguration.ImageLink.SymmetricSecurityKey` | `ImageConfiguration__ImageLink__SymmetricSecurityKey` |
| `EmailSettings.SmtpPassword` | `EmailSettings__SmtpPassword` |
| `UserSettings.LdapSettings.Password` | `UserSettings__LdapSettings__Password` |
| `UserSettings.AzureADSettings.ClientSecret` | `UserSettings__AzureADSettings__ClientSecret` |
| `LoginSettings.OAuthSettings.Okta.AppSecret` (and `Google`, `Microsoft`, `Custom`) | `LoginSettings__OAuthSettings__Okta__AppSecret` |

Type secrets into your shell or let a secret manager create the Secret; do not keep them in files. The **External
Secrets Operator** syncs a Secret from HashiCorp Vault, AWS Secrets Manager or Azure Key Vault; the resulting Secret is
referenced the same way:

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: wsh-secrets
  namespace: worksmarthub
spec:
  refreshInterval: 1h
  secretStoreRef: { name: vault-backend, kind: ClusterSecretStore }
  target: { name: wsh-secrets }
  data:
  - secretKey: ConnectionStrings__InternalDb
    remoteRef: { key: secret/data/wsh/database, property: connection-string }
  - secretKey: LoginSettings__JwtAuthenticationSettings__SymmetricSecurityKey
    remoteRef: { key: secret/data/wsh/keys, property: jwt }
```

Environment variables are read when a container starts; after a Secret changes, restart the pods:
`kubectl rollout restart -n worksmarthub deploy/worksmarthub-deployment`.

The built-in PostgreSQL's `internalDatabase.settings.password` stays in `values.yaml` (PostgreSQL reads it directly);
for production prefer an external database whose connection string is only in a Secret. Keep non-secret settings
(`InternalDbType`, `Schema`, `CacheSettings.Type`, `UserSettings`, log levels) in `appsettings.json` or `app.env`.
Enforce TLS on database connections: PostgreSQL `SSL Mode=Require` (or `VerifyFull`), SQL Server
`Encrypt=True;TrustServerCertificate=False`, MySQL `SslMode=Required`.

### Step 8: Service type and Ingress

**Service type** (`app.serviceType`) controls how the application is reachable from outside the cluster; the Service
listens on `advanced.app.servicePort` (80) and forwards to the container port 8080.

| Type | Reachable from | When |
|------|----------------|------|
| `ClusterIP` (default) | Inside the cluster | Behind the Ingress or a service mesh; `kubectl port-forward` for a test |
| `NodePort` | `<node IP>:<30000-32767>` | Quick access without an ingress controller; often blocked by cluster policies |
| `LoadBalancer` | An IP from your platform | Direct access in a cloud; internal with `app.serviceAnnotations` (`networking.gke.io/load-balancer-type: "Internal"`, `service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"`, `service.beta.kubernetes.io/azure-load-balancer-internal: "true"`) |

Pick one entry point: the Ingress with `ClusterIP`, or a `LoadBalancer`.

**Ingress** (`advanced.ingress.enabled: true` by default, `advanced.ingress.className: nginx`). `app.hostName` sets the
host rule (empty: every host). TLS:

```yaml
app:
  hostName: "wsh.example.com"
advanced:
  ingress:
    annotations:
      cert-manager.io/cluster-issuer: "your-issuer"
    tls:
    - hosts:
      - wsh.example.com
      secretName: wsh-tls
```

**Upload limits and timeouts.** ingress-nginx accepts 1 MB request bodies and ends a proxied request after 60 seconds
by default, while ticket attachments and component images upload in 25 MB chunks. The chart sets:

```yaml
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
```

Your `advanced.ingress.annotations` are merged over them: the same key with your value wins, and your other annotations
are kept. An annotation on the Ingress takes precedence over the controller's ConfigMap, so a limit you raised there
must be set here as well. If you raise the chunk sizes (`TicketAttachment.UploadChunkSize`,
`ImageConfiguration.ImageChunkSize`), keep `proxy-body-size` above them. Other ingress controllers ignore these
annotations; give them the same limits, and also any load balancer or WAF in front of the cluster.

**Disabling the Ingress** (`advanced.ingress.enabled: false`) for a service mesh or your own routing: route to the
Service `worksmarthub` on port 80, and set the forwarded headers as described under
[Reverse proxy and forwarded headers](#reverse-proxy-and-forwarded-headers).

### Step 9: Replicas and scaling

```yaml
app:
  replicationFactor: 1
  resources:
    requests: { memory: 512M, cpu: 250m }
    limits: { memory: 2000M, cpu: 2000m }
```

One replica (the default) works with every setting. **More than one replica** needs all of these:

- a shared database: the built-in PostgreSQL or an external one (**SQLite supports one replica only**);
- a shared cache: Redis (the default) or a database cache;
- a data volume every pod can mount: `accessMode: ReadWriteMany` with an RWX StorageClass (NFS, EFS, Azure Files,
  Filestore), or no data volume (`enabled: false`; logs and live templates then stay in each pod);
- an external File Storage backend for attachments and images.

The chart refuses to render (`helm install` and `helm upgrade` stop with a message) with more than one replica together
with a ReadWriteOnce data volume, SQLite, or the in-memory cache; `helm lint` shows the refusal as an INFO line. The
behaviour of several replicas (migrations under a lock, sessions, background jobs once per occurrence) is described
under [Multi-node and high availability](#multi-node-and-high-availability). Replicas are spread over nodes by a
preferred pod anti-affinity.

**Node placement:** `advanced.nodeConfiguration.appNodesConfiguration`, `dbNodesConfiguration` and
`cacheNodesConfiguration` take `nodeSelector`, `affinity` and `tolerations` for the respective pods. A dedicated node
pool for the application lets a firewall rule allow its outbound connections from those nodes only:

```yaml
advanced:
  nodeConfiguration:
    appNodesConfiguration:
      nodeSelector:
        role: wsh
      tolerations:
      - { key: role, operator: Equal, value: wsh, effect: NoSchedule }
```

An `affinity` given there replaces the chart's anti-affinity.

### Step 10: Install

From this repository:

```bash
helm install worksmarthub ./kubernetes-helm-chart -f my-values.yaml --set app.image.tag=23
```

Preview first with `helm template worksmarthub ./kubernetes-helm-chart -f my-values.yaml --set app.image.tag=23`. The
`test-local` scripts check the chart on a local cluster before you install it elsewhere
([test-local/README.md](https://github.com/WorkWise-Solutions/WorkSmartHub-Docker-Kubernetes/blob/main/test-local/README.md)).

### Step 11: Verify

```bash
kubectl get pods -n worksmarthub
# worksmarthub-deployment-...  1/1 Running   (app; init containers prepare-data, wait-for-db, wait-for-cache)
# db-deployment-...            1/1 Running   (built-in PostgreSQL)
# cache-deployment-...         1/1 Running   (built-in Redis)

kubectl logs -n worksmarthub deploy/worksmarthub-deployment -c worksmarthub-container
# ... Migrating database ... Applying migration '...' ... Migration complete

kubectl logs -n worksmarthub deploy/worksmarthub-deployment -c wait-for-db      # PostgreSQL accepts the login from values.yaml.
kubectl get pvc,svc,ingress -n worksmarthub
```

The first start runs all migrations (about a minute on a small cluster); the pod is Ready when `/health` answers. Open
the application:

- with the Ingress and a host name: `https://wsh.example.com/`;
- with the Ingress without a host name: the address of the Ingress (`kubectl get ingress -n worksmarthub`);
- without Ingress: `kubectl port-forward -n worksmarthub svc/worksmarthub 8080:80`, then `http://localhost:8080/`.

Create the first user on the start page.

### Step 12: Activate the licence

The licence is uploaded in the application and stored in the database; there is no Helm value for it. It survives pod
restarts and upgrades, and every replica reads the same licence.

**In the portal:** open **Licenc** (`/portal/license`) and choose **Új licenc fájl feltöltése**. The page shows the
status, the customer, the start, expiry and renewal warning dates.

**Who can upload.** While the application has no valid licence (none, expired or corrupted), anyone can upload one,
signed in or not, so a new installation can be licensed before its first user exists. A valid licence, also one in its
renewal warning period, can be replaced by a **Super Admin** only (the `SuperAdmin` permission); other callers get the
error `generic.not_authorized`. A file that is not a valid licence is refused with `license.error`.

**Through the API:**

```bash
# no valid licence yet: no sign-in needed
curl -X POST https://wsh.example.com/api/licensing/set \
  -H "Content-Type: application/octet-stream" --data-binary @license.key

# status (no sign-in needed)
curl https://wsh.example.com/api/licensing/status
```

Replacing a valid licence through the API needs a Super Admin's personal access token (created on **Személyes
Hozzáférési Tokenek**, `/portal/pat`). Exchange its refresh token for an access token and send that as a Bearer token.
Every refresh returns a **new** refresh token and the previous one stops working, so store the new one:

```bash
RESPONSE=$(curl -s -X POST https://wsh.example.com/api/pats/refresh-token \
  -H "Content-Type: application/json" -d '{"refreshToken": "<refresh token>"}')
NEW_REFRESH_TOKEN=$(echo "$RESPONSE" | jq -r '.refreshToken.token')     # keep it in place of the old one
ACCESS_TOKEN=$(echo "$RESPONSE" | jq -r '.accessToken.token')

curl -X POST https://wsh.example.com/api/licensing/set \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/octet-stream" --data-binary @license.key
```

The access token can also be sent as `?access_token=<access token>`; the header keeps it out of access logs. A personal
access token authenticates API calls only; it never starts a browser session.

### Upgrade and rollback

```bash
bash build/build-image.sh --version 24 --minikube       # or push worksmarthub:24 to your registry
helm upgrade worksmarthub ./kubernetes-helm-chart -f my-values.yaml --set app.image.tag=24
kubectl rollout status -n worksmarthub deploy/worksmarthub-deployment
kubectl logs -n worksmarthub deploy/worksmarthub-deployment -c worksmarthub-container | grep -E "Migrat|ERR|FTL"
```

- Take a database backup first (see [Migrating between environments](#migrating-between-environments) for the commands).
- Use the same values as for the install and change only the tag. A newer version of this repository may bring new
  chart defaults: compare its `values.yaml` and `appsettings.json` with yours before upgrading.
- The application pod is replaced (with a ReadWriteOnce data volume the old pod stops first: a short downtime),
  migrates the database at start and keeps the data volume. The database and cache pods, the volumes and the users'
  sessions stay.
- A change of the chart's `appsettings.json` restarts the application pod as well; a change of a Secret needs
  `kubectl rollout restart`.

**Rollback.** `helm history worksmarthub` lists the revisions; `helm rollback worksmarthub <revision>` restores the
manifest and image of that revision. **The database is not migrated back**: the earlier version then runs on the newer
schema. To go back to an earlier application version, restore the database backup taken before the upgrade.

**Testing an upgrade first.** `test-local/upgrade-test.sh` performs the procedure on minikube: install with image A,
create data (first user, a second user, a sign-in, a marker file on the data volume, an edited template), `helm upgrade`
to image B, check that only the application pod was replaced (0 restarts), the PVCs, database and cache pods kept their
UIDs, the data and the pre-upgrade sign-in still work and the migrations ran, then `helm rollback` and the same checks:

```bash
bash test-local/upgrade-test.sh --from 23 --to 24
```

### Migrating between environments

To move an installation (for example from Docker or a Windows server to Kubernetes, or to another cluster), copy the
**database** and the **data volume**; together they hold the whole state: users, workspaces, tickets, clients,
invoices, File Storage and notification channel definitions, the licence, the sign-in keys and the audit log. Stored
credentials decrypt on the new installation without anything from the old host.

- Keep the **same database engine** (no built-in conversion) and run the **same or a newer** application version on the
  new side; it migrates the schema at its first start.
- Use the **same signing keys** (`TicketAttachment.DownloadLink.SymmetricSecurityKey`,
  `ImageConfiguration.ImageLink.SymmetricSecurityKey`, `LoginSettings.JwtAuthenticationSettings.SymmetricSecurityKey`),
  or existing links and tokens stop working; set `PublicBaseUrl` to the new address.
- Do not run both installations against the same external File Storage buckets afterwards (see
  [Multi-node and high availability](#multi-node-and-high-availability)).

With the built-in PostgreSQL (stop the application first: `kubectl scale -n worksmarthub deploy/worksmarthub-deployment --replicas=0`):

```bash
# database out of the old installation and into the new one
kubectl exec -n worksmarthub deploy/db-deployment -- sh -c 'pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Fc' > wsh-db.dump
kubectl exec -i -n <new namespace> deploy/db-deployment -- sh -c 'pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" --clean --if-exists --no-owner' < wsh-db.dump

# data volume (run while an application pod exists, e.g. right after scaling the new one to 1)
kubectl exec -n worksmarthub deploy/worksmarthub-deployment -c worksmarthub-container -- tar czf - -C /app/data . > wsh-data.tgz
kubectl exec -i -n <new namespace> deploy/worksmarthub-deployment -c worksmarthub-container -- tar xzf - -C /app/data < wsh-data.tgz
```

From Docker, the volume backup ([Backup and restore](#backup-and-restore)) is the data archive, and a SQLite database
moves with it (`database.db` in the archive; use the [SQLite](#sqlite-single-replica) values on the new side). Restore
the database before the new application starts for the first time, or start it once, stop it, restore, and start again.
Delete the dump and archive files afterwards; they hold credentials.

### Uninstall

```bash
helm uninstall worksmarthub
```

- With `installNameSpace: True` (the default) the namespace belongs to the release and is deleted with everything in
  it, including the PVCs `worksmarthub-data-claim` and `postgres-data-claim`. Whether the data survives depends on the
  StorageClass reclaim policy: `Delete` (common) removes it, `Retain` keeps the PersistentVolumes as `Released`.
- Kept: claims given as `existingClaim`, a namespace you created yourself (`installNameSpace: False`) with your Secrets
  in it (delete it with `kubectl delete namespace worksmarthub`), and the images on the nodes.
- Back up the database and the data volume first (commands under [Migrating between environments](#migrating-between-environments)).

### Production readiness checklist

The defaults suit a first install, not production. Before going live:

**Data and durability**
- [ ] An **external managed database** with backups and point-in-time recovery, or at least regular dumps of the
  built-in PostgreSQL ([Step 3](#step-3-internal-database)). The built-in one is a single copy without failover.
- [ ] A **durable StorageClass** with encryption at rest, and `Retain` or `existingClaim` where the data must outlive
  `helm uninstall` ([Step 5](#step-5-application-settings)).
- [ ] **File Storage** fits the replica count: Physical on the data volume for one replica, an external backend for more.
- [ ] A **shared cache** (Redis) if more than one replica ([Step 4](#step-4-cache)).

**Security**
- [ ] The **default database password** changed ([Database password](#database-password)).
- [ ] **Own signing keys** for `LoginSettings.JwtAuthenticationSettings`, `TicketAttachment.DownloadLink` and
  `ImageConfiguration.ImageLink`, from a Secret; no `SECURITY:` warning at start ([Step 7](#step-7-secure-credentials)).
- [ ] **`PublicBaseUrl`** set to the public address.
- [ ] Credentials in **Secrets**, not in `values.yaml` or `appsettings.json`.
- [ ] **TLS** at the Ingress with a valid certificate; HSTS on in **Biztonsági fejlécek**; the Service `ClusterIP`.
- [ ] **TLS to the database** (`SSL Mode=Require`, `Encrypt=True`, `SslMode=Required`).
- [ ] `UseDetailedExceptions` `false`; `Swagger:Enabled` `false` unless needed.
- [ ] The **identity provider** for production (`LDAP` or `AzureAD`, or OAuth) if not local users.
- [ ] **NetworkPolicy** enforcement (a capable CNI) and the egress rules your security team needs.

**Availability and operations**
- [ ] **Replicas** and **resources** sized; with several replicas all multi-node conditions met ([Step 9](#step-9-replicas-and-scaling)).
- [ ] **Log collection** from stdout (`kubectl logs`) into your logging stack; log levels `Information` or `Warning`.
- [ ] The **licence** activated ([Step 12](#step-12-activate-the-licence)).
- [ ] An **upgrade** tested on a copy ([Upgrade and rollback](#upgrade-and-rollback)).

### Important notes

**Security context.** Every container runs as non-root with all Linux capabilities dropped and no privilege
escalation, every pod has `seccompProfile: RuntimeDefault`, and the application's ServiceAccount does not mount an API
token. That satisfies the `restricted` Pod Security Standard, except the optional `fix-db-permissions` init container:

| Container | UID | Notes |
|-----------|-----|-------|
| Application (`worksmarthub-container`) | 1001 | `appuser` |
| `prepare-data` (init, with the data volume) | 1001 | Creates `FileStorage` and `Logs` |
| `wait-for-db` (init, with the built-in PostgreSQL) | 70 | `psql` login check |
| `wait-for-cache` (init, with the built-in Redis) | 65534 | `nc` check, up to 5 minutes |
| `merge-ca-certs` (init, with `customCaCerts`) | 65534 | Concatenates PEM files |
| PostgreSQL | 70 | `fsGroup: 70` |
| Redis | 999 | |
| `fix-db-permissions` (init, only with `internalDatabase.fixDataPermissions: True`) | 0 | Drops all capabilities except `CHOWN`; off by default |

**Network policies.** The built-in PostgreSQL and Redis get ingress NetworkPolicies (`internalServicesNetworkPolicy.enabled`,
default on) that admit only pods of the namespace; `internalServicesNetworkPolicy.extraIngress` adds sources outside it
(for example a backup job's namespace). They select the pods by `app` together with the chart-specific label
`wsh-internal-service: db` / `cache`, so they never match another workload's pods. NetworkPolicies are enforced only by
a CNI that supports them (Calico, Cilium, Weave Net).

**Probes.** `GET /health` on port 8080: a startup probe allows 10 minutes for the first start (migrations), the
readiness probe takes the pod out of the Service when it stops answering, and the liveness probe restarts it after 2.5
minutes without an answer. `terminationGracePeriodSeconds: 60` covers the application's 45-second shutdown of its
background jobs. The PostgreSQL pod has a `pg_isready` readiness probe, the Redis pod a TCP probe.

**Configuration changes** of the chart's `appsettings.json` roll the application pod (`checksum/config` annotation).

### Air-gapped and restricted networks

Without outbound internet access:

1. **Application image:** build it where the sources are, `--save worksmarthub-23.tar`, and load it on the nodes or push
   it to your internal registry (`app.image.repository`).
2. **Infrastructure images:** pull `postgres:16-alpine`, `redis:7-alpine` and `busybox:1.38.0` outside, push them to
   your registry, and set `internalDatabase.image`, `cacheSystem.image` and `busyboxImage`. The `wait-for-db` init
   container uses `internalDatabase.image`, `prepare-data` and `merge-ca-certs` use the application image.
   ```bash
   for i in postgres:16-alpine redis:7-alpine busybox:1.38.0; do docker pull $i; docker tag $i registry.example.com/$i; docker push registry.example.com/$i; done
   ```
3. **Pull policy:** keep `IfNotPresent`, so pod restarts need no registry.
4. **The application's own outbound connections** (Számlázz.hu, Slack, Teams, Entra ID, object stores) need the
   destinations of [Outbound connectivity](#outbound-connectivity), or the features that use them stay off.

---

## Health checks

`GET /health` answers `200 {"status":"Healthy"}` without authentication as soon as the application serves requests,
after the start-up migrations. It touches nothing else: no database call, no version or node details. The chart's
probes use it. The image has no `HEALTHCHECK` instruction and no curl or wget; `bash` in the image can send the
request, for example in Docker Compose:

```yaml
healthcheck:
  test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/127.0.0.1/8080 && printf 'GET /health HTTP/1.0\\r\\nHost: localhost\\r\\n\\r\\n' >&3 && grep -q '^HTTP/1.[01] 200' <&3"]
  interval: 30s
  timeout: 5s
  retries: 3
  start_period: 120s
```

Load balancers and monitoring check `GET /health` for status 200. The Admin UI's status page and the log files show
more detail to administrators.

## Troubleshooting

**Where the logs are.** Docker: `docker logs -f worksmarthub` (and `journalctl -u worksmarthub -f` for the service, the
`worksmarthub-task.log` next to the Windows script for the task's start-up). Kubernetes:
`kubectl logs -n worksmarthub deploy/worksmarthub-deployment -c worksmarthub-container -f`, the init containers with
`-c prepare-data`, `-c wait-for-db`, `-c wait-for-cache`, `-c merge-ca-certs`, and a crashed container's previous run
with `--previous`. The daily files are in `Logs/` of the data volume and in the Admin UI (**Naplófájlok**).

**Messages at the first start on PostgreSQL that need no action.** On a new, empty PostgreSQL database the log shows
`Cannot load library libgssapi_krb5.so.2` (the PostgreSQL driver looks for the optional Kerberos library, which the image
does not contain, and continues without it) and once `ERR Failed executing DbCommand ... SELECT "MigrationId" ... FROM
"__EFMigrationsHistory"` (the migration history table does not exist yet; it is created right after, and the
`Applying migration` lines and `Migration complete` follow). Neither line appears when the database is already migrated.

**The run script exits with code 78.** A preflight check failed and the message names it: the image is missing, a
container of that name runs already, a stopped container without the data volume exists, or a host directory is not
prepared. The systemd service does not retry this exit code.

**Pods stay `Pending`.** `kubectl describe pod -n worksmarthub <pod>`: usually an unbound PVC (no default StorageClass,
a misspelled `storageClass`, or a ReadWriteOnce volume still attached elsewhere) or too little CPU or memory for the
requests.

**`ErrImagePull` / `ImagePullBackOff`.** The node does not have the image: build it into that node's daemon
(`--minikube`), load it there, or push it to a registry and set `app.image.repository` and `app.imagePullSecrets`. A
`helm upgrade` with a tag the nodes do not have fails the same way.

**The database pod is in `CrashLoopBackOff`.** `kubectl logs -n worksmarthub deploy/db-deployment`. `Permission denied`
on the data directory: the storage ignores `fsGroup`; set `internalDatabase.fixDataPermissions: True` or use a CSI
class. `database files are incompatible with server`: the image's major version differs from the data directory.

**The application pod restarts and its log shows `28P01: password authentication failed`.** The database refused the
password. With the built-in PostgreSQL the data volume was created earlier with another password (changed later in the
values, or a reinstall on a kept volume); the `wait-for-db` log says whether the `values.yaml` login is refused too. Set
the database password to the one in the connection string ([Database password](#database-password)) or put the original
back; on a new installation without data you can also uninstall, delete the `postgres-data-claim` PVC and install again.

**The application cannot reach the database.** The built-in database's host name is `db` (the Service), port 5432; the
credentials of the connection string must match `internalDatabase.settings`. For an external database check DNS from
the pod (`kubectl exec ... -- getent hosts <host>`), the firewall, and TLS settings.

**SQLite migration lock.** The migrations take a lock; on SQLite it is a row in the `__EFMigrationsLock` table, and a
process stopped in the middle of a migration (a killed container, a node failure) leaves that row behind. SQLite serves
one node only, so the application treats such a row as stale: it removes it at the next start, logs
`Removed a stale SQLite migration lock left by an earlier start that did not finish migrating`, and migrates. On SQL
Server, PostgreSQL and MySQL the lock is held by the database connection and ends with it; a second node waits for the
first one to finish migrating.

**The pod is not Ready / probes fail.** `kubectl describe pod` shows the probe results. The probes call `/health` on
port 8080; the listen address (`HttpSettings.Urls` or `HttpSettings__Urls__0`) must use that port. During the first start
the startup probe waits up to 10 minutes for the migrations; a slower start (a large database, a slow disk) shows as
restarts with `Startup probe failed` in the events.

**`appsettings.json is read-only` in the log.** Information only: the file is mounted read-only (always on Kubernetes)
or not writable by uid 1001. New settings keys of a newer release use their defaults in memory; add them to your file
when you need other values.

**`SECURITY:` warnings at start.** A signing key still has the shipped value, or `PublicBaseUrl` is empty; see
[Settings to set before the first real use](#settings-to-set-before-the-first-real-use).

**Upload fails with 413 Request Entity Too Large.** A proxy rejected the body. Check, in this order: a
`proxy-body-size` you set in `advanced.ingress.annotations` (yours wins), an Ingress or controller you manage yourself
(ingress-nginx defaults to 1 MB), a reverse proxy in front of Docker (`client_max_body_size`), and any load balancer or
WAF. A 413 page that names nginx comes from a proxy, not from the application. A 504 after 60 seconds is the proxy's
read timeout.

**Outbound TLS fails (`The remote certificate is invalid`, `unable to get local issuer certificate`).** The endpoint's
CA is not trusted: add it ([Custom CA certificates](#custom-ca-certificates-outbound-tls), [Step 6](#step-6-custom-ca-certificates)).

**The browser shows the previous version after an update.** The server sends the start page with `Cache-Control:
no-cache`, so browsers check it on every load and get the new portal; the script and style bundles carry a content
hash in their file names. A browser that cached the start page of an older build that did not send this header keeps
it until its cache expires or is cleared.

**Ingress not working.** Check the controller (`kubectl get pods -n ingress-nginx`) and the class
(`advanced.ingress.className`). With another controller or a service mesh, disable the chart's Ingress and route to the
Service `worksmarthub` on port 80.
