dokku/docs/deployment/schedulers/k3s.md

# K3s Scheduler

> [!IMPORTANT]
> New as of 0.33.0

```
scheduler-k3s:annotations:set <app|--global> <property> (<value>) [--process-type PROCESS_TYPE] <--resource-type RESOURCE_TYPE> # Set or clear an annotation for a given app/process-type/resource-type combination
scheduler-k3s:annotations:report [<app>|--global] [--format stdout|json] [--process-type PROCESS_TYPE] [--resource-type RESOURCE_TYPE] # Displays a scheduler-k3s annotations report for one or more apps
scheduler-k3s:autoscaling-auth:set <app|--global> <trigger> [<--metadata key=value>...] # Set or clear a scheduler-k3s autoscaling keda trigger authentication resource for an app
scheduler-k3s:autoscaling-auth:report [<app>|--global] [--format stdout|json] [--include-metadata] # Displays a scheduler-k3s autoscaling auth report for one or more apps
scheduler-k3s:charts:report [<chart>] [--format stdout|json] # Displays a scheduler-k3s chart override report
scheduler-k3s:charts:set <chart-name.property> (<value>) # Set or clear a chart-specific helm value
scheduler-k3s:cluster:add [--profile PROFILE] [--role ROLE] [--insecure-allow-unknown-hosts] [--server-ip SERVER_IP] [--taint-scheduling] [--kubelet-args KUBELET_ARGS] <ssh://user@host:port> # Adds a server node to a Dokku-managed cluster
scheduler-k3s:cluster:list [--format json|stdout] # Lists all nodes in a Dokku-managed cluster
scheduler-k3s:cluster:remove [node-id] # Removes client node to a Dokku-managed cluster
scheduler-k3s:ensure-charts # Ensures the k3s charts are installed
scheduler-k3s:initialize [--server-ip SERVER_IP] [--taint-scheduling] [--kubelet-args KUBELET_ARGS] # Initializes a cluster
scheduler-k3s:labels:set <app|--global> <property> (<value>) [--process-type PROCESS_TYPE] <--resource-type RESOURCE_TYPE> # Set or clear a label for a given app/process-type/resource-type combination
scheduler-k3s:labels:report [<app>|--global] [--format stdout|json] [--process-type PROCESS_TYPE] [--resource-type RESOURCE_TYPE] # Displays a scheduler-k3s labels report for one or more apps
scheduler-k3s:node-sysctls:set <sysctl> (<value>) [--global|--profile PROFILE] # Set or clear a node-level kernel sysctl for unprofiled nodes or a single node profile
scheduler-k3s:node-sysctls:report [--format stdout|json] # Displays the node-level kernel sysctls applied to each scope
scheduler-k3s:preview <app> [--context N] [--show-secrets] [--show-secrets-decoded] # Displays a diff between the current and next deployment for an app
scheduler-k3s:profiles:add <profile> [--role ROLE] [--insecure-allow-unknown-hosts] [--taint-scheduling] [--kubelet-args KUBELET_ARGS] # Adds a node profile to the k3s cluster
scheduler-k3s:profiles:list [--format json|stdout] # Lists all node profiles in the k3s cluster
scheduler-k3s:profiles:remove <profile> # Removes a node profile from the k3s cluster
scheduler-k3s:report [<app>|--global] [--format stdout|json] [<flag>] # Displays a scheduler-k3s report for one or more apps
scheduler-k3s:set <app|--global> <property> (<value>) # Set or clear a scheduler-k3s property for an app or globally
scheduler-k3s:show-kubeconfig # Displays the kubeconfig for remote usage
scheduler-k3s:uninstall # Uninstalls k3s from the Dokku server
```

> [!NOTE]
> The k3s plugin replaces the external [scheduler-kubernetes](https://github.com/dokku/dokku-scheduler-kubernetes) plugin. Users can continue to use the external plugin as necessary, but all future development will occur on the official core k3s plugin.

For multi-server support, Dokku provides the ability for users to setup a K3s cluster. As with all schedulers, it is set on a per-app basis. The scheduler can currently be overridden by running the following command:

```shell
dokku scheduler:set node-js-app selected k3s
```

As it is the default, unsetting the `selected` scheduler property is also a valid way to reset the scheduler.

```shell
dokku scheduler:set node-js-app k3s
```

## Usage

> [!IMPORTANT]
> The k3s plugin requires usage of a docker registry to store deployed image artifacts. See the [registry documentation](/docs/advanced-usage/registry-management.md) for more details on how to configure a registry.

### Initializing a cluster

> [!WARNING]
> This command must be run as root

Clusters can be initialized via the `scheduler-k3s:initialize` command. This will start a k3s cluster on the Dokku node itself.

```shell
dokku scheduler-k3s:initialize
```

By default, the k3s installation can run both app and system workloads. For clusters where app workloads are run on distinct worker nodes, initialize the cluster with the `--taint-scheduling` flag, which will allow _only_ Critical cluster components on the k3s control-plane nodes.

```shell
dokku scheduler-k3s:initialize --taint-scheduling
```

By default, Dokku will attempt to auto-detect the IP address of the server. In cases where the auto-detected IP address is incorrect, an override may be specified via the `--server-ip` flag:

```shell
dokku scheduler-k3s:initialize --server-ip 192.168.20.15
```

Dokku's k3s integration natively uses `nginx` as it's ingress load balancer via [ingress-nginx](https://github.com/kubernetes/ingress-nginx). Properties set by the `nginx` plugin will be respected, either by turning them into annotations or creating a custom server/location snippet that the `ingress-nginx` project can use. A `ps:restart` is required after changing nginx properties in order to have them apply to running resources.

Dokku can also use Traefik on cluster initialization via the [Traefik's CRDs](https://doc.traefik.io/traefik/providers/kubernetes-crd/). To change the ingress, set the `--ingress-class` flag:

```shell
dokku scheduler-k3s:initialize --ingress-class traefik
```

Kubelet flags for the initial server node can be supplied by passing `--kubelet-args` with a comma-separated `key=value` list. This is the only way to configure the kubelet on the node created by `scheduler-k3s:initialize`, as that node never passes through `scheduler-k3s:cluster:add`.

```shell
dokku scheduler-k3s:initialize \
  --kubelet-args allowed-unsafe-sysctls=net.ipv6.conf.all.disable_ipv6
```

Multiple kubelet arguments can be specified in the same call by separating them with commas.

```shell
dokku scheduler-k3s:initialize \
  --kubelet-args allowed-unsafe-sysctls=net.ipv6.conf.all.disable_ipv6,max-pods=150
```

### Adding nodes to the cluster

> [!WARNING]
> The `dokku` user _must_ be able to ssh onto the server in order to connect nodes to the cluster. The remote user must be root or have sudo enabled, or the install will fail.

#### Adding a worker node

Nodes that run app workloads can be added via the `scheduler-k3s:cluster:add` command. This will ssh onto the specified server, install k3s, and join it to the current Dokku node in worker mode. Workers are typically used to run app workloads.

```shell
dokku scheduler-k3s:cluster:add  ssh://root@worker-1.example.com
```

Per-node kubelet flags can be supplied by passing `--kubelet-args` with a comma-separated `key=value` list. This is useful for tuning scheduler capacity or enforcing cluster-wide defaults at the node level.

```shell
dokku scheduler-k3s:cluster:add \
  --kubelet-args allowed-unsafe-sysctls=net.ipv6.conf.all.disable_ipv6 \
  ssh://root@worker-1.example.com
```

Multiple kubelet arguments can be specified in the same call by separating them with commas. The following example enables IPv4 forwarding while also increasing the pod density on the worker.

```shell
dokku scheduler-k3s:cluster:add \
  --kubelet-args allowed-unsafe-sysctls=net.ipv6.conf.all.disable_ipv6,max-pods=150 \
  ssh://root@worker-2.example.com
```

If the server isn't in the `known_hosts` file, the connection will fail. This can be bypassed by setting the `--insecure-allow-unknown-hosts` flag:

```shell
dokku scheduler-k3s:cluster:add --insecure-allow-unknown-hosts ssh://root@worker-1.example.com
```

By default, Dokku will attempt to auto-detect the IP address of the Dokku server for the remote server to connect to. In cases where the auto-detected IP address is incorrect, an override may be specified via the `--server-ip` flag:

```shell
dokku scheduler-k3s:cluster:add --server-ip 192.168.20.15 ssh://root@worker-1.example.com
```

#### Adding a server node

> [!NOTE]
> Only the initial Dokku server will be properly configured for push deployment, and should be considered your git remote. Additional server nodes are for ensuring high-availability of the K3s etcd state. Ensure this server is properly backed up and restorable or deployments will not work.

Server nodes are typically used to replicate the cluster state, and it is recommended to have an odd number of nodes spread across several availability zones (datacenters in close proximity within a region). This allows for higher availability in the event of a cluster failure. Server nodes run control-plane services such as the traefik load balancer and the etcd backing store.

Server nodes can also be added with the `scheduler-k3s:cluster:add` command by specifying `--role server`. This will ssh onto the specified server, install k3s, and join it to the current Dokku node in server mode.

```shell
dokku scheduler-k3s:cluster:add  --role server ssh://root@server-1.example.com
```

Server nodes allow any workloads to be scheduled on them by default, in addition to the control-plane, etcd, and the scheduler itself. To avoid app workloads being scheduled on your control-plane, use the `--taint-scheduling` flag:

```shell
dokku scheduler-k3s:cluster:add --role server --taint-scheduling ssh://root@server-1.example.com
```

If the server isn't in the `known_hosts` file, the connection will fail. This can be bypassed by setting the `--insecure-allow-unknown-hosts` flag:

```shell
dokku scheduler-k3s:cluster:add --role server --insecure-allow-unknown-hosts ssh://root@server-1.example.com
```

By default, Dokku will attempt to auto-detect the IP address of the Dokku server for the remote server to connect to. In cases where the auto-detected IP address is incorrect, an override may be specified via the `--server-ip` flag:

```shell
dokku scheduler-k3s:cluster:add --role server --server-ip 192.168.20.15 ssh://root@server-1.example.com
```

#### Changing the network interface

When attaching an worker or server node, the K3s plugin will look at the IP associated with the `eth0` interface and use that to connect the new node to the cluster. To change this, set the `network-interface` property to the appropriate value.

```shell
dokku scheduler-k3s:set --global network-interface eth1
```

### Node Profiles

Node profiles capture repeatable `scheduler-k3s:cluster:add` options so you can join multiple nodes with identical settings. A profile name can be specified for the `scheduler-k3s:cluster:add` command via the  `--profile <name>` flag. Any flags passed directly to `scheduler-k3s:cluster:add` override the stored values for that run.

#### Listing profiles

Display stored profiles to understand which roles and behaviors will be used.

```shell
dokku scheduler-k3s:profiles:list
```

```
name            role
awesome-profile worker
```

This command also takes an optional `--format` flag to specify a format for the output. Options include `json` and `stdout`

#### Adding profiles

Create or update a profile that defines how new nodes should be prepared before joining the cluster.

```shell
dokku scheduler-k3s:profiles:add edge-workers \
  --role worker \
  --insecure-allow-unknown-hosts \
  --kubelet-args protect-kernel-defaults=true,eviction-hard=memory.available<200Mi
```

Profile names must be alphanumeric, may include internal dashes, cannot start/end with a dash, and must be ≤32 characters. Other than the `--server-ip` flag, all flags used for `scheduler-k3s:cluster:add` are valid for the `scheduler-k3s:profiles:add` command.

#### scheduler-k3s:profiles:remove

Delete a profile once it’s no longer required.

```shell
dokku scheduler-k3s:profiles:remove edge-workers
```

Removal only deletes the stored definition; nodes that already joined the cluster keep their existing configuration.

#### The node profile label

When a node joins via `scheduler-k3s:cluster:add --profile <name>`, Dokku labels it with `dokku.com/node-profile=<name>`. This makes a profile selectable after the fact, whether via `kubectl`, a `nodeSelector`, or a node affinity rule.

```shell
kubectl get nodes -L dokku.com/node-profile
```

Nodes added without `--profile` are not labeled, as an empty label value would be indistinguishable from a profile literally named the empty string.

Two limits are worth knowing before relying on this label:

- The server node never carries it. That node is created by `scheduler-k3s:initialize` and never passes through `scheduler-k3s:cluster:add`, so no profile is ever associated with it.
- Nodes that joined before this label existed are not backfilled. Use `kubectl label node <node> dokku.com/node-profile=<name>` to set it on an existing node.

### Changing deployment settings

The k3s plugin provides a number of settings that can be used to managed deployments on a per-app basis. The following table outlines ones not covered elsewhere:

| Name                  | Description                                       | Global Default     |
|-----------------------|---------------------------------------------------|--------------------|
| `deploy-timeout`      | Controls when app deploys will timeout in seconds | `300s`             |
| `kustomize-root-path` | Controls the folder context from the deployed repository used for Kustomize | `config/kustomize` |
| `image-pull-secrets`  | Name of a kubernetes secret used to auth against a registry | Contents of `~/.docker/config.json` from Dokku server |
| `namespace`           | Controls the namespace used for resource creation | `default`          |
| `rollback-on-failure` | Whether to rollback failed deploys                | `false`            |
| `shm-size`            | Default shared memory size for pods               | Kubernetes default |

All settings can be set via the `scheduler-k3s:set` command. Using `deploy-timeout` as an example:

```shell
dokku scheduler-k3s:set node-js-app deploy-timeout 60s
```

The default value may be set by passing an empty value for the option in question:

```shell
dokku scheduler-k3s:set node-js-app deploy-timeout
```

Properties can also be set globally. If not set for an app, the global value will apply.

```shell
dokku scheduler-k3s:set --global deploy-timeout 60s
```

The global default value may be set by passing an empty value for the option.

```shell
dokku scheduler-k3s:set --global deploy-timeout
```

### Restarting apps

A `ps:restart` re-renders the app's Helm chart from its current configuration and upgrades the release, which is how configuration changes are picked up. Pods cycle because each Deployment's pod template carries an `app.kubernetes.io/version` annotation that changes on restart.

A single process type may be targeted, in which case only that process type's pods are replaced. The rest of the release is still upgraded so configuration converges everywhere, but the untargeted Deployments keep their existing annotation and are left running:

```shell
dokku ps:restart node-js-app web
```

The app image does not need to be present on the Dokku host. Kubernetes pulls it from the registry, so a host that has reaped its local copy - as the `registry` plugin does on its own once an app has been deployed a number of times - can still restart, scale, and run one-off commands against the app. The builder type and working directory needed to render the chart are read back from the app's current Helm release when the image is unavailable locally.

There is one exception. An `app.json` with a `scripts.dokku.postdeploy` task runs that task in a container on the Dokku host rather than in the cluster, and so does require the image locally. Apps without a postdeploy task are unaffected.

### Displaying the scheduler report

Configured properties can be inspected with the `scheduler-k3s:report` command. Without arguments, it iterates every app. Passing an app name scopes the report to that app, while `--global` reports the scheduler-wide properties on their own:

```shell
dokku scheduler-k3s:report
dokku scheduler-k3s:report node-js-app
dokku scheduler-k3s:report --global
```

The output can be emitted as JSON for programmatic consumption:

```shell
dokku scheduler-k3s:report --global --format json
```

A single value can also be read directly by passing its report flag, as listed in the [settable properties](#settable-properties) table:

```shell
dokku scheduler-k3s:report node-js-app --scheduler-k3s-computed-deploy-timeout
```

### Previewing deployment changes

Before triggering a deploy, the `scheduler-k3s:preview` command can be used to display a unified diff between the manifests currently stored in the live Helm release for an app and the manifests that the next deploy would roll out:

```shell
dokku scheduler-k3s:preview node-js-app
```

A clean redeploy with no property changes produces no output. After modifying a property that affects the main app chart (for example `dokku resource:limit --memory 256m node-js-app`), the command shows the unified diff for each changed resource. For an app that has never been deployed, the entire proposed manifest is rendered as added lines.

By default each change is shown with 3 lines of surrounding context, git-diff style. The `--context` flag overrides this; pass `-1` to render the full resource around every change:

```shell
dokku scheduler-k3s:preview node-js-app --context 0
dokku scheduler-k3s:preview node-js-app --context -1
```

By default any `kind: Secret` resources have their `data` values redacted so secret bytes never appear in terminal scrollback or CI logs. Two flags adjust this behavior:

```shell
dokku scheduler-k3s:preview node-js-app --show-secrets
dokku scheduler-k3s:preview node-js-app --show-secrets-decoded
```

`--show-secrets` keeps the raw base64-encoded values, and `--show-secrets-decoded` renders them base64-decoded. In current dokku, the only Secret objects in the main app chart come from KEDA autoscaling auth resources configured via [`scheduler-k3s:autoscaling-auth:set`](#workload-autoscaling-authentication); for apps without autoscaling auth, these flags are no-ops.

> [!IMPORTANT]
> The preview only covers the main app helm release (the release named after the app). Dokku installs three auxiliary helm releases per app for config-var storage, image-pull credentials, and TLS certificates, and changes to those releases do **not** appear in this preview. In particular, a `dokku config:set` change followed by `scheduler-k3s:preview` will show no diff because the new config-var value lives in the separate `config-<app>` helm release.

> [!NOTE]
> Helm renders the proposed manifest using a client-side dry-run. The `lookup` template function returns empty values in this mode, so charts overridden via `scheduler-k3s:charts:set` that depend on `lookup` may render differently here than at actual deploy time. Dokku's bundled chart templates do not use `lookup`.

### Exposing services on the network

Dokku will automatically expose the `web` process as a Kubernetes Service, with all others being treated as background processes. In some cases, it may be useful to have other processes exposed as Kubernetes Service objects so as to segregate internal http endpoints from public http endpoints. This can be done by modifying the `app.json` Formation entry for your process type.

```json
{
  "formation": {  
    "internal-web": {
      "service": {
        "exposed": true
      }
    }
  }
}
```

In the above example, the `internal-web` process is exposed as a service. The `PORT` variable for the process will be set to `5000`, and a kubernetes `Service` object will be created pointing at your processes.

> [!NOTE]
> It is not possible to modify the port mapping, nor is it possible to assign domains or SSL to a non-web process.

### Wildcard domains

Both the `nginx` and `traefik` ingress classes route wildcard domains. Add the wildcard as a domain on the app:

```shell
dokku domains:add node-js-app '*.node-js-app.com'
```

A wildcard matches exactly one label, matching DNS itself, so `*.node-js-app.com` covers `api.node-js-app.com` but not `node-js-app.com` or `api.staging.node-js-app.com`. Add the apex as a separate domain if it should also be served.

An exact domain always takes precedence over a wildcard, including across apps. If one app serves `*.node-js-app.com` and another serves `api.node-js-app.com`, requests for `api.node-js-app.com` are routed to the second app.

Only the leading label may be wildcarded. A domain such as `api.*.node-js-app.com` is treated as a literal hostname and will not match anything.

### SSL Certificates

#### Enabling letsencrypt integration

By default, letsencrypt is disabled and https port mappings are ignored. To enable, set the `letsencrypt-email-prod` or `letsencrypt-email-stag` property with the `--global` flag:

```shell
# set the value for prod
dokku scheduler-k3s:set --global letsencrypt-email-prod automated@dokku.sh

# set the value for stag
dokku scheduler-k3s:set --global letsencrypt-email-stag automated@dokku.sh
```

After enabling and rebuilding, all apps with an `http:80` port mapping will have a corresponding `https:443` added and ssl will be automatically enabled. All http requests will then be redirected to https.

#### Customizing the letsencrypt email per app

The `letsencrypt-email-prod` and `letsencrypt-email-stag` properties can also be set per app, overriding the global value for that app. This is useful when different apps should register their certificates under different contact emails.

```shell
dokku scheduler-k3s:set node-js-app letsencrypt-email-prod team@node-js-app.com
```

The value resolves in two steps: the app's `letsencrypt-server` selects which server (`prod` or `staging`) is used, and the matching `letsencrypt-email-<server>` property is then resolved as the app-level value, falling back to the global value. Because the two emails are per-server, an app-level `letsencrypt-email-stag` only takes effect once the app's `letsencrypt-server` is set to `staging`.

When an app sets its own email for the selected server, Dokku renders a namespaced cert-manager `Issuer` into the app's own release using that email instead of pointing the app at the shared global `ClusterIssuer`. Apps without an app-level email continue to use the shared `ClusterIssuer` with the global email.

The default value may be set by passing an empty value for the option, which falls the app back to the global value:

```shell
dokku scheduler-k3s:set node-js-app letsencrypt-email-prod
```

The computed value in effect for an app can be inspected via the report command:

```shell
dokku scheduler-k3s:report node-js-app --scheduler-k3s-computed-letsencrypt-email-prod
```

#### Customizing the letsencrypt server

The letsencrypt integration is set to the production letsencrypt server by default. This can be changed on an app-level by setting the `letsencrypt-server` property with the `scheduler-k3s:set` command

```shell
dokku scheduler-k3s:set node-js-app letsencrypt-server staging
```

The default value may be set by passing an empty value for the option:

```shell
dokku scheduler-k3s:set node-js-app letsencrypt-server
```

The `letsencrypt-server` property can also be set globally. The global default is `production`.

```shell
dokku scheduler-k3s:set --global letsencrypt-server staging
```

The default value may be set by passing an empty value for the option.

```shell
dokku scheduler-k3s:set --global letsencrypt-server staging
```

Automatic certificate issuance can be completely disabled for a given app by setting the `letsencrypt-server` to `false`. This is the single off switch for the app, and also disables a [manually managed issuer](#using-a-manually-managed-cert-manager-issuer).

```shell
dokku scheduler-k3s:set node-js-app letsencrypt-server false
```

The server can also be disabled globally, and then conditionally enabled on a per-app basis:

```shell
dokku scheduler-k3s:set --global letsencrypt-server false
```

Values are validated when the property is set, so a typo fails immediately rather than breaking the next deploy. The valid values are `prod`, `production`, `stag`, `staging`, and `false`.

#### Using a manually managed cert-manager issuer

Dokku's built-in letsencrypt integration uses an `http01` solver, which cannot issue wildcard certificates and cannot satisfy providers that require `dns01`. For those cases, create a cert-manager `Issuer` or `ClusterIssuer` yourself and point an app at it with the `cert-issuer-name` property.

```shell
dokku scheduler-k3s:set node-js-app cert-issuer-name acme-dns
```

Dokku does not create, modify, or delete the issuer - it only references it from the app's generated `Certificate`. Any cert-manager issuer works, including non-ACME ones such as `selfSigned`, `ca`, or `vault`.

The issuer kind defaults to `ClusterIssuer`. To reference a namespaced `Issuer`, set the `cert-issuer-kind` property:

```shell
dokku scheduler-k3s:set node-js-app cert-issuer-kind Issuer
```

> [!WARNING]
> A namespaced `Issuer` must exist in the same namespace as the app, as configured by the `namespace` property. cert-manager cannot reference an `Issuer` across namespaces.

Unlike the letsencrypt integration, no email property is required - setting `cert-issuer-name` is itself what enables https for the app. Certificates are requested for every domain attached to the app.

Both properties can also be set globally, which enables https for every app that has domains, no imported certificate, and no `letsencrypt-server false`:

```shell
dokku scheduler-k3s:set --global cert-issuer-name acme-dns
```

Certificate sources are resolved in the following order:

1. A certificate imported via the `certs` plugin.
2. `letsencrypt-server` set to `false`, which disables issuance entirely.
3. `cert-issuer-name`, resolved app-first and then globally.
4. The built-in letsencrypt integration.

Because an empty app-level property falls back to the global value, an app cannot return to the built-in letsencrypt integration by unsetting `cert-issuer-name` while a global value is configured. Set the app's `cert-issuer-name` to the reserved value `false` instead:

```shell
dokku scheduler-k3s:set node-js-app cert-issuer-name false
```

As a consequence, an issuer literally named `false` cannot be referenced.

The default value may be set by passing an empty value for the option, which falls the app back to the global value:

```shell
dokku scheduler-k3s:set node-js-app cert-issuer-name
```

If the named issuer does not exist in the cluster, Dokku emits a warning before the build starts. The warning never blocks a deploy, since the issuer may be managed independently and applied later. When a certificate fails to issue, inspect it directly:

```shell
kubectl describe certificate node-js-app-web -n default
```

##### Wildcard certificates

A `dns01` issuer can issue wildcard certificates. Add the wildcard as a domain on the app, and it will be included in the generated `Certificate`:

```shell
dokku domains:add node-js-app '*.node-js-app.com'
```

See [wildcard domains](#wildcard-domains) for how a wildcard is matched against incoming requests.

#### Using imported SSL certificates

SSL certificates imported via the `certs` plugin can be used with the k3s scheduler. When a certificate is imported, it is automatically synced to Kubernetes as a TLS secret and will be used for the app's ingress configuration.

To import a certificate:

```shell
dokku certs:add node-js-app server.crt server.key
```

When a certificate is imported:

- A Kubernetes TLS secret named `tls-<app-name>` is created in the app's namespace
- The ingress configuration is updated to use the imported certificate
- Automatic certificate generation is disabled for the app

Imported certificates take precedence over both Let's Encrypt and a [manually managed issuer](#using-a-manually-managed-cert-manager-issuer). If you have an imported certificate alongside either of those, the imported certificate will be used.

To remove an imported certificate:

```shell
dokku certs:remove node-js-app
```

When a certificate is removed:

- The TLS secret is deleted from Kubernetes
- The app is automatically redeployed to update the ingress configuration
- If Let's Encrypt is configured, automatic certificate generation will resume

When an app is destroyed, any associated TLS secret is automatically cleaned up.

### Customizing Annotations and Labels

> [!NOTE]
> The cron ID is used as the process type if your app deploys any cron tasks

#### Setting Annotations

Dokku injects certain annotations into each created resource by default, but it may be necessary to inject others for tighter integration with third-party tools. The `scheduler-k3s:annotations:set` command can be used to perform this task. The command takes an app name and a required `--resource-type` flag.

```shell
dokku scheduler-k3s:annotations:set node-js-app annotation.key annotation.value --resource-type deployment
```

If not specified, the annotation will be applied to all processes within an app, though it may be further scoped to a specific process type via the `--process-type` flag.

```shell
dokku scheduler-k3s:annotations:set node-js-app annotation.key annotation.value --resource-type deployment --process-type web
```

The following resource types are supported:

- `certificate`
- `cronjob`
- `deployment`
- `ingress`
- `job`
- `pod`
- `secret`
- `service`
- `serviceaccount`
- `traefik_ingressroute`
- `traefik_middleware`

Annotation keys may contain `/` (e.g. Kubernetes-style keys such as `prometheus.io/scrape`) and values may span multiple lines; both are preserved verbatim.

A `ps:restart` is required after setting annotations in order to have them apply to running resources.

#### Removing an annotation

To unset an annotation, pass an empty value:

```shell
dokku scheduler-k3s:annotations:set node-js-app annotation.key --resource-type deployment
dokku scheduler-k3s:annotations:set node-js-app annotation.key --resource-type deployment --process-type web
```

A `ps:restart` is required after removing annotations in order to remove them from running resources.

#### Displaying annotations

Configured annotations can be inspected with the `scheduler-k3s:annotations:report` command. Without arguments, it iterates every app and prints all annotations. Passing an app name (or `--global`) scopes the report:

```shell
dokku scheduler-k3s:annotations:report
dokku scheduler-k3s:annotations:report node-js-app
dokku scheduler-k3s:annotations:report --global
```

`--process-type` and `--resource-type` flags narrow the output to a specific scope, matching the flags accepted by `scheduler-k3s:annotations:set`:

```shell
dokku scheduler-k3s:annotations:report node-js-app --resource-type deployment
dokku scheduler-k3s:annotations:report node-js-app --process-type web --resource-type deployment
```

JSON output emits flat keys of the form `{process_type}.{resource_type}.{annotation_key}`. The literal `--global` process type is rendered as `global` to keep keys free of leading dashes:

```shell
dokku scheduler-k3s:annotations:report node-js-app --format json
```

A single value can also be read directly with a flag of the form `--scheduler-k3s-annotations.{process_type}.{resource_type}.{annotation_key}`:

```shell
dokku scheduler-k3s:annotations:report node-js-app --scheduler-k3s-annotations.global.deployment.annotation.key
```

#### Setting Labels

Dokku injects certain labels into each created resource by default, but it may be necessary to inject others for tighter integration with third-party tools. The `scheduler-k3s:labels:set` command can be used to perform this task. The command takes an app name and a required `--resource-type` flag.

```shell
dokku scheduler-k3s:labels:set node-js-app label.key label.value --resource-type deployment
```

If not specified, the label will be applied to all processes within an app, though it may be further scoped to a specific process type via the `--process-type` flag.

```shell
dokku scheduler-k3s:labels:set node-js-app label.key label.value --resource-type deployment --process-type web
```

The following resource types are supported:

- `certificate`
- `cronjob`
- `deployment`
- `ingress`
- `job`
- `pod`
- `secret`
- `service`
- `serviceaccount`
- `traefik_ingressroute`
- `traefik_middleware`

Label keys may contain `/` (e.g. Kubernetes-style keys such as `app.kubernetes.io/part-of`) and values may span multiple lines; both are preserved verbatim.

A `ps:restart` is required after setting labels in order to have them apply to running resources.

#### Removing a label

To unset an label, pass an empty value:

```shell
dokku scheduler-k3s:annotations:set node-js-app label.key --resource-type deployment
dokku scheduler-k3s:labels:set node-js-app label.key --resource-type deployment --process-type web
```

A `ps:restart` is required after removing labels in order to remove them from running resources.

#### Displaying labels

Configured labels can be inspected with the `scheduler-k3s:labels:report` command. The surface mirrors `scheduler-k3s:annotations:report`:

```shell
dokku scheduler-k3s:labels:report
dokku scheduler-k3s:labels:report node-js-app
dokku scheduler-k3s:labels:report --global
dokku scheduler-k3s:labels:report node-js-app --resource-type deployment
dokku scheduler-k3s:labels:report node-js-app --process-type web --resource-type deployment
dokku scheduler-k3s:labels:report node-js-app --format json
dokku scheduler-k3s:labels:report node-js-app --scheduler-k3s-labels.global.deployment.label.key
```

### Autoscaling

#### Workload Autoscaling

> [!IMPORTANT]
> New as of 0.33.8
> Users with older installations will need to manually install Keda.

Autoscaling in k3s is managed by [Keda](https://keda.sh/), which integrates with a variety of external metric providers to allow for autoscaling application workloads.

To enable autoscaling, use the `app.json` `formation.$PROCESS_TYPE.autoscaling` key to manage rules. In addition to the existing configuration used for process management, each process type in the `formation.$PROCESS_TYPE.autoscaling` key can have the following keys:

- `min_quantity`: The minimum number of instances the application can run. If not specified, the `quantity` specified for the app is used.
- `max_quantity`: The maximum number of instances the application can run. If not specified, the higher value of `quantity` and the `min_quantity` is used.
- `polling_interval_seconds`: (default: 30) The interval to wait for polling each of the configured triggers
- `cooldown_seconds`: (default: 300) The number of seconds to wait in between each scaling event
- `triggers`: A list of autoscaling triggers.

Autoscaling triggers are passed as is to Keda, and should match the configuration keda uses for a given [scaler](https://keda.sh/docs/2.19/scalers/). Below is an example for [datadog](https://keda.sh/docs/2.19/scalers/datadog/#example-2---driving-scale-directly):

```json
{
    "formation": {
        "web": {
            "autoscaling": {
                "min_quantity": 1,
                "max_quantity": 10,
                "triggers": [
                    {
                        "name": "name-for-trigger",
                        "type": "datadog",
                        "metadata": {
                            "query": "per_second(sum:http.requests{service:myservice1}).rollup(max, 300))/180,per_second(sum:http.backlog{service:myservice1}).rollup(max, 300)/30",
                            "queryValue": "1",
                            "queryAggregator": "max"
                        }
                    }
                ]
            }
        }
    }
}
```

Each value in the `metadata` stanza can use the following interpolated strings:

- `DOKKU_DEPLOYMENT_NAME`: The name of the deployment being scaled
- `DOKKU_PROCESS_TYPE`: The name of the process being scaled
- `DOKKU_APP_NAME`: The name of the app being scaled

Dokku configures a Keda `fallback` (with a failure threshold of `3` and the configured replica count) on the generated `ScaledObject` so that scaling falls back to the configured replica count when a scaler's metric source becomes unavailable. Keda only allows `fallback` when at least one trigger is something other than `cpu` or `memory`, so Dokku omits the `fallback` block when every configured trigger is a `cpu` or `memory` scaler.

##### HTTP Autoscaling

In addition to the built-in scalers that Keda provides, Dokku also supports Keda's HTTP Add On. This requires that the addon be properly installed and configured. For existing k3s clusters, this can be performed by the `scheduler-k3s:ensure-charts` command:

```shell
dokku scheduler-k3s:ensure-charts
```

> [!NOTE]
> Users who wish to use this functionality on a cluster not managed by Dokku will need to manually install the `keda-http-add-on` into the `keda` namespace. Please consult the `keda-http-add-on` [install documentation](https://kedacore.github.io/http-add-on/install.html) for further details.

> [!WARNING]
> If the `keda-http-add-on` chart is not installed, then this trigger will be ignored.

Once the chart is configured, an `http` trigger can be specified like so:

```json
{
    "formation": {
        "web": {
            "autoscaling": {
                "min_quantity": 1,
                "max_quantity": 10,
                "triggers": [
                    {
                        "type": "http",
                        "metadata": {
                            "scaledown_period_seconds": "150",
                            "request_rate_target_value": "50"
                        }
                    }
                ]
            }
        }
    }
}
```

The following metadata properties are supported with the http autoscaler:

- `scale_by`: (default: `request_rate`) whether to scale by `concurrency` or `request_rate`.
- `scaledown_period_seconds`: (default: `300`) period to wait after the last reported active before scaling the resource back to 0.
- `request_rate_granularity_seconds`: (default: `1`) granualarity of the aggregated requests for the request rate calculation.
- `request_rate_target_value`: (default: `100`) target value for the request rate.
- `request_rate_window_seconds`: (default: `60`) aggregation window for the request rate calculation.
- `concurrency_target_value`: (default: `100`) target value for the request concurrency.

Note that due to Keda limitations, scaling is done by _either_ `concurrency` or `request_rate`.

#### Workload Autoscaling Authentication

Most Keda triggers require some form of authentication to query for data. In the Kubernetes API, they are represented by `TriggerAuthentication` and `ClusterTriggerAuthentication` resources. Dokku can manage these via the `scheduler-k3s:autoscaling-auth` commands, and includes generated resources with each helm release generated by a deploy.

If no app-specific authentication is provided for a given trigger type, Dokku will fallback to any globally defined `ClusterTriggerAuthentication` resources. Autoscaling triggers within an app all share the same `TriggerAuthentication` resources, while `ClusterTriggerAuthentication` resources can be shared across all apps deployed by Dokku within a given cluster.

##### Creating Authentication Resources

Users can specify custom authentication resources directly via the Kubernetes api _or_ use the `scheduler-k3s:autoscaling-auth:set` command to create the resources in the Kubernetes cluster.

```shell
dokku scheduler-k3s:autoscaling-auth:set $APP $TRIGGER --metadata apiKey=some-api-key --metadata appKey=some-app-key
```

For example, the following will configure the authentication for all datadog triggers on the specified app:

```shell
dokku scheduler-k3s:autoscaling-auth:set node-js-app datadog --metadata apiKey=1234567890 --metadata appKey=asdfghjkl --metadata datadogSite=us5.datadoghq.com
```

After execution, Dokku will include the following resources for each specified trigger with the helm release generated on subsequent app deploys:

- `Secret`: an Opaque `Secret` resource storing the authentication credentials
- `TriggerAuthentication`: A `TriggerAuthentication` resource that references the secret for use by triggers

If the `--global` flag is specified instead of an app name, a custom helm chart is created on the fly with the above resources.

##### Removing Authentication Resources

To remove a configured authenticatin resource, run the `scheduler-k3s:autoscaling-auth:set` command with no metadata specified. Subsequent deploys will not include these resources.

```shell
dokku scheduler-k3s:autoscaling-auth:set $APP $TRIGGER_TYPE
```

##### Displaying an Authentication Resource report

To see a list of authentication resources managed by Dokku, run the `scheduler-k3s:autoscaling-auth:report` command. Without arguments, the report iterates every app; passing an app name (or `--global`) scopes the report:

```shell
dokku scheduler-k3s:autoscaling-auth:report
dokku scheduler-k3s:autoscaling-auth:report node-js-app
dokku scheduler-k3s:autoscaling-auth:report --global
```

```
====> node-js-app autoscaling-auth information
      Datadog:                       configured
```

By default, the report will not display configured metadata - making it safe to include in Dokku report output. To include metadata and their values, add the `--include-metadata` flag:

```shell
dokku scheduler-k3s:autoscaling-auth:report node-js-app --include-metadata
```

```
====> node-js-app autoscaling-auth information
      Datadog:                       configured
      Datadog apiKey:                1234567890
      Datadog appKey:                asdfghjkl
      Datadog datadogSite:           us5.datadoghq.com
```

JSON output emits flat keys of the form `{trigger}.{metadata_key}` and includes the actual metadata values so export tools can reconstruct the configured state:

```shell
dokku scheduler-k3s:autoscaling-auth:report node-js-app --format json
```

A single configured metadata key can also be queried with a flag of the form `--scheduler-k3s-autoscaling-auth.{trigger}.{metadata_key}`. The returned value is masked in the same way as stdout output; use `--format json` to read the actual value:

```shell
dokku scheduler-k3s:autoscaling-auth:report node-js-app --scheduler-k3s-autoscaling-auth.datadog.apiKey
```

### Setting kernel sysctls

Kernel sysctls fall into two categories, and which one a sysctl belongs to determines how it must be set.

The kernel maintains a per-namespace copy of `net.*` (network namespace) as well as `kernel.shm*`, `kernel.msg*`, `kernel.sem`, and `fs.mqueue.*` (IPC namespace). These can be set on a single app's pods. Every other sysctl - including all of `vm.*`, and therefore `vm.max_map_count` - holds a single value shared by the entire machine, so it cannot be scoped to a pod and must be applied to the node itself.

#### Namespaced sysctls

Namespaced sysctls are set with the `docker-options` plugin, and are translated into the pod's `securityContext.sysctls`. A `ps:restart` is required to apply them.

```shell
dokku docker-options:add node-js-app deploy "--sysctl net.ipv4.ip_unprivileged_port_start=1024"
```

Passing a non-namespaced sysctl this way fails the deploy rather than silently dropping the value, since it provably cannot take effect within a pod. Note this differs from the `docker-local` scheduler, where such an option is passed straight through to `docker run`.

Kubernetes further splits namespaced sysctls into a *safe* list that any pod may set, and everything else. A sysctl outside the safe list - `net.core.somaxconn`, for example - is rejected at pod admission unless the node's kubelet was started with a matching `allowed-unsafe-sysctls` value, which can be supplied at cluster initialization or when joining a node.

```shell
dokku scheduler-k3s:initialize --kubelet-args allowed-unsafe-sysctls=net.core.somaxconn
```

Dokku does not enforce the safe list itself, as its membership changes between Kubernetes releases. Only the namespaced/non-namespaced distinction, which is a property of the kernel, is validated.

#### Non-namespaced sysctls

Non-namespaced sysctls are a property of the node, not of any app, and are managed with the `node-sysctls:set` command. Dokku applies them via a privileged DaemonSet, so they reach every node without being told which nodes exist, cover nodes joined later, and are reapplied after a node reboots.

```shell
dokku scheduler-k3s:node-sysctls:set --global vm.max_map_count 262144
```

Omitting the value clears it.

```shell
dokku scheduler-k3s:node-sysctls:set --global vm.max_map_count
```

Clearing a sysctl stops Dokku managing it, but does not restore whatever the node had before. The last value written stays in place until that node reboots, which is how `sysctl -w` behaves everywhere else.

Sysctls can also be scoped to a [node profile](#node-profiles) with `--profile`, which applies them only to nodes joined with that profile.

```shell
dokku scheduler-k3s:node-sysctls:set --profile edge-workers vm.max_map_count 524288
```

A profile scope inherits everything set globally and overrides it on conflict, so each node is covered by exactly one DaemonSet and no two ever write the same value. Note that the server node created by `scheduler-k3s:initialize` never carries a profile label, so only globally-scoped sysctls reach it.

Use `node-sysctls:report` to see the resolved set for every scope.

```shell
dokku scheduler-k3s:node-sysctls:report
```

```shell
dokku scheduler-k3s:node-sysctls:report --format json
```

The DaemonSet pulls `busybox` and `registry.k8s.io/pause` by default. On an air-gapped cluster or one behind a registry mirror, point them elsewhere:

```shell
dokku scheduler-k3s:set --global node-sysctls-image registry.internal/busybox:1.36
```

```shell
dokku scheduler-k3s:set --global node-sysctls-pause-image registry.internal/pause:3.9
```

### Integrating Kustomize

Dokku supports integration with [Kustomize](https://kustomize.io/) to further customize the generated helm charts for app deployments. For example, a `config/kustomize/kustomization.yaml` file with the following contents will override the scale for each process deployed to `3`:

```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- rendered.yaml
patches:
- patch: |-
    - op: replace
      path: /spec/replicas
      value: 3
  target:
      group: apps
      version: v1
      kind: Deployment
```

When using Kustomize with Dokku, all Kustomize-files _must_ be placed in the `config/kustomize` folder, with a `kustomization.yaml` file being the entrypoint to Kustomize. Dokku will render the helm chart to a `rendered.yaml` file, and then execute Kustomize with the `config/kustomize` folder as the context.

See the [Kustomize](https://kustomize.io/) website for more details on how to use Kustomize.

### Using kubectl remotely

> [!WARNING]
> Certain ports must be open for interacting with the remote kubernets api. Refer to the [K3s networking documentation](https://docs.k3s.io/installation/requirements?os=debian#networking) for the required open ports between servers prior to running the command.

By default, Dokku assumes that all it controls all actions on the cluster, and thus does not expose the `kubectl` binary for administrators. To interact with kubectl, you will need to retrieve the `kubeconfig` for the cluster and configure your client to use that configuration.

```shell
dokku scheduler-k3s:show-kubeconfig
```

### Interacting with an external Kubernetes cluster

While the k3s scheduler plugin is designed to work with a Dokku-managed k3s cluster, Dokku can be configured to interact with any Kubernetes cluster by setting the global `kubeconfig-path` to a path to a custom kubeconfig on the Dokku server. This property is only available at a global level.

```shell
dokku scheduler-k3s:set --global kubeconfig-path /path/to/custom/kubeconfig
```

To set the default value, omit the value from the `scheduler-k3s:set` call:

```shell
dokku scheduler-k3s:set --global kubeconfig-path
```

The default value for the `kubeconfig-path` is the k3s kubeconfig located at `/etc/rancher/k3s/k3s.yaml`.

### Customizing the Kubernetes context

When interacting with a custom Kubeconfig, the `kube-context` property can be set to specify a specific context within the kubeconfig to use. This property is available only at the global leve.

```shell
dokku scheduler-k3s:set --global kube-context lollipop
```

To set the default value, omit the value from the `scheduler-k3s:set` call:

```shell
dokku scheduler-k3s:set --global kube-context
```

The default value for the `kube-context` is an empty string, and will result in Dokku using the current context within the kubeconfig.

### Customizing Helm Chart Properties

Dokku includes a number of helm charts by default with settings that are optimized for Dokku. That said, it may be useful to further customize the charts for a given environment. Chart overrides are managed via the `scheduler-k3s:charts:set` command, which takes a `<chart-name>.<property>` argument and a value. Chart overrides are always global because helm charts are not managed on a per-app basis.

```shell
dokku scheduler-k3s:charts:set cert-manager.version 1.13.3
```

> [!NOTE]
> Properties follow dot-notation, and are expanded according to Helm's internal logic. See the [Helm documentation](https://helm.sh/docs/helm/helm_install/#helm-install) for `helm install` for further details.

Property names may contain `/` (e.g. for Kubernetes-style annotation keys such as `service.annotations.prometheus.io/scrape`) and values may span multiple lines; both are preserved verbatim.

To unset a chart property, omit the value from the `scheduler-k3s:charts:set` call:

```shell
dokku scheduler-k3s:charts:set cert-manager.version
```

Configured chart overrides can be inspected with the `scheduler-k3s:charts:report` command. Without an argument, it lists every chart known to Dokku along with any configured overrides; passing a chart name scopes the report to that chart:

```shell
dokku scheduler-k3s:charts:report
dokku scheduler-k3s:charts:report cert-manager
dokku scheduler-k3s:charts:report --format json
```

> [!NOTE]
> The legacy form `dokku scheduler-k3s:set --global chart.<chart>.<property> <value>` continues to work but is deprecated and will be removed in a future major release. Migrate to `scheduler-k3s:charts:set` instead.

A `scheduler-k3s:ensure-charts` command with the `--force` flag is required after changing any chart properties in order to have them apply. This will install all charts, not just the ones that have changed.

```shell
dokku scheduler-k3s:ensure-charts --force
```

Alternatively, a comma separated list of chart names can be specified to only force install the specified charts:

```shell
dokku scheduler-k3s:ensure-charts --charts cert-manager
```

### Persistent storage

K3s apps mount persistent volumes through the Dokku storage plugin. A `storage:create --scheduler k3s` call provisions a PersistentVolumeClaim through a small per-entry helm release; `storage:mount <app> <name> --container-dir <path>` then attaches it to an app. Because storage entries are global, the same PVC can be mounted into multiple apps (for `ReadWriteMany` workloads) or into the same app at multiple subpaths.

```shell
# dynamically-provisioned PVC backed by Longhorn
dokku storage:create demo-data \
  --scheduler k3s \
  --size 2Gi \
  --access-mode ReadWriteOnce \
  --storage-class-name longhorn

dokku storage:mount demo demo-data --container-dir /data
dokku storage:wait demo-data
git push dokku master
```

For a hostPath-backed PV (no StorageClass), pass `<path>` as the second positional argument and omit `--storage-class-name`. The plugin renders both the PV and the PVC into the entry's helm release. The `--reclaim-policy` flag (`Retain` or `Delete`) controls whether the underlying PV survives `storage:destroy`. Annotations and labels set via `storage:annotations:set` and `storage:labels:set` propagate to both the PVC and the PV so backup tools (Velero, Longhorn snapshots) can find them.

The legacy `storage:mount <app> <host>:<container>` colon form is rejected on k3s apps; create a named entry instead. See [Persistent Storage](/docs/advanced-usage/persistent-storage.md) for the full command reference.

### Chart upgrade callbacks

Some chart version transitions require side-effects that a plain `helm upgrade` cannot perform. For example, the upgrade of `keda-add-ons-http` to `0.12.2` introduces breaking changes to deployment selectors, which are immutable in Kubernetes; the chart-managed deployments must be deleted before the upgrade can proceed.

Dokku handles these cases internally by registering version-targeted pre-upgrade and post-upgrade callbacks against affected charts. When `scheduler-k3s:ensure-charts` runs, the installed chart version is compared against the registered callbacks: any whose target version is greater than the currently installed version and at most the configured chart version are executed in ascending semver order, each bracketed around an upgrade to that intermediate version. A final upgrade to the configured chart version then completes the run.

These callbacks are not user-configurable. Failures during a callback or its surrounding upgrade abort the run, and a subsequent `scheduler-k3s:ensure-charts` resumes from the current state.

## Scheduler Interface

The following sections describe implemented and unimplemented scheduler functionality for the `k3s` scheduler.

### Implemented Commands and Triggers

This plugin implements various functionality through `plugn` triggers to integrate with Docker for running apps on a single server. The following functionality is supported by the `scheduler-k3s` plugin.

- `apps:clone`
- `apps:destroy`
- `apps:rename`
- `docker-options`:
    - The following docker options are translated into their kubernetes equivalents:
        - `--cap-add`
        - `--cap-drop`
        - `--privileged`
        - `--sysctl` (namespaced sysctls only, see [Setting kernel sysctls](#setting-kernel-sysctls))
- `cron`
- `enter`
- `deploy`
- healthchecks
    - Due to Kubernetes limitations, only a single healthcheck is supported for each of the `liveness`, `readiness`, and `startup` healthchecks
    - Due to Kubernetes limitations, content checks are not supported
    - Ports specified in the `app.json` are ignored in favor of the container port on the port mapping detected
- `logs`
- `nginx`
    - Properties set by the `nginx` plugin will be respected, either by turning them into annotations or creating a custom server/location snippet that the `ingress-nginx` project can use. A `ps:restart` after changing any nginx properties is required in order to have them apply.
    - The `nginx:access-logs` and `nginx:error-logs` commands will fetch logs from one running `ingress-nginx` pod.
    - The `nginx:show-config` command will retrieve any `server` blocks associated with a domain attached to the app from one running `ingress-nginx` pod.
- `ps:restart`
    - Supports targeting a single process type, see [Restarting apps](#restarting-apps)
- `ps:stop`
- `run`
    - The `scheduler-post-run` trigger is not always triggered
- `run:detached`
- `run:list`

### Unimplemented command functionality

- `run:logs`
- `ps:inspect`

The following Dokku functionality is not implemented at this time.

- `vector` log integration
- persistent storage

### Logging support

App logs for the `logs` command are fetched by Dokku from running containers via the Kubernetes api. While the `k3s` scheduler does not integrate with the `logs:vector-*` subcommands, it does respect the global `vector-sink` logs property. When that property is set, the `scheduler-k3s:ensure-charts` command can be utilized to reconfigure vector to ship kubernetes logs to the provided sink.

To setup log shipping, configure the global `vector-sink` property for the `logs` plugin.. Note that this can be run before _or_ after the `scheduler-k3s:ensure-charts` command - if run before, the `scheduler-k3s:ensure-charts` subcommand will pick up the value.

```shell
dokku logs:set --global vector-sink "console://?encoding[codec]=json"
```

Next, run the `scheduler-k3s:ensure-charts` command with the `vector` chart to force the `k3s` scheduler to reconfigure vector with the specified sink:

```shell
dokku scheduler-k3s:ensure-charts --charts vector
```

Please see the [vector logs documentation](/docs/deployment/logs.md#configuring-a-log-sink) for more information on specifying vector sinks.

#### Shipping cron task logs

The global `vector-cron-sink` property is also respected. When set, logs from cron task pods are routed to that sink instead of the sink configured for everything else, matching the behavior described in the [cron task log sink documentation](/docs/deployment/logs.md#configuring-a-cron-task-log-sink).

```shell
dokku logs:set --global vector-cron-sink "console://?encoding[codec]=text"
dokku scheduler-k3s:ensure-charts --charts vector
```

As with `vector-sink`, only the global property is respected - a per-app `vector-cron-sink` has no effect on the `k3s` scheduler.

Cron events carry the same `dokku_app` and `dokku_cron_id` fields as they do on the `docker-local` scheduler, so sink configuration referencing them is portable between the two.

Two differences are worth noting:

- Templated values must be base64 encoded. Sink values containing `{{ }}` are interpreted by Helm at chart install time rather than by Vector, so the `base64enc:` form documented under [log sink DSN format](/docs/deployment/logs.md#log-sink-dsn-format) is required.
- The `file` sink is not useful here. Vector runs as a DaemonSet agent, so a file path resolves to whichever node the agent is running on rather than to durable shared storage. Use a network sink and reference `dokku_cron_id` as a field instead of as a path component.

### Supported Resource Management Properties

The `k3s` scheduler supports a minimal list of resource _limits_ and _reservations_:

- cpu: is specified in number of CPUs a process can access.
- memory: should be specified with a suffix of `b` (bytes), `Ki` (kilobytes), `Mi` (megabytes), `Gi` (gigabytes). Default unit is `Mi` (megabytes).

If unspecified for any task, the default reservation will be `.1` CPU and `128Mi` RAM, with no limit set for either CPU or RAM. This is to avoid issues with overscheduling pods on a cluster. To avoid issues, set more specific values for at least resource reservations. If unbounded utilization is desired, set CPU and Memory to `0m` and `0Mi`, respectively.

> [!NOTE]
> Cron tasks retrieve resource limits based on the computed cron task ID.

## Properties

### Settable properties

> [!NOTE]
> The `Report flags` column lists the CLI argument names accepted by `scheduler-k3s:report`. The JSON keys emitted by `scheduler-k3s:report --format json` are the same names with the leading `--scheduler-k3s-` stripped (e.g. `deploy-timeout`, `global-deploy-timeout`, `computed-deploy-timeout`). Legacy keys with the `scheduler-k3s-` prefix are also emitted during the 0.38.x deprecation window and will be removed in a future major release.

| Property | Scope | Default | Report flags | Description |
|---|---|---|---|---|
| `cert-issuer-kind` | app + global | `ClusterIssuer` | `--scheduler-k3s-cert-issuer-kind`, `--scheduler-k3s-global-cert-issuer-kind`, `--scheduler-k3s-computed-cert-issuer-kind` | Kind of the manually managed cert-manager issuer referenced by `cert-issuer-name`, either `Issuer` or `ClusterIssuer` |
| `cert-issuer-name` | app + global | none | `--scheduler-k3s-cert-issuer-name`, `--scheduler-k3s-global-cert-issuer-name`, `--scheduler-k3s-computed-cert-issuer-name` | Name of a manually managed cert-manager issuer to request certificates from, taking precedence over the letsencrypt integration. Set to `false` to opt an app out of a global value |
| `deploy-timeout` | app + global | `300s` | `--scheduler-k3s-deploy-timeout`, `--scheduler-k3s-global-deploy-timeout`, `--scheduler-k3s-computed-deploy-timeout` | Timeout for a single helm install/upgrade cycle |
| `image-pull-secrets` | app + global | none | `--scheduler-k3s-image-pull-secrets`, `--scheduler-k3s-global-image-pull-secrets`, `--scheduler-k3s-computed-image-pull-secrets` | Comma-separated list of Kubernetes secret names used to pull private images |
| `ingress-class` | global only | `nginx` | `--scheduler-k3s-global-ingress-class`, `--scheduler-k3s-computed-ingress-class` | IngressClass name used for app ingresses (e.g. `nginx`, `traefik`) |
| `kube-context` | global only | none | `--scheduler-k3s-global-kube-context`, `--scheduler-k3s-computed-kube-context` | Kube context name used by helm and kubectl invocations |
| `kubeconfig-path` | global only | `/etc/rancher/k3s/k3s.yaml` | `--scheduler-k3s-global-kubeconfig-path`, `--scheduler-k3s-computed-kubeconfig-path` | Filesystem path to the kubeconfig used to talk to the cluster |
| `kustomize-root-path` | app + global | `config/kustomize` | `--scheduler-k3s-kustomize-root-path`, `--scheduler-k3s-global-kustomize-root-path`, `--scheduler-k3s-computed-kustomize-root-path` | Path within the app to a kustomize root applied after the helm install |
| `letsencrypt-email-prod` | app + global | none | `--scheduler-k3s-letsencrypt-email-prod`, `--scheduler-k3s-global-letsencrypt-email-prod`, `--scheduler-k3s-computed-letsencrypt-email-prod` | Contact email for production certificates. App-level values render a per-app namespaced Issuer; otherwise the shared production ClusterIssuer is used |
| `letsencrypt-email-stag` | app + global | none | `--scheduler-k3s-letsencrypt-email-stag`, `--scheduler-k3s-global-letsencrypt-email-stag`, `--scheduler-k3s-computed-letsencrypt-email-stag` | Contact email for staging certificates. App-level values render a per-app namespaced Issuer; otherwise the shared staging ClusterIssuer is used |
| `letsencrypt-server` | app + global | `prod` | `--scheduler-k3s-letsencrypt-server`, `--scheduler-k3s-global-letsencrypt-server`, `--scheduler-k3s-computed-letsencrypt-server` | ACME directory (`prod` or `staging`) used for app certificates, or `false` to disable all automatic certificate issuance |
| `namespace` | app + global | `default` | `--scheduler-k3s-namespace`, `--scheduler-k3s-global-namespace`, `--scheduler-k3s-computed-namespace` | Kubernetes namespace into which the app's resources are installed |
| `network-interface` | global only | `eth0` | `--scheduler-k3s-global-network-interface`, `--scheduler-k3s-computed-network-interface` | Host network interface used by k3s |
| `node-sysctls-image` | global only | `busybox:1.36` | `--scheduler-k3s-global-node-sysctls-image` | Image used to apply node-level sysctls, override for air-gapped clusters |
| `node-sysctls-pause-image` | global only | `registry.k8s.io/pause:3.9` | `--scheduler-k3s-global-node-sysctls-pause-image` | Image keeping the node sysctls daemonset pods running |
| `rollback-on-failure` | app + global | `false` | `--scheduler-k3s-rollback-on-failure`, `--scheduler-k3s-global-rollback-on-failure`, `--scheduler-k3s-computed-rollback-on-failure` | When `true`, helm rolls back the release if a deploy fails |
| `shm-size` | app + global | none | `--scheduler-k3s-shm-size`, `--scheduler-k3s-global-shm-size`, `--scheduler-k3s-computed-shm-size` | `/dev/shm` size override applied to app containers |
| `token` | global only | none | `--scheduler-k3s-global-token` (masked as `*******` in default stdout output; the raw value is returned when queried via `--format json` or when this flag is requested explicitly) | Cluster join token used by `scheduler-k3s:cluster-add` |
| `chart.<chart-name>.<property>` | global only | none | `--scheduler-k3s-global-chart.<chart-name>.<property>` (dynamic per chart/property) | Override a value injected into the helm chart named `<chart-name>` (one row per chart/property pair). Manage these via the dedicated `scheduler-k3s:charts:set` / `scheduler-k3s:charts:report` commands; the `scheduler-k3s:set`/`scheduler-k3s:report` form is deprecated. |