feat: translate docker-options --sysctl on the k3s scheduler

The `docker-local` scheduler supports `--sysctl` for free because docker options are passed verbatim to `docker run`, but the k3s scheduler silently dropped it. Namespaced sysctls now render into the pod's `securityContext.sysctls` for deployments, cron jobs, and one-off runs. A sysctl the kernel does not namespace fails the deploy instead of being dropped, since it cannot take effect within a pod regardless of what was requested.

docs/advanced-usage/docker-options.md

@@ -38,6 +38,16 @@ More information on supported Docker options can be found [here](https://docs.do
 
 Container options configured via the `docker-options` plugin are not used to modify the process a container runs. Container options are the `[OPTIONS]` portion of the following, where `[CONTAINER_COMMAND]` and `[ARG]` are the process and the arguments passed to it that are launched in the created container: `docker run [OPTIONS] [CONTAINER_COMMAND] [ARG...]`. Please see the documentation for [customizing the run command](/docs/deployment/builders/dockerfiles.md#customizing-the-run-command) or use a [Procfile](/docs/deployment/builders/dockerfiles.md#procfiles-and-multiple-processes) to modify the command used by a Dockerfile-based container.
 
+#### Scheduler support
+
+Docker options are written in Docker's own vocabulary and are passed verbatim to `docker run` by the `docker-local` scheduler. Other schedulers translate only the subset that has an equivalent in their own runtime, and ignore the rest.
+
+The `k3s` scheduler translates `--cap-add`, `--cap-drop`, `--privileged`, and `--sysctl` into their Kubernetes equivalents. See the [k3s scheduler documentation](/docs/deployment/schedulers/k3s.md) for details, including the restriction that only namespaced sysctls can be set on a pod.
+
+```shell
+dokku docker-options:add node-js-app deploy "--sysctl net.ipv4.ip_unprivileged_port_start=1024"
+```
+
 #### Mounting volumes and host directories
 
 Docker supports volume and host directory mounting via the `-v` or `--volume` flags. In order to simplify usage, Dokku provides a `storage` plugin as an abstraction to interact with persistent storage. In most cases, the Dokku project recommends using the persistent storage plugin over directly manipulating docker options at different phases. See the [persistent storage documentation](/docs/advanced-usage/persistent-storage.md) for more information on how to attach persistent storage to your app.

docs/deployment/schedulers/k3s.md

@@ -740,6 +740,44 @@ A single configured metadata key can also be queried with a flag of the form `--
 dokku scheduler-k3s:autoscaling-auth:report node-js-app --scheduler-k3s-autoscaling-auth.datadog.apiKey
 ```
 
+### Setting kernel sysctls
+
+Kernel sysctls fall into two categories, and which one a sysctl belongs to determines how it must be set.
+
+The kernel maintains a per-namespace copy of `net.*` (network namespace) as well as `kernel.shm*`, `kernel.msg*`, `kernel.sem`, and `fs.mqueue.*` (IPC namespace). These can be set on a single app's pods. Every other sysctl - including all of `vm.*`, and therefore `vm.max_map_count` - holds a single value shared by the entire machine, so it cannot be scoped to a pod and must be applied to the node itself.
+
+#### Namespaced sysctls
+
+Namespaced sysctls are set with the `docker-options` plugin, and are translated into the pod's `securityContext.sysctls`. A `ps:restart` is required to apply them.
+
+```shell
+dokku docker-options:add node-js-app deploy "--sysctl net.ipv4.ip_unprivileged_port_start=1024"
+```
+
+Passing a non-namespaced sysctl this way fails the deploy rather than silently dropping the value, since it provably cannot take effect within a pod. Note this differs from the `docker-local` scheduler, where such an option is passed straight through to `docker run`.
+
+Kubernetes further splits namespaced sysctls into a *safe* list that any pod may set, and everything else. A sysctl outside the safe list - `net.core.somaxconn`, for example - is rejected at pod admission unless the node's kubelet was started with a matching `allowed-unsafe-sysctls` value, which can be supplied at cluster initialization or when joining a node.
+
+```shell
+dokku scheduler-k3s:initialize --kubelet-args allowed-unsafe-sysctls=net.core.somaxconn
+```
+
+Dokku does not enforce the safe list itself, as its membership changes between Kubernetes releases. Only the namespaced/non-namespaced distinction, which is a property of the kernel, is validated.
+
+#### Non-namespaced sysctls
+
+Non-namespaced sysctls are a property of the node, not of any app. Set them directly on each node in the cluster.
+
+```shell
+sudo sysctl -w vm.max_map_count=262144
+```
+
+```shell
+echo "vm.max_map_count = 262144" | sudo tee /etc/sysctl.d/99-max-map-count.conf
+```
+
+The second command is what makes the change survive a reboot; `sysctl -w` alone does not. Repeat both on every node, including any added later via `scheduler-k3s:cluster:add`.
+
 ### Integrating Kustomize
 
 Dokku supports integration with [Kustomize](https://kustomize.io/) to further customize the generated helm charts for app deployments. For example, a `config/kustomize/kustomization.yaml` file with the following contents will override the scale for each process deployed to `3`:
@@ -894,6 +932,7 @@ This plugin implements various functionality through `plugn` triggers to integra
         - `--cap-add`
         - `--cap-drop`
         - `--privileged`
+        - `--sysctl` (namespaced sysctls only, see [Setting kernel sysctls](#setting-kernel-sysctls))
 - `cron`
 - `enter`
 - `deploy`