The ingress-nginx ingress implementation is the standard ingress in the Kubernetes community, and it doesn't make sense for us to stray from that just to utilize the k3s default.
In the future, we might drop k3s, but this works well for now.
Rather than require a heavy chown operation across various paths, just chown the files already in the built image during the release process. This ensures we can skip not-only the chown process during the container start that herokuish injects, but also the one that Dokku runs which modifies mounted container paths as well during the pre-deploy.
Note that users will need to ensure any mounted volumes don't have permissions reset by other processes or containers won't be able to access them.
This will provide the possibility for users to talk to existing kubernetes clusters, thereby removing one of the biggest reasons for having the old scheduler-kubernetes plugin around.
While this will now more or less require a local docker login - which can be overridden by setting the image-pull-secrets property - this ensures every deploy has a valid context.
One annoying thing is that the secret is per-app vs global, meaning that if the secret must be rotated, all apps need to be redeployed. A future change can fix that by deploying a global secret, or if folks really desire, this can be done by setting the image-pull-secrets property on the app/globally.
Different options get applied to each depending on what is needed for the role.
This also updates how we initialize the cluster to better support how metrics exposure and cross-server networking.
This utilizes a Kubernetes Job to start a pod in order to take advantage of completed Job cleanup in Kubernetes. It also properly handles interactivity by ensuring the container has a TTY.
Note that there is currently no provision for parsing docker arguments and translating them into their Kubernetes equivalents.
Recent Dokku changes invalidate a bunch of docs around where files need to be placed in order for Dokku to respect them. This doc change clarifies where files are extracted from in cases where source code is available, which should hopefully make users less confused about how the system works.
See https://railsnotes.xyz/blog/deploying-ruby-on-rails-with-dokku-redis-sidekiq-arm-docker-hetzner for the inspiration - I was reading through it and was like 'these docs are definitely incorrect...'.
