fix: match docker options by shell word when removing

Options drained out of the pre-0.38.0 `DOCKER_OPTIONS_<PHASE>` files were copied verbatim rather than re-serialized the way `docker-options:add` stores them, so `docker-options:remove` compared the canonical string it builds against a stored value that could never match it and exited successfully without removing anything. Removal now matches stored options by shell word, and stored options are rewritten into the canonical form once on upgrade, which additionally splits an entry that carried several flags on a single line into one entry per flag so a single flag can be removed and so the readers that match on a flag prefix see one value per entry. The leftover `.migrated` sentinel drain is restored to running ahead of the global short-circuit that had made it unreachable, and the plugin's Go tests are added to the test target that had never run them.
2026-08-29 10:08:53 +02:00 · 2026-08-07 13:35:37 -04:00
parent 30a72f8d95
commit 8d4e7d9793
9 changed files with 523 additions and 43 deletions
--- a/docs/advanced-usage/docker-options.md
+++ b/docs/advanced-usage/docker-options.md
@@ -80,7 +80,15 @@ Option values are stored and passed to the container verbatim. Quoting only cont
 dokku docker-options:add node-js-app deploy '--label "traefik.http.routers.web.rule=Host(`node-js-app.example.com`) && PathPrefix(`/api`)"'
 ```

-A misplaced `--process PROC` (i.e. one specified after the app name instead of before it) is honored as a subcommand flag rather than stored as a docker option, so the example above and the equivalent process-scoped form below behave identically:
+> [!WARNING]
+> Options added before 0.38.25 were expanded by the shell when a container was created. Any such option that relied on shell expansion - `$(...)`, backticks, or `$VAR` - is treated as a literal string after upgrading and must be re-added with the value already resolved:
+>
+> ```shell
+> dokku docker-options:remove node-js-app deploy "--group-add \$(getent group docker | cut -d: -f3)"
+> dokku docker-options:add node-js-app deploy "--group-add $(getent group docker | cut -d: -f3)"
+> ```
+
+A misplaced `--process PROC` (i.e. one specified after the app name instead of before it) is honored as a subcommand flag rather than stored as a docker option, so the two invocations below behave identically:

 ```shell
 dokku docker-options:add --process web node-js-app deploy "--ulimit nofile=12" "--shm-size 256m"
@@ -107,6 +115,13 @@ Multiple docker options can also be removed in a single call, mirroring the spli
 dokku docker-options:remove node-js-app deploy "--ulimit nofile=12" "--shm-size 256m"
 ```

+A stored option is matched by shell word rather than by exact string, so the value only has to be equivalent to the stored one, not byte-identical. Quoting an option one way removes an option that was stored quoted another way:
+
+```shell
+dokku docker-options:add node-js-app deploy "--label 'com.example.owner=platform team'"
+dokku docker-options:remove node-js-app deploy '--label "com.example.owner=platform team"'
+```
+
 #### Clear all Docker options for an app

 Docker options can be removed for a specific app using the `docker-options:clear` command.
@@ -278,3 +293,5 @@ The following properties are recorded internally by the docker-options plugin an
 | `migrated-build` | per-app | Per-app marker recording that the app's legacy `DOCKER_OPTIONS_BUILD` file was drained into the `_default_.build` property list. Only set when the legacy file contained non-empty content | `plugins/docker-options/functions.go` writes `"true"` after the per-phase drain |
 | `migrated-deploy` | per-app | Per-app marker recording that the app's legacy `DOCKER_OPTIONS_DEPLOY` file was drained into the `_default_.deploy` property list. Only set when the legacy file contained non-empty content | `plugins/docker-options/functions.go` writes `"true"` after the per-phase drain |
 | `migrated-run` | per-app | Per-app marker recording that the app's legacy `DOCKER_OPTIONS_RUN` file was drained into the `_default_.run` property list. Only set when the legacy file contained non-empty content | `plugins/docker-options/functions.go` writes `"true"` after the per-phase drain |
+| `migrated-traefik-backticks` | global | Global sentinel recording that stored Traefik labels whose backticks carried a stray backslash have been repaired | `plugins/docker-options/functions.go` writes `"true"` once the install-time repair runs |
+| `migrated-canonical-options` | global | Global sentinel recording that stored options have been rewritten into the canonical form, quoting values whose shell metacharacters were left bare by an older legacy-file drain and splitting entries that carried several flags | `plugins/docker-options/functions.go` writes `"true"` once the install-time rewrite runs |
--- a/docs/appendices/0.38.0-migration-guide.md
+++ b/docs/appendices/0.38.0-migration-guide.md
@@ -27,6 +27,7 @@
 - All Go-implemented plugins (`app-json`, `apps`, `builder`, `buildpacks`, `builds`, `cron`, `docker-options`, `logs`, `network`, `ports`, `proxy`, `ps`, `registry`, `resource`, `scheduler`, `scheduler-k3s`, `storage`) now emit JSON keys from `:report --format json` without the `<plugin>-` head segment, matching the shape bash plugins have always emitted. For example, `dokku ps:report myapp --format json` now contains `stop-timeout-seconds`, `global-stop-timeout-seconds`, and `computed-stop-timeout-seconds` keys. The CLI flag names (`--ps-stop-timeout-seconds`, etc.) are unchanged, and `:set` semantics are unchanged. For backwards compatibility during the 0.38.x patch series, the old `<plugin>-<property>` JSON keys are emitted side-by-side with the new keys, so external scripts reading either shape continue to work. The legacy keys will be dropped in a future major release. External JSON consumers should migrate to the new key shape.
 - The `scheduler-k3s` plugin now manages env config and the dokku-generated image pull Secret as their own helm releases with stable names (`config-{app}` and `pull-secret-{app}`) rather than bundling them into the app helm chart with a per-deploy timestamp suffix (`env-{app}.{ts}` / `ims-{app}.{ts}`). This fixes two bugs: a helm rollback of the app chart no longer deletes Secrets that older ReplicaSets still reference, and the Deployment's `imagePullSecrets` list no longer accumulates references to nonexistent Secrets across deploys. The next deploy of an app switches the Deployment's `envFrom` and `imagePullSecrets` references to the stable names and prunes any leaked entries; existing live Deployments do not need to be patched manually. App rename now also uninstalls the old `tls-{app}`, `config-{app}`, and `pull-secret-{app}` releases under the previous app name; the new name's releases are recreated on the next deploy or certs sync.
 - **New in 0.38.25:** Values supplied through docker options, `dokku run`'s `-e`/`--env` flag, and `--ttl-seconds` are no longer evaluated by the shell when assembling a container's arguments; they are now tokenized and passed through verbatim. This closes a command-injection vector where a `$(...)` or backtick expression in one of these values executed on the host as the `dokku` user during build, deploy, or run. As a result, shell metacharacters such as `$(...)`, backticks, `$VAR`, and globs in these values are treated literally instead of being expanded, and `--ttl-seconds` must now be a plain integer. Existing Traefik docker-options labels (those whose label key begins with `traefik.`) whose backticks were stored with a stray backslash are repaired automatically the first time `dokku` runs after the upgrade, so they become valid on the next deploy.
+- **New in 0.38.26:** Docker options drained out of the pre-0.38.0 `DOCKER_OPTIONS_<PHASE>` files were copied verbatim rather than being re-serialized the way `docker-options:add` stores them, so `docker-options:remove` compared the canonical string it builds against a value that could never match it and exited successfully without removing anything. Removal now matches stored options by shell word instead of by exact string, so those entries can be removed with the value as originally written. Stored options are also rewritten into the canonical form the first time `dokku` runs after the upgrade: values whose shell metacharacters were left unquoted are quoted, and an entry that carried several flags on one line becomes one entry per flag, which additionally fixes `ps:report` reading a restart policy off such a line and the `k3s` scheduler translating only the first of several `--cap-add`/`--sysctl` flags. Options added through `docker-options:add` are already canonical and are left untouched.
 - The storage plugin now treats persistent volumes as named, scheduler-aware first-class resources via `storage:create`, `storage:mount`, `storage:set`, and `storage:destroy`. The legacy `storage:mount <app> <host>:<container>` colon form continues to work on docker-local apps but is deprecated; on k3s apps it is rejected. Existing colon-form mounts are migrated automatically the first time the new storage plugin runs (during the install trigger) - they appear as `legacy-<hash>` entries in `storage:list-entries`. The migration is idempotent and tied to a per-app flag file at `$DOKKU_LIB_ROOT/config/storage/.migrated/<app>`; deleting that file forces a re-scan on the next install. The `storage:ensure-directory` command keeps working but now emits a deprecation warning - prefer `storage:create <name> [<path>]` (the path defaults to the same `$DOKKU_LIB_ROOT/data/storage/<name>` location). Storage entry names must now be DNS-1123 labels of 45 characters or less so they can be used verbatim as Helm release and Kubernetes resource names; underscores and uppercase characters that the older `ensure-directory` validator accepted are rejected for new names. The migration synthesizer always uses lowercase hex hashes so existing data is never locked out.

 ### TLS handshake behavior change
--- a/plugins/docker-options/dockeroptions.go
+++ b/plugins/docker-options/dockeroptions.go
@@ -2,6 +2,7 @@ package dockeroptions

 import (
 	"fmt"
+	"slices"
 	"sort"
 	"strings"

@@ -36,66 +37,126 @@ func SplitOptionString(input string) (options []string, processes []string, err
 		return nil, nil, fmt.Errorf("Unable to parse docker option: %s", err.Error())
 	}

-	var current []string
-	flush := func() error {
-		if len(current) == 0 {
-			return nil
-		}
-		head := current[0]
+	for _, group := range groupOptionTokens(fields) {
+		head := group[0]
 		if head == "--process" {
-			if len(current) < 2 {
-				return fmt.Errorf("--process requires a value")
+			if len(group) < 2 {
+				return nil, nil, fmt.Errorf("--process requires a value")
 			}
-			if len(current) > 2 {
-				return fmt.Errorf("--process accepts a single value, got %d", len(current)-1)
+			if len(group) > 2 {
+				return nil, nil, fmt.Errorf("--process accepts a single value, got %d", len(group)-1)
 			}
-			processes = append(processes, current[1])
-			current = nil
-			return nil
+			processes = append(processes, group[1])
+			continue
 		}
 		if strings.HasPrefix(head, "--process=") {
-			if len(current) > 1 {
-				return fmt.Errorf("--process=value cannot be followed by additional tokens")
+			if len(group) > 1 {
+				return nil, nil, fmt.Errorf("--process=value cannot be followed by additional tokens")
 			}
 			processes = append(processes, head[len("--process="):])
-			current = nil
-			return nil
+			continue
 		}
-		options = append(options, joinShellTokens(current))
-		current = nil
-		return nil
-	}
-
-	for _, tok := range fields {
-		if isFlagToken(tok) && len(current) > 0 {
-			if err := flush(); err != nil {
-				return nil, nil, err
-			}
-		}
-		current = append(current, tok)
-	}
-	if err := flush(); err != nil {
-		return nil, nil, err
+		options = append(options, joinShellTokens(group))
 	}

 	return options, processes, nil
 }

+// groupOptionTokens groups shell words on flag boundaries so each group holds
+// one flag and the values that follow it. A group is emitted for every token
+// that looks like a flag, meaning `--build-arg X=Y --link a` yields
+// [["--build-arg" "X=Y"] ["--link" "a"]]. Tokens preceding the first flag stay
+// in the leading group so malformed input is never silently dropped.
+func groupOptionTokens(fields []string) [][]string {
+	var groups [][]string
+	var current []string
+	for _, tok := range fields {
+		if isFlagToken(tok) && len(current) > 0 {
+			groups = append(groups, current)
+			current = nil
+		}
+		current = append(current, tok)
+	}
+	if len(current) > 0 {
+		groups = append(groups, current)
+	}
+	return groups
+}
+
+// canonicalOptionsFromLine re-serializes a stored option line into the
+// canonical, shell-quoted form `docker-options:add` produces, splitting it on
+// flag boundaries so a line carrying several flags becomes one entry per flag.
+// Unlike SplitOptionString it never lifts `--process`: a stored line is data
+// rather than command-line input.
+//
+// The line is returned unchanged when it cannot be parsed, or when the parser
+// stops before consuming all of it - a line whose tail begins with an unquoted
+// `#` reads as a shell comment, and dropping it would lose stored data.
+func canonicalOptionsFromLine(line string) []string {
+	fields, end, err := literalFieldsWithEnd(line)
+	if err != nil || end < len(strings.TrimRight(line, " \t")) {
+		return []string{line}
+	}
+
+	groups := groupOptionTokens(fields)
+	options := make([]string, 0, len(groups))
+	for _, group := range groups {
+		options = append(options, joinShellTokens(group))
+	}
+	return options
+}
+
+// optionsEqual reports whether two option strings denote the same docker
+// option. Exact equality short-circuits; otherwise both sides are
+// shell-tokenized without expansion and compared word by word, so an option
+// held in a non-canonical form - drained verbatim out of a pre-0.38.25
+// DOCKER_OPTIONS_<PHASE> file, or written directly by another plugin - still
+// matches the canonically re-serialized string the CLI builds. Input that
+// cannot be parsed never matches.
+func optionsEqual(stored string, option string) bool {
+	if stored == option {
+		return true
+	}
+
+	storedFields, err := literalFields(stored)
+	if err != nil {
+		return false
+	}
+
+	optionFields, err := literalFields(option)
+	if err != nil {
+		return false
+	}
+
+	return slices.Equal(storedFields, optionFields)
+}
+
 // literalFields splits input into shell words using the parser directly, so
 // quotes delimit words and are stripped from the stored value, but nothing is
 // expanded. Parameter expansions, command substitutions, and other
 // metacharacters are preserved verbatim. Malformed input, such as an unbalanced
 // quote, returns the parser error.
 func literalFields(input string) ([]string, error) {
+	fields, _, err := literalFieldsWithEnd(input)
+	return fields, err
+}
+
+// literalFieldsWithEnd is literalFields plus the byte offset just past the last
+// word the parser consumed. Callers that must not lose data compare the offset
+// against the length of the input to detect a tail the parser skipped, such as
+// a shell comment.
+func literalFieldsWithEnd(input string) ([]string, int, error) {
 	parser := syntax.NewParser()
 	var fields []string
+	end := 0
 	for word, err := range parser.WordsSeq(strings.NewReader(input)) {
 		if err != nil {
-			return nil, err
+			return nil, 0, err
 		}
 		fields = append(fields, literalWordValue(input, word.Parts))
+		end = int(word.End().Offset())
 	}
-	return fields, nil
+	return fields, end, nil
 }

 // literalWordValue reconstructs the unquoted, unexpanded value of a shell word.
@@ -304,7 +365,11 @@ func RemoveDockerOptionFromPhases(appName string, phases []string, option string
 	return RemoveDockerOptionFromProcessPhases(appName, []string{DefaultProcessType}, phases, option)
 }

-// RemoveDockerOptionFromProcessPhases removes an option from the specified process types and phases.
+// RemoveDockerOptionFromProcessPhases removes an option from the specified
+// process types and phases. Stored options are matched against the requested
+// option by shell word rather than by raw string, so an entry held in a
+// non-canonical form still matches the canonically re-serialized string the
+// CLI hands down.
 func RemoveDockerOptionFromProcessPhases(appName string, processTypes []string, phases []string, option string) error {
 	if len(processTypes) == 0 {
 		processTypes = []string{DefaultProcessType}
@@ -318,7 +383,7 @@ func RemoveDockerOptionFromProcessPhases(appName string, processTypes []string,

 			newOptions := []string{}
 			for _, opt := range options {
-				if opt != option {
+				if !optionsEqual(opt, option) {
 					newOptions = append(newOptions, opt)
 				}
 			}
--- a/plugins/docker-options/dockeroptions_test.go
+++ b/plugins/docker-options/dockeroptions_test.go
@@ -175,6 +175,142 @@ func TestQuoteShellArg(t *testing.T) {
 	}
 }

+func TestOptionsEqual(t *testing.T) {
+	cases := []struct {
+		name   string
+		stored string
+		option string
+		want   bool
+	}{
+		{
+			name:   "identical strings",
+			stored: "-v /tmp:/tmp",
+			option: "-v /tmp:/tmp",
+			want:   true,
+		},
+		{
+			name:   "legacy unquoted command substitution matches canonical form",
+			stored: "--group-add $(getent group docker | cut -d: -f3)",
+			option: "--group-add '$(getent group docker | cut -d: -f3)'",
+			want:   true,
+		},
+		{
+			name:   "legacy unquoted parameter expansion matches canonical form",
+			stored: "--label FOO=$BAR",
+			option: "--label 'FOO=$BAR'",
+			want:   true,
+		},
+		{
+			name:   "double quoted matches single quoted",
+			stored: `--label "hello world"`,
+			option: "--label 'hello world'",
+			want:   true,
+		},
+		{
+			name:   "different values do not match",
+			stored: "-v /tmp:/tmp",
+			option: "-v /var:/var",
+			want:   false,
+		},
+		{
+			name:   "different flags do not match",
+			stored: "--link foo",
+			option: "--link bar",
+			want:   false,
+		},
+		{
+			name:   "multi-flag entry does not match a single flag",
+			stored: "-v /a:/a -v /b:/b",
+			option: "-v /a:/a",
+			want:   false,
+		},
+		{
+			name:   "unbalanced quote never matches",
+			stored: "--label 'unbalanced",
+			option: "--label unbalanced",
+			want:   false,
+		},
+		{
+			name:   "empty stored entry does not match an option",
+			stored: "",
+			option: "-v /tmp:/tmp",
+			want:   false,
+		},
+	}
+
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			if got := optionsEqual(tc.stored, tc.option); got != tc.want {
+				t.Errorf("optionsEqual(%q, %q) = %v, want %v", tc.stored, tc.option, got, tc.want)
+			}
+		})
+	}
+}
+
+func TestCanonicalOptionsFromLine(t *testing.T) {
+	cases := []struct {
+		name string
+		line string
+		want []string
+	}{
+		{
+			name: "quotes an unquoted command substitution",
+			line: "--group-add $(getent group docker | cut -d: -f3)",
+			want: []string{"--group-add '$(getent group docker | cut -d: -f3)'"},
+		},
+		{
+			name: "splits a line carrying several flags",
+			line: "-v /a:/a -v /b:/b",
+			want: []string{"-v /a:/a", "-v /b:/b"},
+		},
+		{
+			name: "leaves an already canonical line untouched",
+			line: "--group-add '$(getent group docker | cut -d: -f3)'",
+			want: []string{"--group-add '$(getent group docker | cut -d: -f3)'"},
+		},
+		{
+			name: "leaves a plain option untouched",
+			line: "--restart=on-failure:5",
+			want: []string{"--restart=on-failure:5"},
+		},
+		{
+			name: "preserves a line with an unbalanced quote",
+			line: "--label 'unbalanced",
+			want: []string{"--label 'unbalanced"},
+		},
+		{
+			name: "preserves a line whose tail reads as a shell comment",
+			line: "-v /a:/a # a note",
+			want: []string{"-v /a:/a # a note"},
+		},
+		{
+			name: "returns nothing for an empty line",
+			line: "",
+			want: nil,
+		},
+	}
+
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			got := canonicalOptionsFromLine(tc.line)
+			if !equalStringSlice(got, tc.want) {
+				t.Errorf("canonicalOptionsFromLine(%q) = %q, want %q", tc.line, got, tc.want)
+			}
+
+			if len(tc.want) == 0 {
+				return
+			}
+
+			// Canonicalization must be idempotent so the one-time repair
+			// pass is a no-op on stores it has already rewritten.
+			again := canonicalizeOptionLines(got)
+			if !equalStringSlice(again, got) {
+				t.Errorf("canonicalizing %q again = %q, want %q", got, again, got)
+			}
+		})
+	}
+}
+
 func equalStringSlice(a, b []string) bool {
 	if len(a) == 0 && len(b) == 0 {
 		return true
--- a/plugins/docker-options/functions.go
+++ b/plugins/docker-options/functions.go
@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"slices"
 	"strings"

 	"github.com/dokku/dokku/plugins/common"
@@ -278,10 +279,6 @@ func convertLegacyMigratedMarker(appName, phase string) error {
 // previous release - who have `migrated-from-files` set AND `.migrated`
 // files on disk - still get those files drained into properties.
 func migrateLegacyDockerOptionsFiles() error {
-	if common.PropertyGet("docker-options", "--global", "migrated-from-files") == "true" {
-		return nil
-	}
-
 	apps, err := common.DokkuApps()
 	if err != nil {
 		if errors.Is(err, common.NoAppsExist) {
@@ -301,6 +298,10 @@ func migrateLegacyDockerOptionsFiles() error {
 		}
 	}

+	if common.PropertyGet("docker-options", "--global", "migrated-from-files") == "true" {
+		return nil
+	}
+
 	for _, appName := range apps {
 		for _, phase := range availablePhases {
 			legacyPath := filepath.Join(common.AppRoot(appName), "DOCKER_OPTIONS_"+strings.ToUpper(phase))
@@ -313,8 +314,9 @@ func migrateLegacyDockerOptionsFiles() error {
 				return err
 			}

-			if len(lines) > 0 {
-				if err := common.PropertyListWrite("docker-options", appName, propertyKey(DefaultProcessType, phase), lines); err != nil {
+			options := canonicalizeOptionLines(lines)
+			if len(options) > 0 {
+				if err := common.PropertyListWrite("docker-options", appName, propertyKey(DefaultProcessType, phase), options); err != nil {
 					return err
 				}
 				if err := common.PropertyWrite("docker-options", appName, migratedPropertyKey(phase), "true"); err != nil {
@@ -352,6 +354,17 @@ func readLegacyOptionsFile(path string) ([]string, error) {
 	return lines, nil
 }

+// canonicalizeOptionLines re-serializes stored option lines into the canonical
+// form `docker-options:add` writes, flattening the per-line results so a line
+// carrying several flags becomes one entry per flag. Order is preserved.
+func canonicalizeOptionLines(lines []string) []string {
+	options := make([]string, 0, len(lines))
+	for _, line := range lines {
+		options = append(options, canonicalOptionsFromLine(line)...)
+	}
+	return options
+}
+
 // traefikLabelMigrationKey gates the one-time pass that repairs docker-option
 // labels whose backticks were stored with a stray leading backslash by an
 // earlier release's option tokenizer (e.g. Traefik rules like Host(\`app\`)).
@@ -451,3 +464,63 @@ func labelSpec(option string) (string, bool) {
 	s = strings.TrimPrefix(s, "\"")
 	return s, true
 }
+
+// canonicalOptionsMigrationKey gates the one-time pass that re-serializes
+// stored docker options into the canonical form. Options drained out of the
+// legacy DOCKER_OPTIONS_<PHASE> flat files by releases before this one were
+// copied verbatim, so they never matched the canonically re-serialized string
+// `docker-options:remove` compares against.
+const canonicalOptionsMigrationKey = "migrated-canonical-options"
+
+// migrateNonCanonicalOptions rewrites stored docker options into the canonical
+// form so the whole store uses the same representation `docker-options:add`
+// writes. Two things get repaired: values whose shell metacharacters were never
+// quoted, and lines carrying several flags, which are split into one entry per
+// flag so a single flag can be removed and so the prefix-matching readers
+// (`getRestartPolicy`, `GetSpecifiedDockerOptionsForPhase`) see one value per
+// entry. Canonicalization is idempotent, so options written through the CLI are
+// left untouched. It runs once, guarded by a global property.
+func migrateNonCanonicalOptions() error {
+	if common.PropertyGet("docker-options", "--global", canonicalOptionsMigrationKey) == "true" {
+		return nil
+	}
+
+	apps, err := common.DokkuApps()
+	if err != nil {
+		if errors.Is(err, common.NoAppsExist) {
+			return common.PropertyWrite("docker-options", "--global", canonicalOptionsMigrationKey, "true")
+		}
+		return err
+	}
+
+	for _, appName := range apps {
+		properties, err := common.PropertyGetAll("docker-options", appName)
+		if err != nil {
+			return err
+		}
+
+		for key := range properties {
+			processType, phase, ok := splitPropertyKey(key)
+			if !ok {
+				continue
+			}
+
+			options, err := GetDockerOptionsForProcessPhase(appName, processType, phase)
+			if err != nil {
+				return err
+			}
+
+			canonical := canonicalizeOptionLines(options)
+			if slices.Equal(canonical, options) {
+				continue
+			}
+
+			if err := writeDockerOptionsForProcessPhase(appName, processType, phase, canonical); err != nil {
+				return err
+			}
+			common.LogInfo1(fmt.Sprintf("Rewrote docker-options into canonical form for %s (%s %s)", appName, processType, phase))
+		}
+	}
+
+	return common.PropertyWrite("docker-options", "--global", canonicalOptionsMigrationKey, "true")
+}
--- a/plugins/docker-options/migration_test.go
+++ b/plugins/docker-options/migration_test.go
@@ -278,6 +278,147 @@ func TestMigrateTraefikLabelBackticks(t *testing.T) {
 	}
 }

+// TestMigrateLegacyDockerOptionsFiles_CanonicalizesLines covers the
+// regression behind #8904: legacy lines were drained verbatim, so an
+// option holding shell metacharacters could never match the
+// canonically re-serialized string `docker-options:remove` builds.
+func TestMigrateLegacyDockerOptionsFiles_CanonicalizesLines(t *testing.T) {
+	dokkuRoot := setupMigrationEnv(t)
+
+	writeLegacyDockerOptionsFile(t, dokkuRoot, "alpha", "DEPLOY", "--group-add $(getent group docker | cut -d: -f3)\n-v /a:/a -v /b:/b\n")
+
+	if err := migrateLegacyDockerOptionsFiles(); err != nil {
+		t.Fatalf("migrateLegacyDockerOptionsFiles: %v", err)
+	}
+
+	got, err := GetDockerOptionsForProcessPhase("alpha", "_default_", "deploy")
+	if err != nil {
+		t.Fatalf("GetDockerOptionsForProcessPhase: %v", err)
+	}
+	want := []string{
+		"--group-add '$(getent group docker | cut -d: -f3)'",
+		"-v /a:/a",
+		"-v /b:/b",
+	}
+	if !equalStrings(got, want) {
+		t.Errorf("alpha deploy = %q, want %q", got, want)
+	}
+}
+
+// TestMigrateNonCanonicalOptions covers the repair pass for installs
+// that already drained their legacy files under an earlier release, so
+// the non-canonical values are sitting in the property store.
+func TestMigrateNonCanonicalOptions(t *testing.T) {
+	dokkuRoot := setupMigrationEnv(t)
+	if err := os.MkdirAll(filepath.Join(dokkuRoot, "alpha"), 0755); err != nil {
+		t.Fatalf("MkdirAll: %v", err)
+	}
+
+	if err := common.PropertyListWrite("docker-options", "alpha", "_default_.deploy", []string{
+		"--group-add $(getent group docker | cut -d: -f3)",
+		"-v /a:/a -v /b:/b",
+	}); err != nil {
+		t.Fatalf("seed default scope: %v", err)
+	}
+	if err := common.PropertyListWrite("docker-options", "alpha", "web.deploy", []string{
+		"--label FOO=$BAR",
+	}); err != nil {
+		t.Fatalf("seed web scope: %v", err)
+	}
+	if err := common.PropertyListWrite("docker-options", "alpha", "_default_.build", []string{
+		"--build-arg FOO=bar",
+	}); err != nil {
+		t.Fatalf("seed build scope: %v", err)
+	}
+
+	if err := migrateNonCanonicalOptions(); err != nil {
+		t.Fatalf("migrateNonCanonicalOptions: %v", err)
+	}
+
+	deploy, err := GetDockerOptionsForProcessPhase("alpha", "_default_", "deploy")
+	if err != nil {
+		t.Fatalf("GetDockerOptionsForProcessPhase deploy: %v", err)
+	}
+	wantDeploy := []string{
+		"--group-add '$(getent group docker | cut -d: -f3)'",
+		"-v /a:/a",
+		"-v /b:/b",
+	}
+	if !equalStrings(deploy, wantDeploy) {
+		t.Errorf("alpha deploy = %q, want %q", deploy, wantDeploy)
+	}
+
+	web, err := GetDockerOptionsForProcessPhase("alpha", "web", "deploy")
+	if err != nil {
+		t.Fatalf("GetDockerOptionsForProcessPhase web.deploy: %v", err)
+	}
+	if !equalStrings(web, []string{"--label 'FOO=$BAR'"}) {
+		t.Errorf("alpha web.deploy = %q, want [--label 'FOO=$BAR']", web)
+	}
+
+	build, err := GetDockerOptionsForProcessPhase("alpha", "_default_", "build")
+	if err != nil {
+		t.Fatalf("GetDockerOptionsForProcessPhase build: %v", err)
+	}
+	if !equalStrings(build, []string{"--build-arg FOO=bar"}) {
+		t.Errorf("alpha build = %q, want [--build-arg FOO=bar]", build)
+	}
+
+	if common.PropertyGet("docker-options", "--global", canonicalOptionsMigrationKey) != "true" {
+		t.Errorf("expected %s guard to be set", canonicalOptionsMigrationKey)
+	}
+
+	if err := migrateNonCanonicalOptions(); err != nil {
+		t.Fatalf("second migrateNonCanonicalOptions: %v", err)
+	}
+	deployAfter, err := GetDockerOptionsForProcessPhase("alpha", "_default_", "deploy")
+	if err != nil {
+		t.Fatalf("GetDockerOptionsForProcessPhase deploy (post re-run): %v", err)
+	}
+	if !equalStrings(deployAfter, wantDeploy) {
+		t.Errorf("alpha deploy after rerun = %q, want %q (idempotency violated)", deployAfter, wantDeploy)
+	}
+}
+
+// TestRemoveDockerOptionFromPhases_MatchesNonCanonicalStoredOption is
+// the regression test for #8904 itself: an option left in the store in
+// its pre-0.38.25 unquoted form must still be removable using the
+// string a user would type.
+func TestRemoveDockerOptionFromPhases_MatchesNonCanonicalStoredOption(t *testing.T) {
+	dokkuRoot := setupMigrationEnv(t)
+	if err := os.MkdirAll(filepath.Join(dokkuRoot, "alpha"), 0755); err != nil {
+		t.Fatalf("MkdirAll: %v", err)
+	}
+
+	stored := "--group-add $(getent group docker | cut -d: -f3)"
+	if err := common.PropertyListWrite("docker-options", "alpha", "_default_.deploy", []string{
+		stored,
+		"-v /keep:/keep",
+	}); err != nil {
+		t.Fatalf("seed: %v", err)
+	}
+
+	options, _, err := SplitOptionString(stored)
+	if err != nil {
+		t.Fatalf("SplitOptionString: %v", err)
+	}
+	if len(options) != 1 {
+		t.Fatalf("SplitOptionString returned %d options, want 1", len(options))
+	}
+
+	if err := RemoveDockerOptionFromPhases("alpha", []string{"deploy"}, options[0]); err != nil {
+		t.Fatalf("RemoveDockerOptionFromPhases: %v", err)
+	}
+
+	got, err := GetDockerOptionsForProcessPhase("alpha", "_default_", "deploy")
+	if err != nil {
+		t.Fatalf("GetDockerOptionsForProcessPhase: %v", err)
+	}
+	if !equalStrings(got, []string{"-v /keep:/keep"}) {
+		t.Errorf("alpha deploy = %q, want [-v /keep:/keep]", got)
+	}
+}
+
 func equalStrings(a, b []string) bool {
 	if len(a) != len(b) {
 		return false
--- a/plugins/docker-options/triggers.go
+++ b/plugins/docker-options/triggers.go
@@ -24,6 +24,10 @@ func TriggerInstall() error {
 		return fmt.Errorf("Unable to repair docker-options label backticks: %v", err)
 	}

+	if err := migrateNonCanonicalOptions(); err != nil {
+		return fmt.Errorf("Unable to rewrite docker-options into canonical form: %v", err)
+	}
+
 	return nil
 }

--- a/tests.mk
+++ b/tests.mk
@@ -149,6 +149,7 @@ ci-go-coverage:
 	@$(MAKE) ci-go-coverage-plugin PLUGIN_NAME=certs
 	@$(MAKE) ci-go-coverage-plugin PLUGIN_NAME=common
 	@$(MAKE) ci-go-coverage-plugin PLUGIN_NAME=config
+	@$(MAKE) ci-go-coverage-plugin PLUGIN_NAME=docker-options
 	@$(MAKE) ci-go-coverage-plugin PLUGIN_NAME=network
 	@$(MAKE) ci-go-coverage-plugin PLUGIN_NAME=buildpacks

@@ -180,6 +181,7 @@ go-tests:
 	@$(MAKE) go-test-plugin PLUGIN_NAME=certs
 	@$(MAKE) go-test-plugin PLUGIN_NAME=common
 	@$(MAKE) go-test-plugin PLUGIN_NAME=config
+	@$(MAKE) go-test-plugin PLUGIN_NAME=docker-options
 	@$(MAKE) go-test-plugin PLUGIN_NAME=network
 	@$(MAKE) go-test-plugin PLUGIN_NAME=buildpacks

--- a/tests/unit/docker-options.bats
+++ b/tests/unit/docker-options.bats
@@ -441,6 +441,47 @@ teardown() {
  assert_output_contains "--link foo" 0
 }

+@test "(docker-options:remove) removes an option stored in the pre-0.38.25 unquoted form" {
+  # Options drained out of a legacy DOCKER_OPTIONS_<PHASE> file by
+  # releases before 0.38.26 were copied verbatim, so a value holding
+  # shell metacharacters sits in the property store unquoted. Seed that
+  # state directly - the add path always canonicalizes, so it cannot
+  # produce it.
+  run /bin/bash -c "dokku docker-options:add $TEST_APP deploy \"-v /tmp/keep:/keep\""
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  legacy_option='--group-add $(getent group docker | cut -d: -f3)'
+  property_file="/var/lib/dokku/config/docker-options/$TEST_APP/_default_.deploy"
+  run /bin/bash -c "printf '%s\n-v /tmp/keep:/keep\n' '$legacy_option' | sudo tee $property_file"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "sudo chown dokku:dokku $property_file"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "dokku docker-options:list $TEST_APP --phase deploy"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+  assert_output_contains "$legacy_option"
+
+  run /bin/bash -c "dokku docker-options:remove $TEST_APP deploy '$legacy_option'"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "dokku docker-options:list $TEST_APP --phase deploy"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+  assert_output "-v /tmp/keep:/keep"
+}
+
@test "(docker-options) dockerfile build skips unsupported flags from multi-flag input" {
  run /bin/bash -c "dokku docker-options:add $TEST_APP build --build-arg PAYPAL_CLIENT_ID=abc --link postgres"
  echo "output: $output"