Merge pull request #4460 from dokku/4219-registry-plugin

Implement registry plugin

.github/workflows/ci.yml

@@ -67,6 +67,8 @@ jobs:
         timeout-minutes: 30
         run: sudo -E ./.github/commands/ci-run ${{ matrix.index }}
         env:
+          DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
+          DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
           SYNC_GITHUB_PASSWORD: ${{ secrets.SYNC_GITHUB_PASSWORD }}
           SYNC_GITHUB_USERNAME: ${{ secrets.SYNC_GITHUB_USERNAME }}
 

docs/advanced-usage/registry-management.md

@@ -0,0 +1,126 @@
+# Registry Management
+
+> New as of 0.25.0
+
+```
+registry:login [--password-stdin] <server> <username> [<password>] # Login to a docker registry
+registry:report [<app>] [<flag>]                                   # Displays a registry report for one or more apps
+registry:set <app> <key> (<value>)                                 # Set or clear a registry property for an app
+```
+
+The registry plugin enables interacting with remote registries, which is useful when either deploying images via `git:from-image` or when interacting with custom schedulers to deploy built image artifacts.
+
+## Usage
+
+### Logging into a registry
+
+The `registry:login` command can be used to log into a docker registry. The following are examples for logging into various common registries:
+
+```shell
+# hub.docker.com
+dokku registry:login docker.io $USERNAME $PASSWORD
+
+# digitalocean
+# the username and password are both defined as the same api token
+dokku registry:login registry.digitalocean.com $DIGITALOCEAN_API_TOKEN $DIGITALOCEAN_API_TOKEN
+
+# github container registry
+# see the following link for information on retrieving a personal access token
+#   https://docs.github.com/en/packages/guides/pushing-and-pulling-docker-images#authenticating-to-github-container-registry
+dokku registry:login ghcr.io $USERNAME $REGISTRY_PAT_TOKEN
+
+# quay
+# a robot user may be used to login
+dokku registry:login quay.io $USERNAME $PASSWORD
+```
+
+For security reasons, the password may also be specified as stdin by specifying the `--password-stdin` flag. This is supported regardless of the registry being logged into.
+
+```shell
+echo "$PASSWORD" | dokku registry:login --password-stdin docker.io $USERNAME
+```
+
+For certain Docker registries - such as Amazon ECR or Google's GCR registries - users may instead wish to use a docker credential helper to automatically authenticate against a server; please see the documentation regarding the credential helper in question for further setup instructions.
+
+### Setting a remote server
+
+To specify a remote server registry for pushes, set the `server` property via the `registry:set` command. The default value for this property is empty string. Setting the value to `docker.io` or `hub.docker.com` will result in the computed value being empty string (as that is the default, implicit registry), while any non-zero length value will have a `/` appended to it if there is not one already.
+
+```shell
+dokku registry:set node-js-app server docker.io
+```
+
+This property can be set for a single app or globally via the `--global` flag. When set globally, the app-specific value will always overide the global value. The default global value for this property is empty string.
+
+```shell
+dokku registry:set --global server docker.io
+```
+
+Setting the property value to an empty string will reset the value to the system default. Resetting the value can be done per app or globally.
+
+```shell
+# per-app
+dokku registry:set node-js-app server
+
+# globally
+dokku registry:set --global server
+```
+
+The following are the values that should be used for common remote servers:
+
+- Amazon Elastic Container Registry:
+  - value: `$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/`
+  - notes: The `$AWS_ACCOUNT_ID` and `$AWS_REGION` should match the values for your account and region, respectively. Additionally, an IAM profile that allows `push` access to the repository specified by `image-repo` should be attached to your Dokku server.
+- Azure Container Registry:
+  - value `$REGISTRY_NAME.azurecr.io/`
+  - notes: The `$AKS_REGISTRY_NAME` should match the name of the registry created on your account.
+- Docker Hub:
+  - value: `docker.io/`
+  - notes: Requires owning the namespace used in the `image-repo` value.
+- Digitalocean:
+  - value: `registry.digitalocean.com/`
+  - notes: Requires setting the correct `image-repo` value for your registry.
+- Github Container Registry:
+  - value: `ghcr.io/`
+  - notes: Requires that the authenticated user has access to the namespace used in the `image-repo` value.
+- Quay.io:
+  - value: `quay.io/`
+
+### Specifying an image repository name
+
+By default, Dokku uses the value `dokku/$APP_NAME` as the image repository that is pushed and deployed. For certain registries, the `dokku` namespace may not be available to your user. In these cases, the value can be set by changing the value of the `image-repo` property via the `registry:set` command.
+
+```shell
+dokku registry:set node-js-app image-repo my-awesome-prefix/node-js-app
+```
+
+Setting the property value to an empty string will reset the value to the system default. Resetting the value has to be done per-app.
+
+```shell
+# per-app
+dokku registry:set node-js-app push-on-release
+```
+
+### Pushing images on build
+
+To push the image on release, set the `push-on-release` property to `true` via the `registry:set` command. The default value for this property is `false`. Setting the property to `true` will result in the imag being tagged with an ID that is incremented with every release. This tag will be what is used for running app code.
+
+```shell
+dokku registry:set node-js-app push-on-release true
+```
+
+This property can be set for a single app or globally via the `--global` flag. When set globally, the app-specific value will always overide the global value. The default global value for this property is `false`.
+
+```shell
+dokku registry:set --global push-on-release true
+```
+
+Setting the property value to an empty string will reset the value to the system default. Resetting the value can be done per app or globally.
+
+```shell
+# per-app
+dokku registry:set node-js-app push-on-release
+
+# globally
+dokku registry:set --global push-on-release
+```

docs/appendices/0.25.0-migration-guide.md

@@ -1,16 +1,29 @@
 # 0.25.0 Migration Guide
 
-## Changes
+## Registry Plugin
+
+The [dokku-registry](https://github.com/dokku/dokku-registry) plugin is now built-in. This comes with a few changes:
+
+- Builder plugins should call `post-release-builder` at the end of the build.
+- The `push` and `pull` command are not implemented. Users wishing to deploy a remote image should use `git:from-image`. Image pushing is not available at this time.
+- At this time, remote docker repositories are not automatically created for AWS, and users must create those repositories for their applications as necessary. This may be implemented in the future.
+- Docker images are only pushed when configured to do so. See the [registry management documentation](/docs/deployment/registry-management.md) for more details.
+
+## Other
+
+### Changes
 
 - The network plugin can now set an `initial-network` for all containers on creation. This is a replacement for specifying the `--network` flag via the `docker-options` plugin. Please see the [network documentation](/docs/networking/network.md#attaching-an-app-to-a-network) for more information.
 - The `dokku run` command now always removes the ephemeral container on exit. Users that need a persistent container should instead specify a `console` process type in their `Procfile` specifying an available shell (usually either `bash` or `sh`) and scale that container appropriately.
+- The `pre-deploy` plugin trigger is now called internally by Dokku. Scheduler plugins should avoid calling this trigger, as any image changes introduced by subsequent trigger calls will be ignored.
 
-## Deprecations
+### Deprecations
 
 - In previous versions of Dokku, the only way to specify a custom `Dockerfile` was to use the `docker-options` plugin to set the `--file` flag for a docker build. As of 0.25.0, the `builder-dockerfile:set` command should be used instead, as outlined in the [docs here](/docs/deployment/builders/dockerfiles.md#changingthe-dockerfile-location). Usage of the old method should be migrated to the new method.
 - The `--rm` and ``--rm-container` flags may be specified but no longer have any effect on `dokku run`.
 - The `--detach` flag is deprecated in favor of the `run:detached` command.
 - The `DOKKU_SCALE` file is deprecated. Please see the [process management documentation](/docs/processes/process-management.md#manually-managing-process-scaling) for more information on it's replacement with the `formations` key of the `app.json` file.
+- The hooks `post-release-buildpack`, `post-release-dockerfile`, and `post-release-pack` are deprecated in favor of `post-release-builder`. See the [plugin triggers documentation](https://dokku.com/docs/development/plugin-triggers/#post-release-builder) for more details.
 
 ## Removals
 

docs/development/plugin-triggers.md

@@ -1345,8 +1345,28 @@ APP="$1"
 haproxy-build-config "$APP"
 ```
 
+### `post-release-builder`
+
+> Warning: Image mutation in this trigger may result in an invalid run state, and is heavily discouraged.
+
+- Description: Invokes a command after the build process is complete.
+- Invoked by: builder plugins
+- Arguments: `$BUILDER_TYPE $APP $IMAGE`
+- Example:
+
+```shell
+#!/usr/bin/env bash
+
+set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
+source "$PLUGIN_CORE_AVAILABLE_PATH/common/functions"
+BUILDER_TYPE="$1"; APP="$2"; IMAGE=$3
+
+# TODO
+```
+
 ### `post-release-buildpack`
 
+> Warning: Deprecated, please use `post-release-builder` instead
 > Warning: Image mutation in this trigger may result in an invalid run state, and is heavily discouraged.
 
 - Description: Allows you to run commands after environment variables are set for the release step of the deploy. Only applies to apps using buildpacks.
@@ -1366,6 +1386,8 @@ APP="$1"; IMAGE_TAG="$2"; IMAGE=$(get_app_image_name $APP $IMAGE_TAG)
 
 ### `post-release-pack`
 
+> Warning: Deprecated, please use `post-release-builder` instead
+
 > Warning: The pack plugin trigger apis are under development and may change
 > between minor releases until the 1.0 release.
 
@@ -1388,6 +1410,8 @@ APP="$1"; IMAGE_TAG="$2"; IMAGE=$(get_app_image_name $APP $IMAGE_TAG)
 
 ### `post-release-dockerfile`
 
+> Warning: Deprecated, please use `post-release-builder` instead
+
 > Warning: Image mutation in this trigger may result in an invalid run state, and is heavily discouraged.
 
 - Description: Allows you to run commands after environment variables are set for the release step of the deploy. Only applies to apps using a dockerfile.

docs/template.html

@@ -150,6 +150,7 @@
             <a href="/{{NAME}}/advanced-usage/backup-recovery/" class="list-group-item">Backup and Recovery</a>
             <a href="/{{NAME}}/advanced-usage/deployment-tasks/" class="list-group-item">Deployment Tasks</a>
             <a href="/{{NAME}}/advanced-usage/docker-options/" class="list-group-item">Docker Container Options</a>
+            <a href="/{{NAME}}/advanced-usage/registry-management/" class="list-group-item">Docker Registry Management</a>
             <a href="/{{NAME}}/advanced-usage/event-logs/" class="list-group-item">Event Logs</a>
             <a href="/{{NAME}}/advanced-usage/persistent-storage/" class="list-group-item">Persistent Storage</a>
             <a href="/{{NAME}}/advanced-usage/plugin-management/" class="list-group-item">Plugin Management</a>

docs/viewdocs.json

@@ -38,6 +38,9 @@
     "dokku-events-logs": "advanced-usage/event-logs/",
     "dokku-storage": "advanced-usage/persistent-storage/",
 
+    "advanced-usage/schedulers/alternate-schedulers": "deployment/schedulers/alternate-schedulers",
+    "advanced-usage/schedulers/docker-local": "deployment/schedulers/docker-local",
+
     "community/tutorials/deploying-with-gitlab-ci": "deployment/continuous-integration/gitlab-ci/",
 
     "plugins": "community/plugins/"

plugins/20_events/post-release-builder

@@ -0,0 +1 @@
+hook

plugins/app-json/triggers.go

@@ -47,11 +47,7 @@ func TriggerPostDeploy(appName string, imageTag string) error {
 
 // TriggerPreDeploy is a trigger to execute predeploy and release deployment tasks
 func TriggerPreDeploy(appName string, imageTag string) error {
-	image, err := common.GetDeployingAppImageName(appName, imageTag, "")
-	if err != nil {
-		return err
-	}
-
+	image := common.GetAppImageName(appName, imageTag, "")
 	if err := refreshAppJSON(appName, image); err != nil {
 		return err
 	}

plugins/builder-dockerfile/builder-release

@@ -17,6 +17,7 @@ trigger-builder-dockerfile-builder-release() {
   local IMAGE=$(get_app_image_name "$APP" "$IMAGE_TAG")
   docker-image-labeler --label=com.dokku.image-stage=release --label=com.dokku.app-name=$APP --label=org.label-schema.schema-version=1.0 --label=org.label-schema.vendor=dokku --label=dokku "$IMAGE"
   plugn trigger post-release-dockerfile "$APP" "$IMAGE_TAG"
+  plugn trigger post-release-builder "$BUILDER_TYPE" "$APP" "$IMAGE"
 }
 
 trigger-builder-dockerfile-builder-release "$@"

plugins/builder-herokuish/builder-release

@@ -42,6 +42,7 @@ trigger-builder-herokuish-builder-release() {
 
   docker-image-labeler --label=com.dokku.image-stage=release --label=com.dokku.app-name=$APP --label=org.label-schema.schema-version=1.0 --label=org.label-schema.vendor=dokku --label=dokku "$IMAGE"
   plugn trigger post-release-buildpack "$APP" "$IMAGE_TAG"
+  plugn trigger post-release-builder "$BUILDER_TYPE" "$APP" "$IMAGE"
 }
 
 trigger-builder-herokuish-builder-release "$@"

plugins/builder-pack/builder-release

@@ -17,6 +17,7 @@ trigger-builder-pack-builder-release() {
   local IMAGE=$(get_app_image_name "$APP" "$IMAGE_TAG")
   docker-image-labeler --label=com.dokku.image-stage=release --label=com.dokku.app-name=$APP --label=org.label-schema.schema-version=1.0 --label=org.label-schema.vendor=dokku --label=dokku "$IMAGE"
   plugn trigger post-release-pack "$APP" "$IMAGE_TAG"
+  plugn trigger post-release-builder "$BUILDER_TYPE" "$APP" "$IMAGE"
 }
 
 trigger-builder-pack-builder-release "$@"

plugins/builder/report.go

@@ -38,6 +38,7 @@ func reportComputedSelected(appName string) string {
 
 	return value
 }
+
 func reportGlobalSelected(appName string) string {
 	return common.PropertyGet("builder", "--global", "selected")
 }

plugins/builder/subcommands.go

@@ -4,7 +4,7 @@ import (
 	"github.com/dokku/dokku/plugins/common"
 )
 
-// CommandReport displays a network report for one or more apps
+// CommandReport displays a builder report for one or more apps
 func CommandReport(appName string, format string, infoFlag string) error {
 	if len(appName) == 0 {
 		apps, err := common.DokkuApps()

plugins/common/docker.go

@@ -210,7 +210,7 @@ func DockerCleanup(appName string, forceCleanup bool) error {
 	}
 
 	// delete dangling images
-	imageIDs, _ := listDanglingImages(appName)
+	imageIDs, _ := ListDanglingImages(appName)
 	if len(imageIDs) > 0 {
 		RemoveImages(imageIDs)
 	}
@@ -301,6 +301,35 @@ func IsImageHerokuishBased(image string, appName string) bool {
 	return output != ""
 }
 
+// ListDanglingImages lists all dangling image ids for a given app
+func ListDanglingImages(appName string) ([]string, error) {
+	command := []string{
+		DockerBin(),
+		"image",
+		"list",
+		"--quiet",
+		"--filter",
+		"dangling=true",
+	}
+
+	if appName != "" {
+		command = append(command, []string{"--filter", fmt.Sprintf("label=com.dokku.app-name=%v", appName)}...)
+	}
+
+	var stderr bytes.Buffer
+	listCmd := NewShellCmd(strings.Join(command, " "))
+	listCmd.ShowOutput = false
+	listCmd.Command.Stderr = &stderr
+	b, err := listCmd.Output()
+
+	if err != nil {
+		return []string{}, errors.New(strings.TrimSpace(stderr.String()))
+	}
+
+	output := strings.Split(strings.TrimSpace(string(b[:])), "\n")
+	return output, nil
+}
+
 // RemoveImages removes images by ID
 func RemoveImages(imageIDs []string) {
 	command := []string{
@@ -356,34 +385,6 @@ func listContainers(status string, appName string) ([]string, error) {
 	return output, nil
 }
 
-func listDanglingImages(appName string) ([]string, error) {
-	command := []string{
-		DockerBin(),
-		"image",
-		"list",
-		"--quiet",
-		"--filter",
-		"dangling=true",
-	}
-
-	if appName != "" {
-		command = append(command, []string{"--filter", fmt.Sprintf("label=com.dokku.app-name=%v", appName)}...)
-	}
-
-	var stderr bytes.Buffer
-	listCmd := NewShellCmd(strings.Join(command, " "))
-	listCmd.ShowOutput = false
-	listCmd.Command.Stderr = &stderr
-	b, err := listCmd.Output()
-
-	if err != nil {
-		return []string{}, errors.New(strings.TrimSpace(stderr.String()))
-	}
-
-	output := strings.Split(strings.TrimSpace(string(b[:])), "\n")
-	return output, nil
-}
-
 func pruneUnusedImages(appName string) {
 	command := []string{
 		DockerBin(),

plugins/common/functions

@@ -665,7 +665,7 @@ release_and_deploy() {
   source "$PLUGIN_AVAILABLE_PATH/config/functions"
 
   local APP="$1"
-  local IMAGE_TAG="$2"
+  local IMAGE_TAG="${2:-latest}"
   local IMAGE=$(get_app_image_name "$APP" "$IMAGE_TAG")
   local DOKKU_DOCKERFILE_PORTS
 
@@ -687,6 +687,7 @@ release_and_deploy() {
     local DOKKU_SKIP_DEPLOY=${DOKKU_APP_SKIP_DEPLOY:="$DOKKU_GLOBAL_SKIP_DEPLOY"}
 
     dokku_log_info1 "Releasing $APP..."
+    plugn trigger pre-deploy "$APP" "$IMAGE_TAG"
     dokku_release "$APP" "$IMAGE_SOURCE_TYPE" "$IMAGE_TAG"
 
     if [[ "$DOKKU_SKIP_DEPLOY" != "true" ]]; then

plugins/common/subprocess.go

@@ -61,6 +61,13 @@ func (sc *ShellCmd) Execute() bool {
 	return true
 }
 
+// Start is a wrapper around exec.Command.Start()
+func (sc *ShellCmd) Start() error {
+	sc.setup()
+
+	return sc.Command.Start()
+}
+
 // Output is a lightweight wrapper around exec.Command.Output()
 func (sc *ShellCmd) Output() ([]byte, error) {
 	sc.setup()

plugins/config/docker-args-deploy

@@ -10,7 +10,7 @@ trigger-config-docker-args() {
   declare APP="$1" IMAGE_TAG="$2"
   local ENV_ARGS IMAGE STDIN trigger
 
-  IMAGE=$(get_deploying_app_image_name "$APP" "$IMAGE_TAG")
+  IMAGE=$(get_app_image_name "$APP" "$IMAGE_TAG")
   STDIN=$(cat)
   trigger="$0 config_docker_args"
 

plugins/ps/triggers.go

@@ -224,10 +224,7 @@ func TriggerPostStop(appName string) error {
 
 // TriggerPreDeploy ensures an app has an up to date scale parameters
 func TriggerPreDeploy(appName string, imageTag string) error {
-	image, err := common.GetDeployingAppImageName(appName, imageTag, "")
-	if err != nil {
-		return err
-	}
+	image := common.GetAppImageName(appName, imageTag, "")
 
 	if err := removeProcfile(appName); err != nil {
 		return err

plugins/registry/.gitignore

@@ -0,0 +1,10 @@
+/commands
+/core-post-deploy
+/subcommands/*
+/triggers/*
+/triggers
+/network-*
+/install
+/post-*
+/report
+/deployed-*

plugins/registry/Makefile

@@ -0,0 +1,6 @@
+SUBCOMMANDS = subcommands/login subcommands/report subcommands/set
+TRIGGERS = triggers/deployed-app-image-repo triggers/deployed-app-image-tag triggers/deployed-app-repository triggers/install triggers/post-delete triggers/post-release-builder triggers/report
+BUILD = commands subcommands triggers
+PLUGIN_NAME = registry
+
+include ../../common.mk

plugins/registry/functions.go

@@ -0,0 +1,138 @@
+package registry
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"strconv"
+	"strings"
+
+	"github.com/codeskyblue/go-sh"
+	"github.com/dokku/dokku/plugins/common"
+)
+
+func getRegistryServerForApp(appName string) string {
+	value := common.PropertyGet("registry", appName, "server")
+	if value == "" {
+		value = common.PropertyGet("registry", "--global", "server")
+	}
+
+	value = strings.TrimSuffix(value, "/")
+	if value == "hub.docker.com" || value == "docker.io" {
+		value = ""
+	}
+
+	if value != "" {
+		value = value + "/"
+	}
+
+	return value
+}
+
+func isPushEnabled(appName string) bool {
+	return reportComputedPushOnRelease(appName) == "true"
+}
+
+func incrementTagVersion(appName string) (int, error) {
+	tag := common.PropertyGet("registry", appName, "tag-version")
+	if tag == "" {
+		tag = "0"
+	}
+
+	version, err := strconv.Atoi(tag)
+	if err != nil {
+		return 0, fmt.Errorf("Unable to convert existing tag version (%s) to integer: %v", tag, err)
+	}
+
+	version++
+	common.LogVerboseQuiet(fmt.Sprintf("Bumping tag to %d", version))
+	if err = common.PropertyWrite("registry", appName, "tag-version", strconv.Itoa(version)); err != nil {
+		return 0, err
+	}
+
+	return version, nil
+}
+
+func pushToRegistry(appName string, tag int, imageID string, imageRepo string) error {
+	common.LogVerboseQuiet("Retrieving image info for app")
+
+	registryServer := getRegistryServerForApp(appName)
+	imageTag, _ := common.GetRunningImageTag(appName)
+
+	fullImage := fmt.Sprintf("%s%s:%d", registryServer, imageRepo, tag)
+
+	common.LogVerboseQuiet(fmt.Sprintf("Tagging %s:%d in registry format", imageRepo, tag))
+	if !dockerTag(imageID, fullImage) {
+		// TODO: better error
+		return errors.New("Unable to tag image")
+	}
+
+	if !dockerTag(imageID, fmt.Sprintf("%s:%d", imageRepo, tag)) {
+		// TODO: better error
+		return errors.New("Unable to tag image")
+	}
+
+	// For the future, we should also add the ability to create the remote repository
+	// This is only really important for registries that do not support creation on push
+	// Examples include AWS and Quay.io
+
+	common.LogVerboseQuiet(fmt.Sprintf("Pushing %s", fullImage))
+	if !dockerPush(fullImage) {
+		// TODO: better error
+		return errors.New("Unable to push image")
+	}
+
+	// Only clean up when the scheduler is not docker-local
+	// other schedulers do not retire local images
+	if common.GetAppScheduler(appName) != "docker-local" {
+		common.LogVerboseQuiet("Cleaning up")
+		imageCleanup(appName, fmt.Sprintf("%s%s", registryServer, imageRepo), imageTag, tag)
+		if fmt.Sprintf("%s%s", registryServer, imageRepo) != imageRepo {
+			imageCleanup(appName, imageRepo, imageTag, tag)
+		}
+	}
+
+	common.LogVerboseQuiet(fmt.Sprintf("Image %s pushed", fullImage))
+	return nil
+}
+
+func dockerTag(imageID string, imageTag string) bool {
+	cmd := sh.Command(common.DockerBin(), "image", "tag", imageID, imageTag)
+	cmd.Stdout = nil
+	cmd.Stderr = nil
+	if err := cmd.Run(); err != nil {
+		return false
+	}
+
+	return true
+}
+
+func dockerPush(imageTag string) bool {
+	cmd := sh.Command(common.DockerBin(), "image", "push", imageTag)
+	cmd.Stdout = os.Stdout
+	cmd.Stderr = os.Stderr
+	if err := cmd.Run(); err != nil {
+		return false
+	}
+
+	return true
+}
+
+func imageCleanup(appName string, imageRepo string, imageTag string, tag int) {
+	// # keep last two images in place
+	oldTag := tag - 2
+	tenImagesAgoTag := tag - 12
+
+	imagesToRemove := []string{}
+	for oldTag > 0 {
+		imagesToRemove = append(imagesToRemove, fmt.Sprintf("%s:%d", imageRepo, oldTag))
+		oldTag = oldTag - 1
+		if tenImagesAgoTag == oldTag {
+			break
+		}
+	}
+
+	imageIDs, _ := common.ListDanglingImages(appName)
+	imagesToRemove = append(imagesToRemove, imageIDs...)
+	common.RemoveImages(imagesToRemove)
+}

plugins/registry/go.mod

@@ -0,0 +1,12 @@
+module github.com/dokku/dokku/plugins/registry
+
+go 1.15
+
+require (
+  github.com/codegangsta/inject v0.0.0-20150114235600-33e0aa1cb7c0
+  github.com/codeskyblue/go-sh v0.0.0-20190412065543-76bd3d59ff27
+  github.com/dokku/dokku/plugins/common v0.0.0-00010101000000-000000000000
+  github.com/spf13/pflag v1.0.5 // indirect
+)
+
+replace github.com/dokku/dokku/plugins/common => ../common

plugins/registry/go.sum

@@ -0,0 +1,10 @@
+github.com/codegangsta/inject v0.0.0-20150114235600-33e0aa1cb7c0 h1:sDMmm+q/3+BukdIpxwO365v/Rbspp2Nt5XntgQRXq8Q=
+github.com/codegangsta/inject v0.0.0-20150114235600-33e0aa1cb7c0/go.mod h1:4Zcjuz89kmFXt9morQgcfYZAYZ5n8WHjt81YYWIwtTM=
+github.com/codeskyblue/go-sh v0.0.0-20190412065543-76bd3d59ff27 h1:HHUr4P/aKh4quafGxDT9LDasjGdlGkzLbfmmrlng3kA=
+github.com/codeskyblue/go-sh v0.0.0-20190412065543-76bd3d59ff27/go.mod h1:VQx0hjo2oUeQkQUET7wRwradO6f+fN5jzXgB/zROxxE=
+github.com/ryanuber/columnize v1.1.2-0.20190319233515-9e6335e58db3 h1:utdYOikI1XjNtTFGCwSM6OmFJblU4ld4gACoJsbadJg=
+github.com/ryanuber/columnize v1.1.2-0.20190319233515-9e6335e58db3/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
+github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
+github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+golang.org/x/sync v0.0.0-20201207232520-09787c993a3a h1:DcqTD9SDLc+1P/r1EmRBwnVsrOwW+kk2vWf9n+1sGhs=
+golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=

plugins/registry/plugin.toml

@@ -0,0 +1,4 @@
+[plugin]
+description = "dokku core registry plugin"
+version = "0.24.0"
+[plugin.config]

plugins/registry/registry.go

@@ -0,0 +1,16 @@
+package registry
+
+var (
+	// DefaultProperties is a map of all valid network properties with corresponding default property values
+	DefaultProperties = map[string]string{
+		"image-repo":      "",
+		"push-on-release": "false",
+		"server":          "",
+	}
+
+	// GlobalProperties is a map of all valid global network properties
+	GlobalProperties = map[string]bool{
+		"push-on-release": true,
+		"server":          true,
+	}
+)

plugins/registry/report.go

@@ -0,0 +1,84 @@
+package registry
+
+import (
+	"github.com/dokku/dokku/plugins/common"
+)
+
+// ReportSingleApp is an internal function that displays the registry report for one or more apps
+func ReportSingleApp(appName string, format string, infoFlag string) error {
+	if err := common.VerifyAppName(appName); err != nil {
+		return err
+	}
+
+	flags := map[string]common.ReportFunc{
+		"--registry-computed-image-repo":      reportComputedImageRepo,
+		"--registry-image-repo":               reportImageRepo,
+		"--registry-computed-push-on-release": reportComputedPushOnRelease,
+		"--registry-global-push-on-release":   reportGlobalPushOnRelease,
+		"--registry-push-on-release":          reportPushOnRelease,
+		"--registry-computed-server":          reportComputedServer,
+		"--registry-global-server":            reportGlobalServer,
+		"--registry-server":                   reportServer,
+		"--registry-tag-version":              reportTagVersion,
+	}
+
+	flagKeys := []string{}
+	for flagKey := range flags {
+		flagKeys = append(flagKeys, flagKey)
+	}
+
+	trimPrefix := false
+	uppercaseFirstCharacter := true
+	infoFlags := common.CollectReport(appName, infoFlag, flags)
+	return common.ReportSingleApp("registry", appName, infoFlag, infoFlags, flagKeys, format, trimPrefix, uppercaseFirstCharacter)
+}
+
+func reportComputedImageRepo(appName string) string {
+	imageRepo := reportImageRepo(appName)
+	if imageRepo == "" {
+		imageRepo = common.GetAppImageRepo(appName)
+	}
+
+	return imageRepo
+}
+
+func reportImageRepo(appName string) string {
+	return common.PropertyGet("registry", appName, "image-repo")
+}
+
+func reportComputedPushOnRelease(appName string) string {
+	value := reportPushOnRelease(appName)
+	if value == "" {
+		value = reportGlobalPushOnRelease(appName)
+	}
+
+	if value == "" {
+		value = DefaultProperties["push-on-release"]
+	}
+
+	return value
+}
+
+func reportGlobalPushOnRelease(appName string) string {
+	return common.PropertyGet("registry", "--global", "push-on-release")
+}
+
+func reportPushOnRelease(appName string) string {
+	return common.PropertyGet("registry", appName, "push-on-release")
+}
+
+func reportComputedServer(appName string) string {
+	return getRegistryServerForApp(appName)
+}
+
+func reportGlobalServer(appName string) string {
+	return common.PropertyGet("registry", "--global", "server")
+}
+
+func reportServer(appName string) string {
+	return common.PropertyGet("registry", appName, "server")
+}
+
+func reportTagVersion(appName string) string {
+	return common.PropertyGet("registry", appName, "tag-version")
+}

plugins/registry/src/commands/commands.go

@@ -0,0 +1,57 @@
+package main
+
+import (
+	"flag"
+	"fmt"
+	"os"
+	"strconv"
+	"strings"
+
+	"github.com/dokku/dokku/plugins/common"
+)
+
+const (
+	helpHeader = `Usage: dokku registry[:COMMAND]
+
+Manage registry settings for an app
+
+Additional commands:`
+
+	helpContent = `
+    registry:login [--password-stdin] <server> <username> [<password>], Login to a docker registry
+    registry:report [<app>] [<flag>], Displays a registry report for one or more apps
+    registry:set <app> <property> (<value>), Set or clear a registry property for an app
+`
+)
+
+func main() {
+	flag.Usage = usage
+	flag.Parse()
+
+	cmd := flag.Arg(0)
+	switch cmd {
+	case "registry", "registry:help":
+		usage()
+	case "help":
+		command := common.NewShellCmd(fmt.Sprintf("ps -o command= %d", os.Getppid()))
+		command.ShowOutput = false
+		output, err := command.Output()
+
+		if err == nil && strings.Contains(string(output), "--all") {
+			fmt.Println(helpContent)
+		} else {
+			fmt.Print("\n    registry, Manage registry settings for an app\n")
+		}
+	default:
+		dokkuNotImplementExitCode, err := strconv.Atoi(os.Getenv("DOKKU_NOT_IMPLEMENTED_EXIT"))
+		if err != nil {
+			fmt.Println("failed to retrieve DOKKU_NOT_IMPLEMENTED_EXIT environment variable")
+			dokkuNotImplementExitCode = 10
+		}
+		os.Exit(dokkuNotImplementExitCode)
+	}
+}
+
+func usage() {
+	common.CommandUsage(helpHeader, helpContent)
+}

plugins/registry/src/subcommands/subcommands.go

@@ -0,0 +1,58 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"strings"
+
+	"github.com/dokku/dokku/plugins/common"
+	"github.com/dokku/dokku/plugins/registry"
+
+	flag "github.com/spf13/pflag"
+)
+
+// main entrypoint to all subcommands
+func main() {
+	parts := strings.Split(os.Args[0], "/")
+	subcommand := parts[len(parts)-1]
+
+	var err error
+	switch subcommand {
+	case "login":
+		args := flag.NewFlagSet("registry:login", flag.ExitOnError)
+		passwordStdin := args.Bool("password-stdin", false, "--password-stdin: read password from stdin")
+		args.Parse(os.Args[2:])
+		server := args.Arg(0)
+		username := args.Arg(1)
+		password := args.Arg(2)
+		err = registry.CommandLogin(server, username, password, *passwordStdin)
+	case "report":
+		args := flag.NewFlagSet("registry:report", flag.ExitOnError)
+		format := args.String("format", "stdout", "format: [ stdout | json ]")
+		osArgs, infoFlag, flagErr := common.ParseReportArgs("registry", os.Args[2:])
+		if flagErr == nil {
+			args.Parse(osArgs)
+			appName := args.Arg(0)
+			err = registry.CommandReport(appName, *format, infoFlag)
+		}
+	case "set":
+		args := flag.NewFlagSet("registry:set", flag.ExitOnError)
+		global := args.Bool("global", false, "--global: set a global property")
+		args.Parse(os.Args[2:])
+		appName := args.Arg(0)
+		property := args.Arg(1)
+		value := args.Arg(2)
+		if *global {
+			appName = "--global"
+			property = args.Arg(0)
+			value = args.Arg(1)
+		}
+		err = registry.CommandSet(appName, property, value)
+	default:
+		err = fmt.Errorf("Invalid plugin subcommand call: %s", subcommand)
+	}
+
+	if err != nil {
+		common.LogFailWithError(err)
+	}
+}

plugins/registry/src/triggers/triggers.go

@@ -0,0 +1,49 @@
+package main
+
+import (
+	"flag"
+	"fmt"
+	"os"
+	"strings"
+
+	"github.com/dokku/dokku/plugins/common"
+	"github.com/dokku/dokku/plugins/registry"
+)
+
+// main entrypoint to all triggers
+func main() {
+	parts := strings.Split(os.Args[0], "/")
+	trigger := parts[len(parts)-1]
+	flag.Parse()
+
+	var err error
+	switch trigger {
+	case "deployed-app-image-repo":
+		appName := flag.Arg(0)
+		err = registry.TriggerDeployedAppImageRepo(appName)
+	case "deployed-app-image-tag":
+		appName := flag.Arg(0)
+		err = registry.TriggerDeployedAppImageTag(appName)
+	case "deployed-app-repository":
+		appName := flag.Arg(0)
+		err = registry.TriggerDeployedAppRepository(appName)
+	case "install":
+		err = registry.TriggerInstall()
+	case "post-delete":
+		appName := flag.Arg(0)
+		err = registry.TriggerPostDelete(appName)
+	case "post-release-builder":
+		appName := flag.Arg(1)
+		image := flag.Arg(2)
+		err = registry.TriggerPostReleaseBuilder(appName, image)
+	case "report":
+		appName := flag.Arg(0)
+		err = registry.ReportSingleApp(appName, "", "")
+	default:
+		err = fmt.Errorf("Invalid plugin trigger call: %s", trigger)
+	}
+
+	if err != nil {
+		common.LogFailWithError(err)
+	}
+}

plugins/registry/subcommands.go

@@ -0,0 +1,77 @@
+package registry
+
+import (
+	"bytes"
+	"errors"
+	"io/ioutil"
+	"os"
+	"strings"
+
+	"github.com/dokku/dokku/plugins/common"
+)
+
+// CommandLogin logs a user into the specified server
+func CommandLogin(server string, username string, password string, passwordStdin bool) error {
+	if passwordStdin {
+		stdin, err := ioutil.ReadAll(os.Stdin)
+		if err != nil {
+			return err
+		}
+
+		password = strings.TrimSpace(string(stdin))
+	}
+
+	if server == "" {
+		return errors.New("Missing server argument")
+	}
+	if username == "" {
+		return errors.New("Missing username argument")
+	}
+	if password == "" {
+		return errors.New("Missing password argument")
+	}
+
+	command := []string{
+		common.DockerBin(),
+		"login",
+		"--username",
+		username,
+		"--password-stdin",
+		server,
+	}
+
+	buffer := bytes.Buffer{}
+	buffer.Write([]byte(password + "\n"))
+
+	loginCmd := common.NewShellCmd(strings.Join(command, " "))
+	loginCmd.Command.Stdin = &buffer
+	if !loginCmd.Execute() {
+		return errors.New("Failed to log into registry")
+	}
+
+	return nil
+}
+
+// CommandReport displays a registry report for one or more apps
+func CommandReport(appName string, format string, infoFlag string) error {
+	if len(appName) == 0 {
+		apps, err := common.DokkuApps()
+		if err != nil {
+			return err
+		}
+		for _, appName := range apps {
+			if err := ReportSingleApp(appName, format, infoFlag); err != nil {
+				return err
+			}
+		}
+		return nil
+	}
+
+	return ReportSingleApp(appName, format, infoFlag)
+}
+
+// CommandSet set or clear a registry property for an app
+func CommandSet(appName string, property string, value string) error {
+	common.CommandPropertySet("registry", appName, property, value, DefaultProperties, GlobalProperties)
+	return nil
+}

plugins/registry/triggers.go

@@ -0,0 +1,82 @@
+package registry
+
+import (
+	"errors"
+	"fmt"
+	"strings"
+
+	"github.com/dokku/dokku/plugins/common"
+)
+
+// TriggerDeployedAppImageRepo outputs the associated image repo to stdout
+func TriggerDeployedAppImageRepo(appName string) error {
+	imageRepo := common.PropertyGet("registry", appName, "image-repo")
+	if imageRepo == "" {
+		imageRepo = common.GetAppImageRepo(appName)
+	}
+
+	fmt.Println(imageRepo)
+	return nil
+}
+
+// TriggerDeployedAppImageTag outputs the associated image tag to stdout
+func TriggerDeployedAppImageTag(appName string) error {
+	if !isPushEnabled(appName) {
+		return nil
+	}
+
+	tagVersion := common.PropertyGet("registry", appName, "tag-version")
+	if tagVersion == "" {
+		tagVersion = "1"
+	}
+
+	fmt.Println(tagVersion)
+	return nil
+}
+
+// TriggerDeployedAppRepository outputs the associated registry repository to stdout
+func TriggerDeployedAppRepository(appName string) error {
+	fmt.Println(getRegistryServerForApp(appName))
+	return nil
+}
+
+// TriggerInstall runs the install step for the registry plugin
+func TriggerInstall() error {
+	if err := common.PropertySetup("registry"); err != nil {
+		return fmt.Errorf("Unable to install the registry plugin: %s", err.Error())
+	}
+
+	return nil
+}
+
+// TriggerPostDelete destroys the registry property for a given app container
+func TriggerPostDelete(appName string) error {
+	return common.PropertyDestroy("registry", appName)
+}
+
+// TriggerPostReleaseBuilder pushes the image to the remote registry
+func TriggerPostReleaseBuilder(appName string, image string) error {
+	imageID, _ := common.DockerInspect(image, "{{ .Id }}")
+	imageRepo := common.GetAppImageRepo(appName)
+	computedImageRepo := reportComputedImageRepo(appName)
+	newImage := strings.Replace(image, imageRepo+":", computedImageRepo+":", 1)
+
+	if computedImageRepo != imageRepo {
+		if !dockerTag(imageID, newImage) {
+			// TODO: better error
+			return errors.New("Unable to tag image")
+		}
+	}
+
+	if !isPushEnabled(appName) {
+		return nil
+	}
+
+	common.LogInfo1("Pushing image to registry")
+	imageTag, err := incrementTagVersion(appName)
+	if err != nil {
+		return err
+	}
+
+	return pushToRegistry(appName, imageTag, imageID, computedImageRepo)
+}

plugins/scheduler-docker-local/internal-functions

@@ -184,8 +184,14 @@ fn-scheduler-docker-local-retire-images() {
     fi
 
     if echo "$RM_OUTPUT" | grep -q "image has dependent child images"; then
-      dokku_log_warn "Image ${IMAGE_ID} has children images, skipping rm and marking dead"
-      DEAD_IMAGES+=("$IMAGE_ID")
+      TAG_COUNT="$(docker inspect "$IMAGE_ID" --format '{{ json .RepoTags }}' | jq '. | length')"
+      if [[ "$TAG_COUNT" -eq 0 ]]; then
+        dokku_log_warn "Image ${IMAGE_ID} has children images and is untagged, skipping rm and marking dead"
+        DEAD_IMAGES+=("$IMAGE_ID")
+        continue
+      fi
+
+      dokku_log_warn "Image ${IMAGE_ID} has children images and has $TAG_COUNT tags, skipping rm"
       continue
     fi
 

plugins/scheduler-docker-local/pre-deploy

@@ -65,7 +65,7 @@ scheduler-docker-local-pre-deploy-chown-app() {
 scheduler-docker-local-pre-deploy-precheck() {
   declare desc="Outputs the checks messages if necessary"
   declare APP="$1" IMAGE_TAG="$2"
-  local IMAGE=$(get_deploying_app_image_name "$APP" "$IMAGE_TAG")
+  local IMAGE=$(get_app_image_name "$APP" "$IMAGE_TAG")
   local CHECKS_FILE=$(mktemp "/tmp/dokku-${DOKKU_PID}-${FUNCNAME[0]}.XXXXXX")
   trap "rm -rf '$CHECKS_FILE' >/dev/null" RETURN INT TERM EXIT
   copy_from_image "$IMAGE" "CHECKS" "$CHECKS_FILE" 2>/dev/null || true

plugins/scheduler-docker-local/scheduler-deploy

@@ -22,7 +22,6 @@ trigger-scheduler-docker-local-scheduler-deploy() {
   DOKKU_HEROKUISH=false
   DOKKU_CNB=false
   IMAGE=$(get_deploying_app_image_name "$APP" "$IMAGE_TAG")
-  plugn trigger pre-deploy "$APP" "$IMAGE_TAG"
 
   is_image_cnb_based "$IMAGE" && DOKKU_CNB=true
   is_image_herokuish_based "$IMAGE" "$APP" && DOKKU_HEROKUISH=true

tests/unit/registry.bats

@@ -0,0 +1,296 @@
+#!/usr/bin/env bats
+
+load test_helper
+
+setup() {
+  global_setup
+  create_app
+  dokku config:set $TEST_APP DOKKU_WAIT_TO_RETIRE=30
+}
+
+teardown() {
+  destroy_app
+  global_teardown
+}
+
+@test "(registry) registry:help" {
+  run /bin/bash -c "dokku registry"
+  echo "output: $output"
+  echo "status: $status"
+  assert_output_contains "Manage registry settings for an app"
+  help_output="$output"
+
+  run /bin/bash -c "dokku registry:help"
+  echo "output: $output"
+  echo "status: $status"
+  assert_output_contains "Manage registry settings for an app"
+  assert_output "$help_output"
+}
+
+@test "(registry) registry:login" {
+  if [[ -z "$DOCKERHUB_USERNAME" ]] || [[ -z "$DOCKERHUB_TOKEN" ]]; then
+    skip "skipping due to missing docker.io credentials DOCKERHUB_USERNAME:DOCKERHUB_TOKEN"
+  fi
+
+  run /bin/bash -c "dokku registry:login docker.io $DOCKERHUB_USERNAME $DOCKERHUB_TOKEN"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+  assert_output_contains "Login Succeeded"
+
+  run /bin/bash -c "echo $DOCKERHUB_TOKEN | dokku registry:login docker.io --password-stdin $DOCKERHUB_USERNAME"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+  assert_output_contains "Login Succeeded"
+}
+
+@test "(registry) registry:set server" {
+  run /bin/bash -c "dokku registry:set --global server ghcr.io"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "dokku registry:report $TEST_APP --registry-server"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+  assert_output_not_exists
+
+  run /bin/bash -c "dokku registry:report $TEST_APP --registry-global-server"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+  assert_output "ghcr.io"
+
+  run /bin/bash -c "dokku registry:report $TEST_APP --registry-computed-server"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+  assert_output "ghcr.io/"
+
+  run /bin/bash -c "dokku registry:set --global server docker.io"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "dokku registry:report $TEST_APP --registry-server"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+  assert_output_not_exists
+
+  run /bin/bash -c "dokku registry:report $TEST_APP --registry-global-server"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+  assert_output "docker.io"
+
+  run /bin/bash -c "dokku registry:report $TEST_APP --registry-computed-server"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+  assert_output_not_exists
+
+  run /bin/bash -c "dokku registry:set $TEST_APP server docker.io"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "dokku registry:report $TEST_APP --registry-computed-server"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+  assert_output_not_exists
+
+  run /bin/bash -c "dokku registry:report $TEST_APP --registry-global-server"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+  assert_output "docker.io"
+
+  run /bin/bash -c "dokku registry:report $TEST_APP --registry-server"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+  assert_output "docker.io"
+
+  run /bin/bash -c "dokku registry:set $TEST_APP server ghcr.io"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "dokku registry:report $TEST_APP --registry-computed-server"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+  assert_output "ghcr.io/"
+
+  run /bin/bash -c "dokku registry:report $TEST_APP --registry-global-server"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+  assert_output "docker.io"
+
+  run /bin/bash -c "dokku registry:report $TEST_APP --registry-server"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+  assert_output "ghcr.io"
+}
+
+@test "(registry) registry:set image-repo" {
+  run /bin/bash -c "docker images"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "dokku registry:set $TEST_APP image-repo heroku/$TEST_APP"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run deploy_app
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "docker inspect heroku/$TEST_APP:latest"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "docker images"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+}
+
+@test "(registry) registry:set push-on-release" {
+  if [[ -z "$DOCKERHUB_USERNAME" ]] || [[ -z "$DOCKERHUB_TOKEN" ]]; then
+    skip "skipping due to missing docker.io credentials DOCKERHUB_USERNAME:DOCKERHUB_TOKEN"
+  fi
+
+  run /bin/bash -c "dokku registry:set $TEST_APP push-on-release true"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "dokku registry:set $TEST_APP image-repo dokku/test-app"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run deploy_app
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  sleep 60
+
+  run /bin/bash -c "dokku ps:retire"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "docker container ls -a"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "docker image ls"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "dokku ps:rebuild $TEST_APP"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  sleep 60
+
+  run /bin/bash -c "dokku ps:retire"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "docker container ls -a"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "docker image ls"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "dokku config:set $TEST_APP key=VALUE"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  sleep 60
+
+  run /bin/bash -c "dokku ps:retire"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "docker container ls -a"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "docker image ls"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "dokku ps:rebuild $TEST_APP"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  sleep 60
+
+  run /bin/bash -c "dokku ps:retire"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "dokku ps:retire"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "docker container ls -a"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "docker image ls"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+
+  run /bin/bash -c "docker image inspect dokku/test-app:1"
+  echo "output: $output"
+  echo "status: $status"
+  assert_failure
+
+  run /bin/bash -c "docker image inspect dokku/test-app:2"
+  echo "output: $output"
+  echo "status: $status"
+  assert_failure
+
+  run /bin/bash -c "docker image inspect dokku/test-app:3"
+  echo "output: $output"
+  echo "status: $status"
+  assert_failure
+
+  run /bin/bash -c "docker image inspect dokku/test-app:4"
+  echo "output: $output"
+  echo "status: $status"
+  assert_success
+}