Merge pull request #4209 from dokku/cnb-support

Add experimental support for Cloud Native Buildpacks (CNB)

docs/deployment/application-deployment.md

@@ -102,7 +102,7 @@ Once the deploy is complete, the application's web URL will be generated as abov
 
 Dokku supports deploying applications in a few ways:
 
-- [Heroku buildpacks](https://devcenter.heroku.com/articles/buildpacks) via [Herokuish](https://github.com/gliderlabs/herokuish#buildpacks): See the [buildpacks documentation](/docs/deployment/methods/buildpacks.md) to learn about the different ways to specify a buildpack.
+- [Heroku buildpacks](https://devcenter.heroku.com/articles/buildpacks) via [Herokuish](https://github.com/gliderlabs/herokuish#buildpacks): See the [herokuish buildpacks documentation](/docs/deployment/methods/herokuish-buildpacks.md) to learn about the different ways to specify a buildpack.
 - [Dockerfile](https://docs.docker.com/reference/builder/): See the [dockerfile documentation](/docs/deployment/methods/dockerfiles.md) to learn about the different ways to configure Dockerfile-based deploys.
 - [Docker Image](https://docs.docker.com/get-started/overview/#docker-objects): See the [docker image documentation](/docs/deployment/methods/images.md) to learn about how to deploy a Docker Image.
 
@@ -211,7 +211,7 @@ See the [image tagging documentation](/docs/deployment/methods/images.md) for mo
 
 ## Specifying a custom buildpack
 
-See the [buildpack documentation](/docs/deployment/methods/buildpacks.md) for more information on how to specify a set of custom buildpacks for your application.
+See the [herokuish buildpack documentation](/docs/deployment/methods/herokuish-buildpacks.md) for more information on how to specify a set of custom buildpacks for your application.
 
 ## Removing a deployed app
 

docs/deployment/methods/cloud-native-buildpacks.md

@@ -0,0 +1,35 @@
+# Cloud Native Buildpacks (Experimental)
+
+> New as of 0.22.0
+
+Cloud Native Buildpacks are an evolution over the Buildpacks technology provided by the Herokuish builder. See the [herokuish buildpacks documentation](/docs/deployment/methods/herokuish.md) for more information on how to clear buildpack build cache for an application.
+
+> Warning: This functionality uses the `pack` cli from the [Cloud Native Buildpacks](https://buildpacks.io) project to build apps. As the integration is experimental in Dokku, it is likely to change over time.
+
+## Usage
+
+To use this builder instead of either `Dockerfile` or `herokuish`, you must set the `DOKKU_CNB_EXPERIMENTAL` environment variable for your app to `1`.
+
+```shell
+dokku config:set --no-restart node-js-app DOKKU_CNB_EXPERIMENTAL=1
+```
+
+The `pack` cli tool is not included by default with Dokku or as a dependency. It must also be installed as shown on [this page](https://buildpacks.io/docs/tools/pack/).
+
+Builds will proceed with the `pack` cli for the app from then on.
+
+## Caveats
+
+As this functionality is highly experimental, there are a number of caveats. Please note that not all issuesare listed below.
+
+- The `heroku/buildpacks:latest` CNB builder is currently enforced. Specifying specific builders is not currently possible.
+  - A future release will allow modifying the chosen builder.
+- Specifying specific buildpacks is not currently possible.
+  - A future release will add support for the `buildpacks` plugin.
+- There is currently no way to specify extra arguments for `pack` cli invocations.
+  - A future release will add support for injecting extra arguments during the build process.
+- The default process type is `web`.
+- Build cache is stored in Docker volumes instead of on disk. As such, `repo:purge-cache` currently has no effect.
+  - A future version will add integration with the `repo` plugin.
+- `pack` is not currently included with Dokku, nor is it added as a package dependency.
+  - A future version will include it as a package dependency.

docs/deployment/methods/git.md

@@ -8,7 +8,11 @@ git:report [<app>] [<flag>]              # Displays a git report for one or more
 git:set <app> <key> (<value>)            # Set or clear a git property for an app
 ```
 
-Git-based deployment has been the traditional method of deploying applications in Dokku. As of v0.12.0, Dokku introduces a few ways to customize the experience of deploying via `git push`. A Git-based deployment currently supports building applications via both [Buildpack](/docs/deployment/methods/buildpacks.md) and [Dockerfile](/docs/deployment/methods/dockerfiles.md). 
+Git-based deployment has been the traditional method of deploying applications in Dokku. As of v0.12.0, Dokku introduces a few ways to customize the experience of deploying via `git push`. A Git-based deployment currently supports building applications via:
+
+- [Cloud Native Buildpacks](/docs/deployment/methods/cloud-native-buildpacks.md)
+- [Herokuish Buildpack](/docs/deployment/methods/herokuish-buildpacks.md)
+- [Dockerfiles](/docs/deployment/methods/dockerfiles.md)
 
 ## Usage
 

docs/deployment/methods/herokuish-buildpacks.md

@@ -1,4 +1,4 @@
-# Buildpack Deployment
+# Herokuish Buildpack Deployment
 
 > Subcommands new as of 0.15.0
 

docs/development/plugin-creation.md

@@ -97,12 +97,12 @@ If a plugin depends on a specific command-line tool, check whether that tool exi
 ```shell
 # `command -v` example
 if ! command -v "nginx" &>/dev/null; then
-  log-fail "Missing nginx, install it"
+  dokku_log_fail "Missing nginx, install it"
 fi
 
 # `which` example
 if ! which nginx >/dev/null 2>&1; then
-  log-fail "Missing nginx, install it"
+  dokku_log_fail "Missing nginx, install it"
 fi
 ```
 

docs/development/plugin-triggers.md

@@ -1015,6 +1015,24 @@ set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
 # TODO
 ```
 
+### `post-build-cnb`
+
+> Warning: The cnb plugin trigger apis are under development and may change
+> between minor releases until the 1.0 release.
+
+- Description: Allows you to run commands after the build image is create for a given app. Only applies to apps using cnb.
+- Invoked by: `internal function dokku_build() (build phase)`
+- Arguments: `$APP` `$SOURCECODE_WORK_DIR`
+- Example:
+
+```shell
+#!/usr/bin/env bash
+
+set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
+
+# TODO
+```
+
 ### `post-build-dockerfile`
 
 - Description: Allows you to run commands after the build image is create for a given app. Only applies to apps using a dockerfile.
@@ -1231,6 +1249,29 @@ verify_app_name "$APP"
 # TODO
 ```
 
+### `post-release-cnb`
+
+> Warning: The cnb plugin trigger apis are under development and may change
+> between minor releases until the 1.0 release.
+
+> Warning: Image mutation in this trigger may result in an invalid run state, and is heavily discouraged.
+
+- Description: Allows you to run commands after environment variables are set for the release step of the deploy. Only applies to apps using cnb.
+- Invoked by: `internal function dokku_release() (release phase)`
+- Arguments: `$APP $IMAGE_TAG`
+- Example:
+
+```shell
+#!/usr/bin/env bash
+
+set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
+source "$PLUGIN_CORE_AVAILABLE_PATH/common/functions"
+APP="$1"; IMAGE_TAG="$2"; IMAGE=$(get_app_image_name $APP $IMAGE_TAG)
+verify_app_name "$APP"
+
+# TODO
+```
+
 ### `post-release-dockerfile`
 
 > Warning: Image mutation in this trigger may result in an invalid run state, and is heavily discouraged.
@@ -1284,6 +1325,24 @@ set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
 # TODO
 ```
 
+### `pre-build-cnb`
+
+> Warning: The cnb plugin trigger apis are under development and may change
+> between minor releases until the 1.0 release.
+
+- Description: Allows you to run commands before the build image is created for a given app. For instance, this can be useful to add env vars to your container. Only applies to apps using cnb.
+- Invoked by: `internal function dokku_build() (build phase)`
+- Arguments: `$APP` `$SOURCECODE_WORK_DIR`
+- Example:
+
+```shell
+#!/usr/bin/env bash
+
+set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
+
+# TODO
+```
+
 ### `pre-build-dockerfile`
 
 - Description: Allows you to run commands before the build image is created for a given app. For instance, this can be useful to add env vars to your container. Only applies to apps using a dockerfile.
@@ -1425,6 +1484,27 @@ DOCKER_COMMIT_LABEL_ARGS=("--change" "LABEL org.label-schema.schema-version=1.0"
 docker commit "${DOCKER_COMMIT_LABEL_ARGS[@]}" $CID $IMAGE >/dev/null
 ```
 
+
+### `pre-release-cnb`
+
+> Warning: The cnb plugin trigger apis are under development and may change
+> between minor releases until the 1.0 release.
+
+- Description: Allows you to run commands before environment variables are set for the release step of the deploy. Only applies to apps using cnb.
+- Invoked by: `internal function dokku_release() (release phase)`
+- Arguments: `$APP $IMAGE_TAG`
+- Example:
+
+```shell
+#!/usr/bin/env bash
+
+set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
+source "$PLUGIN_CORE_AVAILABLE_PATH/common/functions"
+APP="$1"; IMAGE_TAG="$2";
+
+# TODO
+```
+
 ### `pre-release-dockerfile`
 
 - Description: Allows you to run commands before environment variables are set for the release step of the deploy. Only applies to apps using a dockerfile.

docs/template.html

@@ -128,7 +128,8 @@
 
             <a href="#" class="list-group-item disabled">Deployment Methods</a>
 
-            <a href="/{{NAME}}/deployment/methods/buildpacks/" class="list-group-item">Buildpack Deployment</a>
+            <a href="/{{NAME}}/deployment/methods/cloud-native-buildpacks/" class="list-group-item">Cloud Native Buildpacks</a>
+            <a href="/{{NAME}}/deployment/methods/herokuish-buildpacks/" class="list-group-item">Herokuish Buildpacks</a>
             <a href="/{{NAME}}/deployment/methods/dockerfiles/" class="list-group-item">Dockerfile Deployment</a>
             <a href="/{{NAME}}/deployment/methods/images/" class="list-group-item">Docker Image Deployment</a>
             <a href="/{{NAME}}/deployment/methods/git/" class="list-group-item">Git Deployment</a>

docs/viewdocs.json

@@ -10,7 +10,8 @@
     "process-management": "deployment/process-management/",
     "remote-commands": "deployment/remote-commands/",
 
-    "deployment/buildpacks": "deployment/methods/buildpacks/",
+    "deployment/buildpacks": "deployment/methods/herokuish-buildpacks/",
+    "deployment/methods/buildpacks": "deployment/methods/herokuish-buildpacks/",
     "deployment/dockerfiles": "deployment/methods/dockerfiles/",
     "deployment/images": "deployment/methods/images/",
     "configuration-management": "configuration/environment-variables/",

plugins/20_events/post-build-cnb

@@ -0,0 +1 @@
+hook

plugins/20_events/post-release-cnb

@@ -0,0 +1 @@
+hook

plugins/20_events/pre-build-cnb

@@ -0,0 +1 @@
+hook

plugins/20_events/pre-release-cnb

@@ -0,0 +1 @@
+hook

plugins/builder-cnb/builder-build

@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+set -eo pipefail
+[[ $DOKKU_TRACE ]] && set -x
+source "$PLUGIN_CORE_AVAILABLE_PATH/common/functions"
+source "$PLUGIN_AVAILABLE_PATH/config/functions"
+
+trigger-builder-cnb-builder-build() {
+  declare desc="builder-cnb builder-build plugin trigger"
+  declare trigger="builder-build"
+  declare BUILDER_TYPE="$1" APP="$2" SOURCECODE_WORK_DIR="$3"
+
+  if [[ "$BUILDER_TYPE" != "cnb" ]]; then
+    return
+  fi
+
+  if ! command -v "pack" &>/dev/null; then
+    dokku_log_fail "Missing pack, install it"
+  fi
+
+  local IMAGE=$(get_app_image_name "$APP")
+
+  pushd "$SOURCECODE_WORK_DIR" &>/dev/null
+
+  local TMP_FILE=$(mktemp "/tmp/dokku-${DOKKU_PID}-${FUNCNAME[0]}.XXXXXX")
+  trap "rm -rf '$TMP_FILE' >/dev/null" RETURN INT TERM
+
+  config_export app "$APP" --format envfile --merged > "$TMP_FILE"
+
+  plugn trigger pre-build-cnb "$APP" "$SOURCECODE_WORK_DIR"
+
+  pack build "$IMAGE" --builder heroku/buildpacks --path "$SOURCECODE_WORK_DIR" --default-process web
+
+  echo "FROM $IMAGE" | suppress_output "$DOCKER_BIN" image build --label=com.dokku.image-stage=build --label=org.label-schema.schema-version=1.0 --label=org.label-schema.vendor=dokku --label=com.dokku.app-name=$APP --label=dokku -t "$IMAGE" -
+
+  plugn trigger post-build-cnb "$APP" "$SOURCECODE_WORK_DIR"
+}
+
+trigger-builder-cnb-builder-build "$@"

plugins/builder-cnb/builder-release

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+set -eo pipefail
+[[ $DOKKU_TRACE ]] && set -x
+source "$PLUGIN_CORE_AVAILABLE_PATH/common/functions"
+
+trigger-builder-cnb-builder-release() {
+  declare desc="builder-cnb builder-release plugin trigger"
+  declare trigger="builder-release"
+  declare BUILDER_TYPE="$1" APP="$2" IMAGE_TAG="$3"
+
+  if [[ "$BUILDER_TYPE" != "cnb" ]]; then
+    return
+  fi
+
+  plugn trigger pre-release-cnb "$APP" "$IMAGE_TAG"
+
+  local IMAGE=$(get_app_image_name "$APP" "$IMAGE_TAG")
+  echo "FROM $IMAGE" | suppress_output "$DOCKER_BIN" image build --label=com.dokku.image-stage=release --label=com.dokku.app-name=$APP --label=org.label-schema.schema-version=1.0 --label=org.label-schema.vendor=dokku --label=dokku -t "$IMAGE" -
+  plugn trigger post-release-cnb "$APP" "$IMAGE_TAG"
+}
+
+trigger-builder-cnb-builder-release "$@"

plugins/builder-cnb/plugin.toml

@@ -0,0 +1,4 @@
+[plugin]
+description = "dokku core builder-cnb plugin"
+version = "0.21.4"
+[plugin.config]

plugins/builder-dockerfile/builder-release

@@ -12,7 +12,6 @@ trigger-builder-dockerfile-builder-release() {
     return
   fi
 
-  # buildstep plugins don't necessarily make sense for dockerfiles. call the new breed!!!
   plugn trigger pre-release-dockerfile "$APP" "$IMAGE_TAG"
 
   local IMAGE=$(get_app_image_name "$APP" "$IMAGE_TAG")

plugins/common/functions

@@ -293,20 +293,41 @@ get_running_image_tag() {
   echo "$RUNNING_IMAGE_TAG"
 }
 
+is_image_cnb_based() {
+  declare desc="returns true if app image is based on cnb"
+  declare IMAGE="$1"
+
+  if [[ -z "$IMAGE" ]]; then
+    return 1
+  fi
+
+  CNB_STACK_ID="$("$DOCKER_BIN" image inspect --format '{{ index .Config.Labels "io.buildpacks.stack.id" }}' "$IMAGE")"
+  if [[ -n "$CNB_STACK_ID" ]]; then
+    return 0
+  fi
+
+  return 1
+}
+
 is_image_herokuish_based() {
-  declare desc="returns true if app image is based on herokuish"
+  declare desc="returns true if app image is based on herokuish or a buildpack-like env"
   declare IMAGE="$1" APP="$2"
   local DOKKU_APP_USER
-  if [[ -n "$APP" ]]; then
-    DOKKU_APP_USER=$(config_get "$APP" DOKKU_APP_USER || true)
-  fi
-  DOKKU_APP_USER=${DOKKU_APP_USER:="herokuishuser"}
   local USER_VALUE
 
   if [[ -z "$IMAGE" ]]; then
     return 1
   fi
 
+  if is_image_cnb_based "$IMAGE"; then
+    return 0
+  fi
+
+  if [[ -n "$APP" ]]; then
+    DOKKU_APP_USER=$(config_get "$APP" DOKKU_APP_USER || true)
+  fi
+  DOKKU_APP_USER=${DOKKU_APP_USER:="herokuishuser"}
+
   # due to how the build process works, all herokuish images have the Environment variable USER=$DOKKU_APP_USER
   USER_VALUE="$("$DOCKER_BIN" image inspect --format '{{range .Config.Env}}{{if eq . "USER='"$DOKKU_APP_USER"'"}}{{println .}}{{end}}{{end}}' "$IMAGE")"
   [[ "$USER_VALUE" == "" ]] && return 1
@@ -566,6 +587,11 @@ dokku_release() {
   declare desc="release phase"
   declare APP="$1" IMAGE_SOURCE_TYPE="$2" IMAGE_TAG="$3"
 
+  local IMAGE=$(get_app_image_name "$APP" "$IMAGE_TAG")
+  if is_image_cnb_based "$IMAGE"; then
+    IMAGE_SOURCE_TYPE="cnb"
+  fi
+
   plugn trigger builder-release "$IMAGE_SOURCE_TYPE" "$APP" "$IMAGE_TAG"
 }
 
@@ -630,9 +656,17 @@ dokku_receive() {
   local IMAGE=$(get_app_image_name "$APP")
   local IMAGE_SOURCE_TYPE="$2"
   local TMP_WORK_DIR="$3"
-  local DOKKU_SKIP_CLEANUP="$(config_get "$APP" DOKKU_SKIP_CLEANUP || true)"
+
   docker_cleanup "$APP"
-  dokku_log_info1 "Building $APP from $IMAGE_SOURCE_TYPE..."
+
+  local DOKKU_CNB_EXPERIMENTAL="$(config_get "$APP" DOKKU_CNB_EXPERIMENTAL || true)"
+  if [[ "$DOKKU_CNB_EXPERIMENTAL" == "1" ]]; then
+    IMAGE_SOURCE_TYPE="cnb"
+    dokku_log_info1 "Building $APP from $IMAGE_SOURCE_TYPE (experimental)..."
+  else
+    dokku_log_info1 "Building $APP from $IMAGE_SOURCE_TYPE..."
+  fi
+
   DOKKU_QUIET_OUTPUT=1 config_set --no-restart "$APP" DOKKU_APP_TYPE="$IMAGE_SOURCE_TYPE"
   dokku_build "$APP" "$IMAGE_SOURCE_TYPE" "$TMP_WORK_DIR"
   plugn trigger release-and-deploy "$APP"

plugins/scheduler-docker-local/scheduler-enter

@@ -72,7 +72,8 @@ trigger-scheduler-docker-local-scheduler-enter() {
   local EXEC_CMD=""
   has_tty && local DOKKU_RUN_OPTS+=" -i -t"
   local IMAGE="$("$DOCKER_BIN" container inspect --format '{{ .Config.Image }}' "$CONTAINER_ID")"
-  is_image_herokuish_based "$IMAGE" "$APP" && local EXEC_CMD="/exec"
+  is_image_herokuish_based "$IMAGE" "$APP" && EXEC_CMD="/exec"
+  is_image_cnb_based "$IMAGE" && EXEC_CMD="" && DOKKU_RUN_OPTS+=" -w /app"
   # shellcheck disable=SC2086
   "$DOCKER_BIN" container exec $DOKKU_RUN_OPTS "$CONTAINER_ID" $EXEC_CMD "${@:-$DOKKU_APP_SHELL}"
 }