diff --git a/docs/processes/entering-containers.md b/docs/processes/entering-containers.md new file mode 100644 index 000000000..5d8f7d2d5 --- /dev/null +++ b/docs/processes/entering-containers.md @@ -0,0 +1,29 @@ +# Entering containers + +> New as of 0.4.0 + +``` +enter [ || --container-id ] # Connect to a specific app container +``` + +## Usage + +The `enter` command can be used to enter a running container. The following variations of the command exist: + +```shell +dokku enter node-js-app web +dokku enter node-js-app web.1 +dokku enter node-js-app --container-id ID +``` + +Additionally, the `enter` command can be executed with no ``. If only a single `` is defined in the app's Procfile, executing `enter` will drop the terminal into the only running container. This behavior is not supported when specifying a custom command; as described below. + +By default, it runs a `/bin/bash`, but can also be used to run a custom command: + +```shell +# just echo hi +dokku enter node-js-app web echo hi + +# run a long-running command, as one might for a cron task +dokku enter node-js-app web python script/background-worker.py +``` diff --git a/docs/processes/one-off-tasks.md b/docs/processes/one-off-tasks.md new file mode 100644 index 000000000..895d1d8a2 --- /dev/null +++ b/docs/processes/one-off-tasks.md @@ -0,0 +1,62 @@ +# One-off Tasks + +``` +run [ --env KEY=VALUE | -e KEY=VALUE ] # Run a command in a new container using the current application image +``` + +Sometimes it is necessary to run a one-off command under an application. Dokku makes it easy to run a fresh container via the `run` command. + +## Usage + +The `run` command can be used to run a one-off process for a specific command. This will start a new container and run the desired command within that container. Note that this container will be stay around even after command completes. The container will be the same container as was used to start the currently deployed application. + +```shell +# runs `ls -lah` in the `/app` directory of the application `node-js-app` +dokku run node-js-app ls -lah + +# optionally, run can be passed custom environment variables +dokku run --env "NODE_ENV=development" --env "PATH=/custom/path" node-js-app npm run mytask +``` + +The `run` command can also be used to run a command defined in the app `Procfile`: + +``` +console: bundle exec racksh +``` + +```shell +# runs `bundle exec racksh` in the `/app` directory of the application `my-app` +dokku run my-app console +``` + +If the container running the command should be removed after exit, the `--rm-container` or `--rm` global flags can be specified to remove the containers automatically: + +```shell +dokku --rm-container run node-js-app ls -lah +dokku --rm run node-js-app ls -lah +``` + +Alternatively, a global property can be set to always remove `run` containers. + +```shell +# don't keep `run` containers around +dokku config:set --global DOKKU_RM_CONTAINER=1 + +# revert the above setting and keep containers around +dokku config:unset --global DOKKU_RM_CONTAINER +``` + +Containers may have specific labels attached. In order to avoid issues with dokku internals, do not use any labels beginning with either `com.dokku` or `org.label-schema`. + +```shell +dokku --label=com.example.test-label=value run node-js-app ls -lah +``` + +Finally, a container can be run in "detached" mode via the `--detach` Dokku flag. Running a process in detached mode will immediately return a `CONTAINER_ID`. It is up to the user to then further manage this container in whatever manner they see fit, as Dokku will *not* automatically terminate the container. + +```shell +dokku --detach run node-js-app ls -lah +# returns the ID of the new container +``` + +> Note that the `--rm-container` or `--rm` flags cannot be used when running containers in detached mode, and attempting to do so will result in the `--detach` flag being ignored. diff --git a/docs/deployment/process-management.md b/docs/processes/process-management.md similarity index 100% rename from docs/deployment/process-management.md rename to docs/processes/process-management.md diff --git a/docs/deployment/one-off-processes.md b/docs/processes/scheduled-cron-tasks.md similarity index 50% rename from docs/deployment/one-off-processes.md rename to docs/processes/scheduled-cron-tasks.md index 45d9895c6..d9f051ed1 100644 --- a/docs/deployment/one-off-processes.md +++ b/docs/processes/scheduled-cron-tasks.md @@ -1,69 +1,110 @@ -# One-off Processes and Cron +# Scheduled Cron T +asks -Sometimes you need to either inspect running containers or run a one-off command under an application. In those cases, Dokku makes it easy to either connect to a running container or run a fresh container. - -## Run a command in an app environment +> New as of 0.23.0 ``` -run [ --env KEY=VALUE | -e KEY=VALUE ] # Run a command in a new container using the current application image +cron:list # List scheduled cron tasks for an app +cron:report [] [] # Display report about an app ``` -The `run` command can be used to run a one-off process for a specific command. This will start a new container and run the desired command within that container. Note that this container will be stay around even after command completes. The container will be the same container as was used to start the currently deployed application. +## Usage + +### Dokku Managed Cron + +Dokku automates scheduled `dokku run` commands via it's `app.json` cron integration. + +#### Specifying commands + +The `app.json` file for a given app can define a special `cron` key that contains a list of commands to run on given schedules. The following is a simple example `app.json` that simply runs the command `true` once a day: + +``` +{ + "cron": [ + { + "command": "true", + "schedule": "@daily" + } + ] +} +``` + +A cron entry takes the following properties: + +- `command`: A command to be run within the built app image. Specified commands can also be `Procfile` entries. +- `schedule`: A [cron-compatible](https://en.wikipedia.org/wiki/Cron#Overview) scheduling definition upon which to run the command. Seconds are generally not supported. + +Zero or more cron commands can be specified per app. Cron entries are validated after the build artifact is created but before the app is deployed, and the cron schedule is updated during the post-deploy phase. + +#### Task Environment + +When running scheduled cron tasks, there are a few items to be aware of: + +- Scheduled cron tasks are performed within the app environment available at runtime. If the app image does not exist, the command may fail to execute. +- Schedules are performed on the hosting server's timezone, which is typically UTC. +- At this time, only the `PATH` and `SHELL` environment variables are specified in the cron template. +- Each scheduled task is executed within a one-off `run` container, and thus inherit any docker-options specified for `run` containers.Resources are never shared between scheduled tasks. +- Tasks are always executed using the `--rm` flag, and are never executed in detached mode. +- Scheduled cron tasks are supported on a per-scheduler basis, and are currently only implemented by the `docker-local` scheduler. +- Tasks for _all_ apps managed by the `docker-local` scheduler are written to a single crontab file owned by the `dokku` user. The `dokku` user's crontab should be considered reserved for this purpose. + +#### Listing Cron tasks + +Cron tasks for an app can be listed via the `cron:list` command. This command takes an `app` argument. ```shell -# runs `ls -lah` in the `/app` directory of the application `node-js-app` -dokku run node-js-app ls -lah - -# optionally, run can be passed custom environment variables -dokku run --env "NODE_ENV=development" --env "PATH=/custom/path" node-js-app npm run mytask +dokku cron:list node-js-app ``` -The `run` command can also be used to run a command defined in your Procfile: +``` +ID Schedule Command +cGhwPT09cGhwIHRlc3QucGhwPT09QGRhaWx5 @daily node index.js +cGhwPT09dHJ1ZT09PSogKiAqICogKg== * * * * * true +``` -``` -console: bundle exec racksh -``` +#### Displaying reports for an app + +You can get a report about the cron configuration for apps using the `cron:report` command: ```shell -# runs `bundle exec racksh` in the `/app` directory of the application `my-app` -dokku run my-app console +dokku cron:report node-js-app ``` -If you want to remove the container after a command has started, you can run the following command: +``` +=====> node-js-app cron information + Cron task count: 2 +=====> python-sample cron information + Cron task count: 0 +=====> ruby-sample cron information + Cron task count: 10 +``` + +You can run the command for a specific app also. ```shell -# don't keep `run` containers around -dokku config:set --global DOKKU_RM_CONTAINER=1 - -# revert the above setting and keep containers around -dokku config:unset --global DOKKU_RM_CONTAINER +dokku cron:report node-js-app ``` -You may also use the `--rm-container` or `--rm` Dokku flags to remove the containers automatically: +``` +=====> node-js-app cron information + Cron task count: 2 +``` + +You can pass flags which will output only the value of the specific information you want. For example: ```shell -dokku --rm-container run node-js-app ls -lah -dokku --rm run node-js-app ls -lah +dokku cron:report node-js-app --cron-task-count ``` -Containers may have specific labels attached. In order to avoid issues with dokku internals, do not use any labels beginning with either `com.dokku` or `org.label-schema`. +### Self Managed Cron -```shell -dokku --label=com.example.test-label=value run node-js-app ls -lah -``` +> Warning: Self-managed cron tasks should be considered advanced usage. While the instructions are available, users are highly encouraged to use the built-in scheduled cron task support unless absolutely necessary. -Finally, you may wish to run a container in "detached" mode via the `--detach` Dokku flag. Running a process in detached mode will immediately return a `CONTAINER_ID`. It is up to the user to then further manage this container in whatever manner they see fit, as Dokku will *not* automatically terminate the container. +Some installations may require more fine-grained control over cron usage. The following are advanced instructions for configuring cron. -```shell -dokku --detach run node-js-app ls -lah -# returns the ID of the new container -``` +#### Using `run` for cron tasks -> Note that you may not use the `--rm-container` or `--rm` flags when running containers in detached mode, and attempting to do so will result in the `--detach` flag being ignored. - -### Using `run` for cron tasks - -You can always use a one-off container to run an application task: +You can always use a one-off container to run an app task: ```shell dokku --rm run node-js-app some-command @@ -72,35 +113,8 @@ dokku --rm-container run node-js-app some-command For tasks that should not be interrupted, run is the *preferred* method of handling cron tasks, as the container will continue running even during a deploy or scaling event. The trade-off is that there will be an increase in memory usage if there are multiple concurrent tasks running. -## Entering existing containers -> New as of 0.4.0 - -``` -enter [ || --container-id ] # Connect to a specific app container -``` - -The `enter` command can be used to enter a running container. The following variations of the command exist: - -```shell -dokku enter node-js-app web -dokku enter node-js-app web.1 -dokku enter node-js-app --container-id ID -``` - -Additionally, you can run `enter` with no ``. If only a single `` is defined in your app, you will be dropped into the only running container. This behavior is not supported when specifying a custom command; as described below. - -By default, it runs a `/bin/bash`, but can also be used to run a custom command: - -```shell -# just echo hi -dokku enter node-js-app web echo hi - -# run a long-running command, as one might for a cron task -dokku enter node-js-app web python script/background-worker.py -``` - -### Using `enter` for cron tasks +#### Using `enter` for cron tasks Your Procfile can have the following entry: @@ -124,7 +138,7 @@ Note that you can also run multiple commands at the same time to reduce memory u For tasks that will properly resume, you *should* use the above method, as running tasks will be interrupted during deploys and scaling events, and subsequent commands will always run with the latest container. Note that if you scale the cron container down, this may interrupt proper running of the task. -## General cron recommendations +#### General cron recommendations Regularly scheduled tasks can be a bit of a pain with Dokku. The following are general recommendations to follow to help ensure successful task runs. @@ -138,14 +152,14 @@ Regularly scheduled tasks can be a bit of a pain with Dokku. The following are g - Run tasks at the lowest traffic times if possible. - Use cron to *trigger* jobs, not run them. Use a real queuing system such as rabbitmq to actually process jobs. - Try to keep tasks quiet so that mails only send on errors. -- Do not silence standard error or standard out. If you silence the former, you will miss failures. Silencing the latter means you should actually make application changes to handle log levels. +- Do not silence standard error or standard out. If you silence the former, you will miss failures. Silencing the latter means you should actually make app changes to handle log levels. - Use a service such as [Dead Man's Snitch](https://deadmanssnitch.com) to verify that cron tasks completed successfully. - Add lots of comments to your cronfile, including what a task is doing, so that you don't spend time deciphering the file later. - Place your cronfiles in a pattern such as `/etc/cron.d/APP`. - Do not use non-ASCII characters in your cronfile names. cron is finicky. - Remember to have trailing newlines in your cronfile! cron is finicky. -The following is a sample cronfile that you can use for your applications: +The following is a sample cronfile that you can use for your apps: ```cron # server cron jobs diff --git a/docs/template.html b/docs/template.html index 415c9c138..8f43984cf 100644 --- a/docs/template.html +++ b/docs/template.html @@ -124,8 +124,6 @@ Application Management Application Logs Remote Commands - One-Off Processes/Cron - Process Scaling User Management Zero Downtime Deploy Checks @@ -145,6 +143,13 @@ Nginx Configuration SSL Configuration + Process Management + + Entering Containers + One-Off Tasks + Process Scaling + Scheduled Cron Tasks + Network Management DNS Configuration diff --git a/docs/viewdocs.json b/docs/viewdocs.json index 24cb978ae..3c9c2759d 100644 --- a/docs/viewdocs.json +++ b/docs/viewdocs.json @@ -7,9 +7,12 @@ "application-deployment": "deployment/application-deployment/", "checks-examples": "deployment/zero-downtime-deploys/", - "process-management": "deployment/process-management/", "remote-commands": "deployment/remote-commands/", + "process-management": "processes/process-management/", + "deployment/process-management/": "processes/process-management/", + "deployment/one-off-processes": "processes/deployment/one-off-tasks/", + "deployment/buildpacks": "deployment/methods/herokuish-buildpacks/", "deployment/methods/buildpacks": "deployment/methods/herokuish-buildpacks/", "deployment/dockerfiles": "deployment/methods/dockerfiles/", diff --git a/plugins/00_dokku-standard/subcommands/run b/plugins/00_dokku-standard/subcommands/run index aee53826a..0cfd97888 100755 --- a/plugins/00_dokku-standard/subcommands/run +++ b/plugins/00_dokku-standard/subcommands/run @@ -8,11 +8,24 @@ cmd-run() { declare cmd="run" [[ "$1" == "$cmd" ]] && shift 1 declare APP="" - + local SCHEDULER_ID declare -a RUN_ENV RUN_ENV=() while [[ $# -gt 0 ]]; do case $1 in + --cron-id=*) + local arg=$(printf "%s" "$1" | sed -E 's/(^--cron-id=)//g') + SCHEDULER_ID+=("$arg") + shift + ;; + --cron-id) + if [[ ! $2 ]]; then + dokku_log_warn "expected $1 to have an argument" + break + fi + CRON_ID+=("$2") + shift 2 + ;; -e=* | --env=*) local arg=$(printf "%s" "$1" | sed -E 's/(^-e=)|(^--env=)//g') RUN_ENV+=("$arg") @@ -37,7 +50,7 @@ cmd-run() { verify_app_name "$APP" local DOKKU_SCHEDULER=$(get_app_scheduler "$APP") - plugn trigger scheduler-run "$DOKKU_SCHEDULER" "$APP" "${#RUN_ENV[@]}" "${RUN_ENV[@]}" "$@" + DOKKU_CRON_ID="$CRON_ID" plugn trigger scheduler-run "$DOKKU_SCHEDULER" "$APP" "${#RUN_ENV[@]}" "${RUN_ENV[@]}" "$@" } cmd-run "$@" diff --git a/plugins/app-json/.gitignore b/plugins/app-json/.gitignore index d04745dda..f7dc085bc 100644 --- a/plugins/app-json/.gitignore +++ b/plugins/app-json/.gitignore @@ -2,3 +2,4 @@ /triggers /pre-* /post-* +/install diff --git a/plugins/app-json/Makefile b/plugins/app-json/Makefile index 1640a6195..b3f6ecead 100644 --- a/plugins/app-json/Makefile +++ b/plugins/app-json/Makefile @@ -1,4 +1,4 @@ -TRIGGERS = triggers/post-deploy triggers/pre-deploy +TRIGGERS = triggers/install triggers/post-delete triggers/post-deploy triggers/pre-deploy BUILD = triggers PLUGIN_NAME = app-json diff --git a/plugins/app-json/appjson.go b/plugins/app-json/appjson.go index e83d7eaa4..5290c40dc 100644 --- a/plugins/app-json/appjson.go +++ b/plugins/app-json/appjson.go @@ -1,22 +1,14 @@ package appjson import ( - "bytes" - "context" - "encoding/json" - "errors" - "fmt" - "io/ioutil" - "os" - "strings" + "path/filepath" "github.com/dokku/dokku/plugins/common" - shellquote "github.com/kballard/go-shellquote" - "golang.org/x/sync/errgroup" ) // AppJSON is a struct that represents an app.json file as understood by Dokku type AppJSON struct { + Cron []CronCommand `json:"cron"` Scripts struct { Dokku struct { Predeploy string `json:"predeploy"` @@ -25,382 +17,18 @@ type AppJSON struct { } `json:"scripts"` } -func constructScript(command string, shell string, isHerokuishImage bool, hasEntrypoint bool) []string { - if hasEntrypoint { - words, err := shellquote.Split(strings.TrimSpace(command)) - if err != nil { - common.LogWarn(fmt.Sprintf("Skipping command construction for app with ENTRYPOINT: %v", err.Error())) - return nil - } - return words - } - - script := []string{"set -eo pipefail;"} - if os.Getenv("DOKKU_TRACE") == "1" { - script = append(script, "set -x;") - } - - if isHerokuishImage { - script = append(script, []string{ - "if [[ -d '/app' ]]; then", - " export HOME=/app;", - " cd $HOME;", - "fi;", - "if [[ -d '/app/.profile.d' ]]; then", - " for file in /app/.profile.d/*; do source $file; done;", - "fi;", - - "if [[ -d '/cache' ]]; then", - " rm -rf /tmp/cache ;", - " ln -sf /cache /tmp/cache;", - "fi;", - }...) - } - - if strings.HasPrefix(command, "/") { - commandBin := strings.Split(command, " ")[0] - script = append(script, []string{ - fmt.Sprintf("if [[ ! -x \"%s\" ]]; then", commandBin), - " echo specified binary is not executable;", - " exit 1;", - "fi;", - }...) - } - - script = append(script, fmt.Sprintf("%s || exit 1;", command)) - - if isHerokuishImage { - script = append(script, []string{ - "if [[ -d '/cache' ]]; then", - " rm -f /tmp/cache;", - "fi;", - }...) - } - - return []string{shell, "-c", strings.Join(script, " ")} +// CronCommand is a struct that represents a single cron command from an app.json file +type CronCommand struct { + Command string `json:"command"` + Schedule string `json:"schedule"` } -// getPhaseScript extracts app.json from app image and returns the appropriate json key/value -func getPhaseScript(appName string, image string, phase string) (string, error) { - appJSONFile, err := ioutil.TempFile(os.TempDir(), fmt.Sprintf("dokku-%s-%s", common.MustGetEnv("DOKKU_PID"), "getPhaseScript")) - if err != nil { - return "", fmt.Errorf("Cannot create temporary file: %v", err) - } - - defer os.Remove(appJSONFile.Name()) - - common.CopyFromImage(appName, image, "app.json", appJSONFile.Name()) - if !common.FileExists(appJSONFile.Name()) { - return "", nil - } - - b, err := ioutil.ReadFile(appJSONFile.Name()) - if err != nil { - return "", fmt.Errorf("Cannot read app.json file: %v", err) - } - - if strings.TrimSpace(string(b)) == "" { - return "", nil - } - - var appJSON AppJSON - if err = json.Unmarshal(b, &appJSON); err != nil { - return "", fmt.Errorf("Cannot parse app.json: %v", err) - } - - if phase == "predeploy" { - return appJSON.Scripts.Dokku.Predeploy, nil - } - - return appJSON.Scripts.Dokku.Postdeploy, nil +// GetAppjsonDirectory returns the directory containing a given app's extracted app.json file +func GetAppjsonDirectory(appName string) string { + return filepath.Join(common.MustGetEnv("DOKKU_LIB_ROOT"), "data", "app-json", appName) } -// getReleaseCommand extracts the release command from a given app's procfile -func getReleaseCommand(appName string, image string) string { - err := common.SuppressOutput(func() error { - return common.PlugnTrigger("procfile-extract", []string{appName, image}...) - }) - - if err != nil { - return "" - } - - processType := "release" - port := "5000" - b, _ := common.PlugnTriggerOutput("procfile-get-command", []string{appName, processType, port}...) - return strings.TrimSpace(string(b[:])) -} - -func getDokkuAppShell(appName string) string { - shell := "/bin/bash" - globalShell := "" - appShell := "" - - ctx := context.Background() - errs, ctx := errgroup.WithContext(ctx) - errs.Go(func() error { - b, _ := common.PlugnTriggerOutput("config-get-global", []string{"DOKKU_APP_SHELL"}...) - globalShell = strings.TrimSpace(string(b[:])) - return nil - }) - errs.Go(func() error { - b, _ := common.PlugnTriggerOutput("config-global", []string{"DOKKU_APP_SHELL"}...) - appShell = strings.TrimSpace(string(b[:])) - return nil - }) - - errs.Wait() - if appShell != "" { - shell = appShell - } else if globalShell != "" { - shell = globalShell - } - - return shell -} - -func executeScript(appName string, imageTag string, phase string) error { - common.LogInfo1(fmt.Sprintf("Checking for %s task", phase)) - image, err := common.GetDeployingAppImageName(appName, imageTag, "") - if err != nil { - common.LogFail(err.Error()) - } - command := "" - phaseSource := "" - if phase == "release" { - command = getReleaseCommand(appName, image) - phaseSource = "Procfile" - } else { - var err error - phaseSource = "app.json" - if command, err = getPhaseScript(appName, image, phase); err != nil { - common.LogExclaim(err.Error()) - } - } - - if command == "" { - common.LogVerbose(fmt.Sprintf("No %s task found, skipping", phase)) - return nil - } - - common.LogInfo1(fmt.Sprintf("Executing %s task from %s: %s", phase, phaseSource, command)) - isHerokuishImage := common.IsImageHerokuishBased(image, appName) - dockerfileEntrypoint := "" - dockerfileCommand := "" - if !isHerokuishImage { - dockerfileEntrypoint, _ = getEntrypointFromImage(image) - dockerfileCommand, _ = getCommandFromImage(image) - } - - hasEntrypoint := dockerfileEntrypoint != "" - dokkuAppShell := getDokkuAppShell(appName) - script := constructScript(command, dokkuAppShell, isHerokuishImage, hasEntrypoint) - - imageSourceType := "dockerfile" - if isHerokuishImage { - imageSourceType = "herokuish" - } - - cacheDir := fmt.Sprintf("%s/cache", common.AppRoot(appName)) - cacheHostDir := fmt.Sprintf("%s/cache", common.AppHostRoot(appName)) - if !common.DirectoryExists(cacheDir) { - os.MkdirAll(cacheDir, 0755) - } - - var dockerArgs []string - if b, err := common.PlugnTriggerSetup("docker-args-deploy", []string{appName, imageTag}...).SetInput("").Output(); err == nil { - words, err := shellquote.Split(strings.TrimSpace(string(b[:]))) - if err != nil { - return err - } - - dockerArgs = append(dockerArgs, words...) - } - - if b, err := common.PlugnTriggerSetup("docker-args-process-deploy", []string{appName, imageSourceType, imageTag}...).SetInput("").Output(); err == nil { - words, err := shellquote.Split(strings.TrimSpace(string(b[:]))) - if err != nil { - return err - } - - dockerArgs = append(dockerArgs, words...) - } - - filteredArgs := []string{"restart", "cpus", "memory", "memory-swap", "memory-reservation", "gpus"} - for _, filteredArg := range filteredArgs { - // re := regexp.MustCompile("--" + filteredArg + "=[0-9A-Za-z!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~]+ ") - - var filteredDockerArgs []string - for _, dockerArg := range dockerArgs { - if !strings.HasPrefix(dockerArg, "--"+filteredArg+"=") { - filteredDockerArgs = append(filteredDockerArgs, dockerArg) - } - } - - dockerArgs = filteredDockerArgs - } - - dockerArgs = append(dockerArgs, "--label=dokku_phase_script="+phase) - dockerArgs = append(dockerArgs, "-v", cacheHostDir+":/cache") - if os.Getenv("DOKKU_TRACE") != "" { - dockerArgs = append(dockerArgs, "--env", "DOKKU_TRACE="+os.Getenv("DOKKU_TRACE")) - } - - containerID, err := createdContainerID(appName, dockerArgs, image, script, phase) - if err != nil { - common.LogFail(fmt.Sprintf("Failed to create %s execution container: %s", phase, err.Error())) - } - - if !waitForExecution(containerID) { - common.LogInfo2Quiet(fmt.Sprintf("Start of %s %s task (%s) output", appName, phase, containerID[0:9])) - common.LogVerboseQuietContainerLogs(containerID) - common.LogInfo2Quiet(fmt.Sprintf("End of %s %s task (%s) output", appName, phase, containerID[0:9])) - common.LogFail(fmt.Sprintf("Execution of %s task failed: %s", phase, command)) - } - - common.LogInfo2Quiet(fmt.Sprintf("Start of %s %s task (%s) output", appName, phase, containerID[0:9])) - common.LogVerboseQuietContainerLogs(containerID) - common.LogInfo2Quiet(fmt.Sprintf("End of %s %s task (%s) output", appName, phase, containerID[0:9])) - - if phase != "predeploy" { - return nil - } - - commitArgs := []string{"container", "commit"} - if !isHerokuishImage { - if dockerfileEntrypoint != "" { - commitArgs = append(commitArgs, "--change", dockerfileEntrypoint) - } - - if dockerfileCommand != "" { - commitArgs = append(commitArgs, "--change", dockerfileCommand) - } - } - - commitArgs = append(commitArgs, []string{ - "--change", - "LABEL org.label-schema.schema-version=1.0", - "--change", - "LABEL org.label-schema.vendor=dokku", - "--change", - fmt.Sprintf("LABEL com.dokku.app-name=%s", appName), - "--change", - fmt.Sprintf("LABEL com.dokku.%s-phase=true", phase), - }...) - commitArgs = append(commitArgs, containerID, image) - containerCommitCmd := common.NewShellCmdWithArgs( - common.DockerBin(), - commitArgs..., - ) - containerCommitCmd.ShowOutput = false - containerCommitCmd.Command.Stderr = os.Stderr - if !containerCommitCmd.Execute() { - common.LogFail(fmt.Sprintf("Commiting of '%s' to image failed: %s", phase, command)) - } - - return common.PlugnTrigger("scheduler-register-retired", []string{appName, containerID}...) -} - -func getEntrypointFromImage(image string) (string, error) { - output, err := common.DockerInspect(image, "{{json .Config.Entrypoint}}") - if err != nil { - return "", err - } - if output == "null" { - return "", err - } - - var entrypoint []string - if err = json.Unmarshal([]byte(output), &entrypoint); err != nil { - return "", err - } - - if len(entrypoint) == 3 && entrypoint[0] == "/bin/sh" && entrypoint[1] == "-c" { - return fmt.Sprintf("ENTRYPOINT %s", entrypoint[2]), nil - } - - serializedEntrypoint, err := json.Marshal(entrypoint) - return fmt.Sprintf("ENTRYPOINT %s", string(serializedEntrypoint)), err -} - -func getCommandFromImage(image string) (string, error) { - output, err := common.DockerInspect(image, "{{json .Config.Cmd}}") - if err != nil { - return "", err - } - if output == "null" { - return "", err - } - - var command []string - if err = json.Unmarshal([]byte(output), &command); err != nil { - return "", err - } - - if len(command) == 3 && command[0] == "/bin/sh" && command[1] == "-c" { - return fmt.Sprintf("CMD %s", command[2]), nil - } - - serializedEntrypoint, err := json.Marshal(command) - return fmt.Sprintf("CMD %s", string(serializedEntrypoint)), err -} - -func waitForExecution(containerID string) bool { - containerStartCmd := common.NewShellCmdWithArgs( - common.DockerBin(), - "container", - "start", - containerID, - ) - containerStartCmd.ShowOutput = false - containerStartCmd.Command.Stderr = os.Stderr - if !containerStartCmd.Execute() { - return false - } - - containerWaitCmd := common.NewShellCmdWithArgs( - common.DockerBin(), - "container", - "wait", - containerID, - ) - - containerWaitCmd.ShowOutput = false - containerWaitCmd.Command.Stderr = os.Stderr - b, err := containerWaitCmd.Output() - if err != nil { - return false - } - - containerExitCode := strings.TrimSpace(string(b[:])) - return containerExitCode == "0" -} - -func createdContainerID(appName string, dockerArgs []string, image string, command []string, phase string) (string, error) { - runLabelArgs := fmt.Sprintf("--label=com.dokku.app-name=%s", appName) - - arguments := strings.Split(common.MustGetEnv("DOKKU_GLOBAL_RUN_ARGS"), " ") - arguments = append(arguments, runLabelArgs) - arguments = append(arguments, dockerArgs...) - - arguments = append([]string{"container", "create"}, arguments...) - arguments = append(arguments, image) - arguments = append(arguments, command...) - - containerCreateCmd := common.NewShellCmdWithArgs( - common.DockerBin(), - arguments..., - ) - var stderr bytes.Buffer - containerCreateCmd.ShowOutput = false - containerCreateCmd.Command.Stderr = &stderr - - b, err := containerCreateCmd.Output() - if err != nil { - return "", errors.New(stderr.String()) - } - - containerID := strings.TrimSpace(string(b)) - err = common.PlugnTrigger("post-container-create", []string{"app", appName, containerID, phase}...) - return containerID, err +// GetAppjsonPath returns the path to a given app's extracted app.json file +func GetAppjsonPath(appName string) string { + return filepath.Join(GetAppjsonDirectory(appName), "app.json") } diff --git a/plugins/app-json/functions.go b/plugins/app-json/functions.go new file mode 100644 index 000000000..e79f2d0e6 --- /dev/null +++ b/plugins/app-json/functions.go @@ -0,0 +1,409 @@ +package appjson + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io/ioutil" + "os" + "path/filepath" + "strings" + + "github.com/dokku/dokku/plugins/common" + shellquote "github.com/kballard/go-shellquote" + "golang.org/x/sync/errgroup" +) + +func constructScript(command string, shell string, isHerokuishImage bool, hasEntrypoint bool) []string { + if hasEntrypoint { + words, err := shellquote.Split(strings.TrimSpace(command)) + if err != nil { + common.LogWarn(fmt.Sprintf("Skipping command construction for app with ENTRYPOINT: %v", err.Error())) + return nil + } + return words + } + + script := []string{"set -eo pipefail;"} + if os.Getenv("DOKKU_TRACE") == "1" { + script = append(script, "set -x;") + } + + if isHerokuishImage { + script = append(script, []string{ + "if [[ -d '/app' ]]; then", + " export HOME=/app;", + " cd $HOME;", + "fi;", + "if [[ -d '/app/.profile.d' ]]; then", + " for file in /app/.profile.d/*; do source $file; done;", + "fi;", + + "if [[ -d '/cache' ]]; then", + " rm -rf /tmp/cache ;", + " ln -sf /cache /tmp/cache;", + "fi;", + }...) + } + + if strings.HasPrefix(command, "/") { + commandBin := strings.Split(command, " ")[0] + script = append(script, []string{ + fmt.Sprintf("if [[ ! -x \"%s\" ]]; then", commandBin), + " echo specified binary is not executable;", + " exit 1;", + "fi;", + }...) + } + + script = append(script, fmt.Sprintf("%s || exit 1;", command)) + + if isHerokuishImage { + script = append(script, []string{ + "if [[ -d '/cache' ]]; then", + " rm -f /tmp/cache;", + "fi;", + }...) + } + + return []string{shell, "-c", strings.Join(script, " ")} +} + +// getPhaseScript extracts app.json from app image and returns the appropriate json key/value +func getPhaseScript(appName string, phase string) (string, error) { + if !common.FileExists(GetAppjsonPath(appName)) { + return "", nil + } + + b, err := ioutil.ReadFile(GetAppjsonPath(appName)) + if err != nil { + return "", fmt.Errorf("Cannot read app.json file: %v", err) + } + + if strings.TrimSpace(string(b)) == "" { + return "", nil + } + + var appJSON AppJSON + if err = json.Unmarshal(b, &appJSON); err != nil { + return "", fmt.Errorf("Cannot parse app.json: %v", err) + } + + if phase == "predeploy" { + return appJSON.Scripts.Dokku.Predeploy, nil + } + + return appJSON.Scripts.Dokku.Postdeploy, nil +} + +// getReleaseCommand extracts the release command from a given app's procfile +func getReleaseCommand(appName string, image string) string { + err := common.SuppressOutput(func() error { + return common.PlugnTrigger("procfile-extract", []string{appName, image}...) + }) + + if err != nil { + return "" + } + + processType := "release" + port := "5000" + b, _ := common.PlugnTriggerOutput("procfile-get-command", []string{appName, processType, port}...) + return strings.TrimSpace(string(b[:])) +} + +func getDokkuAppShell(appName string) string { + shell := "/bin/bash" + globalShell := "" + appShell := "" + + ctx := context.Background() + errs, ctx := errgroup.WithContext(ctx) + errs.Go(func() error { + b, _ := common.PlugnTriggerOutput("config-get-global", []string{"DOKKU_APP_SHELL"}...) + globalShell = strings.TrimSpace(string(b[:])) + return nil + }) + errs.Go(func() error { + b, _ := common.PlugnTriggerOutput("config-global", []string{"DOKKU_APP_SHELL"}...) + appShell = strings.TrimSpace(string(b[:])) + return nil + }) + + errs.Wait() + if appShell != "" { + shell = appShell + } else if globalShell != "" { + shell = globalShell + } + + return shell +} + +func executeScript(appName string, image string, imageTag string, phase string) error { + common.LogInfo1(fmt.Sprintf("Checking for %s task", phase)) + command := "" + phaseSource := "" + if phase == "release" { + command = getReleaseCommand(appName, image) + phaseSource = "Procfile" + } else { + var err error + phaseSource = "app.json" + if command, err = getPhaseScript(appName, phase); err != nil { + common.LogExclaim(err.Error()) + } + } + + if command == "" { + common.LogVerbose(fmt.Sprintf("No %s task found, skipping", phase)) + return nil + } + + common.LogInfo1(fmt.Sprintf("Executing %s task from %s: %s", phase, phaseSource, command)) + isHerokuishImage := common.IsImageHerokuishBased(image, appName) + dockerfileEntrypoint := "" + dockerfileCommand := "" + if !isHerokuishImage { + dockerfileEntrypoint, _ = getEntrypointFromImage(image) + dockerfileCommand, _ = getCommandFromImage(image) + } + + hasEntrypoint := dockerfileEntrypoint != "" + dokkuAppShell := getDokkuAppShell(appName) + script := constructScript(command, dokkuAppShell, isHerokuishImage, hasEntrypoint) + + imageSourceType := "dockerfile" + if isHerokuishImage { + imageSourceType = "herokuish" + } + + cacheDir := fmt.Sprintf("%s/cache", common.AppRoot(appName)) + cacheHostDir := fmt.Sprintf("%s/cache", common.AppHostRoot(appName)) + if !common.DirectoryExists(cacheDir) { + os.MkdirAll(cacheDir, 0755) + } + + var dockerArgs []string + if b, err := common.PlugnTriggerSetup("docker-args-deploy", []string{appName, imageTag}...).SetInput("").Output(); err == nil { + words, err := shellquote.Split(strings.TrimSpace(string(b[:]))) + if err != nil { + return err + } + + dockerArgs = append(dockerArgs, words...) + } + + if b, err := common.PlugnTriggerSetup("docker-args-process-deploy", []string{appName, imageSourceType, imageTag}...).SetInput("").Output(); err == nil { + words, err := shellquote.Split(strings.TrimSpace(string(b[:]))) + if err != nil { + return err + } + + dockerArgs = append(dockerArgs, words...) + } + + filteredArgs := []string{"restart", "cpus", "memory", "memory-swap", "memory-reservation", "gpus"} + for _, filteredArg := range filteredArgs { + // re := regexp.MustCompile("--" + filteredArg + "=[0-9A-Za-z!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~]+ ") + + var filteredDockerArgs []string + for _, dockerArg := range dockerArgs { + if !strings.HasPrefix(dockerArg, "--"+filteredArg+"=") { + filteredDockerArgs = append(filteredDockerArgs, dockerArg) + } + } + + dockerArgs = filteredDockerArgs + } + + dockerArgs = append(dockerArgs, "--label=dokku_phase_script="+phase) + dockerArgs = append(dockerArgs, "-v", cacheHostDir+":/cache") + if os.Getenv("DOKKU_TRACE") != "" { + dockerArgs = append(dockerArgs, "--env", "DOKKU_TRACE="+os.Getenv("DOKKU_TRACE")) + } + + containerID, err := createdContainerID(appName, dockerArgs, image, script, phase) + if err != nil { + common.LogFail(fmt.Sprintf("Failed to create %s execution container: %s", phase, err.Error())) + } + + if !waitForExecution(containerID) { + common.LogInfo2Quiet(fmt.Sprintf("Start of %s %s task (%s) output", appName, phase, containerID[0:9])) + common.LogVerboseQuietContainerLogs(containerID) + common.LogInfo2Quiet(fmt.Sprintf("End of %s %s task (%s) output", appName, phase, containerID[0:9])) + common.LogFail(fmt.Sprintf("Execution of %s task failed: %s", phase, command)) + } + + common.LogInfo2Quiet(fmt.Sprintf("Start of %s %s task (%s) output", appName, phase, containerID[0:9])) + common.LogVerboseQuietContainerLogs(containerID) + common.LogInfo2Quiet(fmt.Sprintf("End of %s %s task (%s) output", appName, phase, containerID[0:9])) + + if phase != "predeploy" { + return nil + } + + commitArgs := []string{"container", "commit"} + if !isHerokuishImage { + if dockerfileEntrypoint != "" { + commitArgs = append(commitArgs, "--change", dockerfileEntrypoint) + } + + if dockerfileCommand != "" { + commitArgs = append(commitArgs, "--change", dockerfileCommand) + } + } + + commitArgs = append(commitArgs, []string{ + "--change", + "LABEL org.label-schema.schema-version=1.0", + "--change", + "LABEL org.label-schema.vendor=dokku", + "--change", + fmt.Sprintf("LABEL com.dokku.app-name=%s", appName), + "--change", + fmt.Sprintf("LABEL com.dokku.%s-phase=true", phase), + }...) + commitArgs = append(commitArgs, containerID, image) + containerCommitCmd := common.NewShellCmdWithArgs( + common.DockerBin(), + commitArgs..., + ) + containerCommitCmd.ShowOutput = false + containerCommitCmd.Command.Stderr = os.Stderr + if !containerCommitCmd.Execute() { + common.LogFail(fmt.Sprintf("Commiting of '%s' to image failed: %s", phase, command)) + } + + return common.PlugnTrigger("scheduler-register-retired", []string{appName, containerID}...) +} + +func getEntrypointFromImage(image string) (string, error) { + output, err := common.DockerInspect(image, "{{json .Config.Entrypoint}}") + if err != nil { + return "", err + } + if output == "null" { + return "", err + } + + var entrypoint []string + if err = json.Unmarshal([]byte(output), &entrypoint); err != nil { + return "", err + } + + if len(entrypoint) == 3 && entrypoint[0] == "/bin/sh" && entrypoint[1] == "-c" { + return fmt.Sprintf("ENTRYPOINT %s", entrypoint[2]), nil + } + + serializedEntrypoint, err := json.Marshal(entrypoint) + return fmt.Sprintf("ENTRYPOINT %s", string(serializedEntrypoint)), err +} + +func getCommandFromImage(image string) (string, error) { + output, err := common.DockerInspect(image, "{{json .Config.Cmd}}") + if err != nil { + return "", err + } + if output == "null" { + return "", err + } + + var command []string + if err = json.Unmarshal([]byte(output), &command); err != nil { + return "", err + } + + if len(command) == 3 && command[0] == "/bin/sh" && command[1] == "-c" { + return fmt.Sprintf("CMD %s", command[2]), nil + } + + serializedEntrypoint, err := json.Marshal(command) + return fmt.Sprintf("CMD %s", string(serializedEntrypoint)), err +} + +func waitForExecution(containerID string) bool { + containerStartCmd := common.NewShellCmdWithArgs( + common.DockerBin(), + "container", + "start", + containerID, + ) + containerStartCmd.ShowOutput = false + containerStartCmd.Command.Stderr = os.Stderr + if !containerStartCmd.Execute() { + return false + } + + containerWaitCmd := common.NewShellCmdWithArgs( + common.DockerBin(), + "container", + "wait", + containerID, + ) + + containerWaitCmd.ShowOutput = false + containerWaitCmd.Command.Stderr = os.Stderr + b, err := containerWaitCmd.Output() + if err != nil { + return false + } + + containerExitCode := strings.TrimSpace(string(b[:])) + return containerExitCode == "0" +} + +func createdContainerID(appName string, dockerArgs []string, image string, command []string, phase string) (string, error) { + runLabelArgs := fmt.Sprintf("--label=com.dokku.app-name=%s", appName) + + arguments := strings.Split(common.MustGetEnv("DOKKU_GLOBAL_RUN_ARGS"), " ") + arguments = append(arguments, runLabelArgs) + arguments = append(arguments, dockerArgs...) + + arguments = append([]string{"container", "create"}, arguments...) + arguments = append(arguments, image) + arguments = append(arguments, command...) + + containerCreateCmd := common.NewShellCmdWithArgs( + common.DockerBin(), + arguments..., + ) + var stderr bytes.Buffer + containerCreateCmd.ShowOutput = false + containerCreateCmd.Command.Stderr = &stderr + + b, err := containerCreateCmd.Output() + if err != nil { + return "", errors.New(stderr.String()) + } + + containerID := strings.TrimSpace(string(b)) + err = common.PlugnTrigger("post-container-create", []string{"app", appName, containerID, phase}...) + return containerID, err +} + +func refreshAppJSON(appName string, image string) error { + baseDirectory := filepath.Join(common.MustGetEnv("DOKKU_LIB_ROOT"), "data", "app-json") + if !common.DirectoryExists(baseDirectory) { + return errors.New("Run 'dokku plugin:install' to ensure the correct directories exist") + } + + directory := GetAppjsonDirectory(appName) + if !common.DirectoryExists(directory) { + if err := os.MkdirAll(directory, 0755); err != nil { + return err + } + } + + appjsonPath := GetAppjsonPath(appName) + if common.FileExists(appjsonPath) { + if err := os.Remove(appjsonPath); err != nil { + return errors.New("Unable to remove previous app.json file") + } + } + + common.CopyFromImage(appName, image, "app.json", appjsonPath) + return nil +} diff --git a/plugins/app-json/src/triggers/triggers.go b/plugins/app-json/src/triggers/triggers.go index 200ae162e..5ef4cb5f5 100644 --- a/plugins/app-json/src/triggers/triggers.go +++ b/plugins/app-json/src/triggers/triggers.go @@ -6,7 +6,7 @@ import ( "os" "strings" - "github.com/dokku/dokku/plugins/app-json" + appjson "github.com/dokku/dokku/plugins/app-json" "github.com/dokku/dokku/plugins/common" ) @@ -18,6 +18,11 @@ func main() { var err error switch trigger { + case "install": + err = appjson.TriggerInstall() + case "post-delete": + appName := flag.Arg(0) + err = appjson.TriggerPostDelete(appName) case "post-deploy": appName := flag.Arg(0) imageTag := flag.Arg(3) diff --git a/plugins/app-json/triggers.go b/plugins/app-json/triggers.go index 88d74c824..e83f065ad 100644 --- a/plugins/app-json/triggers.go +++ b/plugins/app-json/triggers.go @@ -1,8 +1,47 @@ package appjson +import ( + "os" + "path/filepath" + + "github.com/dokku/dokku/plugins/common" +) + +// TriggerInstall initializes app restart policies +func TriggerInstall() error { + directory := filepath.Join(common.MustGetEnv("DOKKU_LIB_ROOT"), "data", "app-json") + if err := os.MkdirAll(directory, 0755); err != nil { + return err + } + + if err := common.SetPermissions(directory, 0755); err != nil { + return err + } + + return nil +} + +// TriggerPostDelete destroys the app-json data for a given app container +func TriggerPostDelete(appName string) error { + directory := filepath.Join(common.MustGetEnv("DOKKU_LIB_ROOT"), "data", "app-json", appName) + dataErr := os.RemoveAll(directory) + propertyErr := common.PropertyDestroy("app-json", appName) + + if dataErr != nil { + return dataErr + } + + return propertyErr +} + // TriggerPostDeploy is a trigger to execute the postdeploy deployment task func TriggerPostDeploy(appName string, imageTag string) error { - if err := executeScript(appName, imageTag, "postdeploy"); err != nil { + image, err := common.GetDeployingAppImageName(appName, imageTag, "") + if err != nil { + return err + } + + if err := executeScript(appName, image, imageTag, "postdeploy"); err != nil { return err } return nil @@ -10,11 +49,20 @@ func TriggerPostDeploy(appName string, imageTag string) error { // TriggerPreDeploy is a trigger to execute predeploy and release deployment tasks func TriggerPreDeploy(appName string, imageTag string) error { - if err := executeScript(appName, imageTag, "predeploy"); err != nil { + image, err := common.GetDeployingAppImageName(appName, imageTag, "") + if err != nil { return err } - if err := executeScript(appName, imageTag, "release"); err != nil { + if err := refreshAppJSON(appName, image); err != nil { + return err + } + + if err := executeScript(appName, image, imageTag, "predeploy"); err != nil { + return err + } + + if err := executeScript(appName, image, imageTag, "release"); err != nil { return err } return nil diff --git a/plugins/buildpacks/report.go b/plugins/buildpacks/report.go index e67bdd72b..ca53623c2 100644 --- a/plugins/buildpacks/report.go +++ b/plugins/buildpacks/report.go @@ -6,7 +6,7 @@ import ( "github.com/dokku/dokku/plugins/common" ) -// ReportSingleApp is an internal function that displays the app report for one or more apps +// ReportSingleApp is an internal function that displays the buildpacks report for one or more apps func ReportSingleApp(appName, infoFlag string) error { if err := common.VerifyAppName(appName); err != nil { return err diff --git a/plugins/cron/.gitignore b/plugins/cron/.gitignore new file mode 100644 index 000000000..06649f17c --- /dev/null +++ b/plugins/cron/.gitignore @@ -0,0 +1,6 @@ +/commands +/subcommands/* +/triggers/* +/triggers +/post-* +/report diff --git a/plugins/cron/Makefile b/plugins/cron/Makefile new file mode 100644 index 000000000..a0f2f16d7 --- /dev/null +++ b/plugins/cron/Makefile @@ -0,0 +1,6 @@ +SUBCOMMANDS = subcommands/list subcommands/report +TRIGGERS = triggers/post-delete triggers/post-deploy triggers/report +BUILD = commands subcommands triggers +PLUGIN_NAME = cron + +include ../../common.mk diff --git a/plugins/cron/functions.go b/plugins/cron/functions.go new file mode 100644 index 000000000..8318856ce --- /dev/null +++ b/plugins/cron/functions.go @@ -0,0 +1,161 @@ +package cron + +import ( + "encoding/base64" + "encoding/json" + "fmt" + "html/template" + "io/ioutil" + "os" + "path/filepath" + "strings" + + appjson "github.com/dokku/dokku/plugins/app-json" + "github.com/dokku/dokku/plugins/common" + + cronparser "github.com/robfig/cron/v3" +) + +type templateCommand struct { + ID string + App string + Command string + Schedule string +} + +func fetchCronEntries(appName string) ([]templateCommand, error) { + commands := []templateCommand{} + scheduler := common.GetAppScheduler(appName) + if scheduler != "docker-local" { + return commands, nil + } + + appjsonPath := appjson.GetAppjsonPath(appName) + if !common.FileExists(appjsonPath) { + return commands, nil + } + + b, err := ioutil.ReadFile(appjsonPath) + if err != nil { + return commands, fmt.Errorf("Cannot read app.json file for %s: %v", appName, err) + } + + if strings.TrimSpace(string(b)) == "" { + return commands, nil + } + + var appJSON appjson.AppJSON + if err = json.Unmarshal(b, &appJSON); err != nil { + return commands, fmt.Errorf("Cannot parse app.json for %s: %v", appName, err) + } + + for _, c := range appJSON.Cron { + parser := cronparser.NewParser(cronparser.Minute | cronparser.Hour | cronparser.Dom | cronparser.Month | cronparser.Dow | cronparser.Descriptor) + _, err := parser.Parse(c.Schedule) + if err != nil { + return commands, err + } + + commands = append(commands, templateCommand{ + App: appName, + Command: c.Command, + Schedule: c.Schedule, + ID: generateCommandID(appName, c), + }) + } + + return commands, nil +} + +func deleteCrontab() error { + command := common.NewShellCmd("crontab -l -u dokku") + command.ShowOutput = false + if !command.Execute() { + return nil + } + + command = common.NewShellCmd("crontab -r -u dokku") + command.ShowOutput = false + out, err := command.CombinedOutput() + if err != nil { + return fmt.Errorf("Unable to remove schedule file: %v", string(out)) + } + + common.LogInfo1("Removed") + return nil +} + +func writeCronEntries() error { + apps, err := common.DokkuApps() + if err != nil { + return deleteCrontab() + } + + commands := []templateCommand{} + for _, appName := range apps { + scheduler := common.GetAppScheduler(appName) + if scheduler != "docker-local" { + continue + } + + c, err := fetchCronEntries(appName) + if err != nil { + return err + } + + commands = append(commands, c...) + } + + if len(commands) == 0 { + return deleteCrontab() + } + + data := map[string]interface{}{ + "Commands": commands, + } + + t, err := getCronTemplate() + if err != nil { + return err + } + + tmpFile, err := ioutil.TempFile(os.TempDir(), fmt.Sprintf("dokku-%s-%s", common.MustGetEnv("DOKKU_PID"), "CopyFromImage")) + if err != nil { + return fmt.Errorf("Cannot create temporary schedule file: %v", err) + } + + defer tmpFile.Close() + defer os.Remove(tmpFile.Name()) + + if err := t.Execute(tmpFile, data); err != nil { + return fmt.Errorf("Unable to template out schedule file: %v", err) + } + + command := common.NewShellCmd(fmt.Sprintf("crontab -u dokku %s", tmpFile.Name())) + command.ShowOutput = false + out, err := command.CombinedOutput() + if err != nil { + return fmt.Errorf("Unable to update schedule file: %v", out) + } + + common.LogInfo1("Updated schedule file") + + return nil +} + +func getCronTemplate() (*template.Template, error) { + t := template.New("cron") + + templatePath := filepath.Join(common.MustGetEnv("PLUGIN_ENABLED_PATH"), "cron", "templates", "cron.tmpl") + b, err := ioutil.ReadFile(templatePath) + if err != nil { + return t, fmt.Errorf("Cannot read template file: %v", err) + } + + s := strings.TrimSpace(string(b)) + return t.Parse(s) +} + +func generateCommandID(appName string, c appjson.CronCommand) string { + return base64.StdEncoding.EncodeToString([]byte(appName + "===" + c.Command + "===" + c.Schedule)) +} diff --git a/plugins/cron/go.mod b/plugins/cron/go.mod new file mode 100644 index 000000000..822b875bd --- /dev/null +++ b/plugins/cron/go.mod @@ -0,0 +1,15 @@ +module github.com/dokku/dokku/plugins/cron + +go 1.15 + +require ( + github.com/dokku/dokku/plugins/app-json v0.0.0-00010101000000-000000000000 + github.com/dokku/dokku/plugins/common v0.0.0-00010101000000-000000000000 + github.com/robfig/cron/v3 v3.0.1 + github.com/ryanuber/columnize v1.1.2-0.20190319233515-9e6335e58db3 + github.com/spf13/pflag v1.0.5 // indirect +) + +replace github.com/dokku/dokku/plugins/app-json => ../app-json + +replace github.com/dokku/dokku/plugins/common => ../common diff --git a/plugins/cron/go.sum b/plugins/cron/go.sum new file mode 100644 index 000000000..914d85c0c --- /dev/null +++ b/plugins/cron/go.sum @@ -0,0 +1,15 @@ +github.com/codegangsta/inject v0.0.0-20150114235600-33e0aa1cb7c0 h1:sDMmm+q/3+BukdIpxwO365v/Rbspp2Nt5XntgQRXq8Q= +github.com/codegangsta/inject v0.0.0-20150114235600-33e0aa1cb7c0/go.mod h1:4Zcjuz89kmFXt9morQgcfYZAYZ5n8WHjt81YYWIwtTM= +github.com/codeskyblue/go-sh v0.0.0-20190412065543-76bd3d59ff27 h1:HHUr4P/aKh4quafGxDT9LDasjGdlGkzLbfmmrlng3kA= +github.com/codeskyblue/go-sh v0.0.0-20190412065543-76bd3d59ff27/go.mod h1:VQx0hjo2oUeQkQUET7wRwradO6f+fN5jzXgB/zROxxE= +github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 h1:Z9n2FFNUXsshfwJMBgNA0RU6/i7WVaAegv3PtuIHPMs= +github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51/go.mod h1:CzGEWj7cYgsdH8dAjBGEr58BoE7ScuLd+fwFZ44+/x8= +github.com/robfig/cron v1.2.0 h1:ZjScXvvxeQ63Dbyxy76Fj3AT3Ut0aKsyd2/tl3DTMuQ= +github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs= +github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro= +github.com/ryanuber/columnize v1.1.2-0.20190319233515-9e6335e58db3 h1:utdYOikI1XjNtTFGCwSM6OmFJblU4ld4gACoJsbadJg= +github.com/ryanuber/columnize v1.1.2-0.20190319233515-9e6335e58db3/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts= +github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= +github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +golang.org/x/sync v0.0.0-20201207232520-09787c993a3a h1:DcqTD9SDLc+1P/r1EmRBwnVsrOwW+kk2vWf9n+1sGhs= +golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= diff --git a/plugins/cron/plugin.toml b/plugins/cron/plugin.toml new file mode 100644 index 000000000..4c3d019a2 --- /dev/null +++ b/plugins/cron/plugin.toml @@ -0,0 +1,4 @@ +[plugin] +description = "dokku core cron plugin" +version = "0.22.9" +[plugin.config] diff --git a/plugins/cron/report.go b/plugins/cron/report.go new file mode 100644 index 000000000..f3766740b --- /dev/null +++ b/plugins/cron/report.go @@ -0,0 +1,33 @@ +package cron + +import ( + "strconv" + + "github.com/dokku/dokku/plugins/common" +) + +// ReportSingleApp is an internal function that displays the cron report for one or more apps +func ReportSingleApp(appName, infoFlag string) error { + if err := common.VerifyAppName(appName); err != nil { + return err + } + + flags := map[string]common.ReportFunc{ + "--cron-task-count": reportTasks, + } + + flagKeys := []string{} + for flagKey := range flags { + flagKeys = append(flagKeys, flagKey) + } + + trimPrefix := false + uppercaseFirstCharacter := true + infoFlags := common.CollectReport(appName, infoFlag, flags) + return common.ReportSingleApp("cron", appName, infoFlag, infoFlags, flagKeys, trimPrefix, uppercaseFirstCharacter) +} + +func reportTasks(appName string) string { + c, _ := fetchCronEntries(appName) + return strconv.Itoa(len(c)) +} diff --git a/plugins/cron/src/commands/commands.go b/plugins/cron/src/commands/commands.go new file mode 100644 index 000000000..402ac2799 --- /dev/null +++ b/plugins/cron/src/commands/commands.go @@ -0,0 +1,56 @@ +package main + +import ( + "flag" + "fmt" + "os" + "strconv" + "strings" + + "github.com/dokku/dokku/plugins/common" +) + +const ( + helpHeader = `Usage: dokku cron[:COMMAND] + +Manage scheduled cron tasks + +Additional commands:` + + helpContent = ` + cron:list , List scheduled cron tasks for an app + cron:report [] [], Display report about an app +` +) + +func main() { + flag.Usage = usage + flag.Parse() + + cmd := flag.Arg(0) + switch cmd { + case "cron", "cron:help": + usage() + case "help": + command := common.NewShellCmd(fmt.Sprintf("ps -o command= %d", os.Getppid())) + command.ShowOutput = false + output, err := command.Output() + + if err == nil && strings.Contains(string(output), "--all") { + fmt.Println(helpContent) + } else { + fmt.Print("\n cron, Manage scheduled cron tasks\n") + } + default: + dokkuNotImplementExitCode, err := strconv.Atoi(os.Getenv("DOKKU_NOT_IMPLEMENTED_EXIT")) + if err != nil { + fmt.Println("failed to retrieve DOKKU_NOT_IMPLEMENTED_EXIT environment variable") + dokkuNotImplementExitCode = 10 + } + os.Exit(dokkuNotImplementExitCode) + } +} + +func usage() { + common.CommandUsage(helpHeader, helpContent) +} diff --git a/plugins/cron/src/subcommands/subcommands.go b/plugins/cron/src/subcommands/subcommands.go new file mode 100644 index 000000000..a996040b3 --- /dev/null +++ b/plugins/cron/src/subcommands/subcommands.go @@ -0,0 +1,41 @@ +package main + +import ( + "fmt" + "os" + "strings" + + "github.com/dokku/dokku/plugins/common" + "github.com/dokku/dokku/plugins/cron" + + flag "github.com/spf13/pflag" +) + +// main entrypoint to all subcommands +func main() { + parts := strings.Split(os.Args[0], "/") + subcommand := parts[len(parts)-1] + + var err error + switch subcommand { + case "list": + args := flag.NewFlagSet("cron:list", flag.ExitOnError) + args.Parse(os.Args[2:]) + appName := args.Arg(0) + err = cron.CommandList(appName) + case "report": + args := flag.NewFlagSet("cron:report", flag.ExitOnError) + osArgs, infoFlag, flagErr := common.ParseReportArgs("cron", os.Args[2:]) + if flagErr == nil { + args.Parse(osArgs) + appName := args.Arg(0) + err = cron.CommandReport(appName, infoFlag) + } + default: + common.LogFail(fmt.Sprintf("Invalid plugin subcommand call: %s", subcommand)) + } + + if err != nil { + common.LogFail(err.Error()) + } +} diff --git a/plugins/cron/src/triggers/triggers.go b/plugins/cron/src/triggers/triggers.go new file mode 100644 index 000000000..df8ea8895 --- /dev/null +++ b/plugins/cron/src/triggers/triggers.go @@ -0,0 +1,35 @@ +package main + +import ( + "flag" + "fmt" + "os" + "strings" + + "github.com/dokku/dokku/plugins/common" + cron "github.com/dokku/dokku/plugins/cron" +) + +// main entrypoint to all triggers +func main() { + parts := strings.Split(os.Args[0], "/") + trigger := parts[len(parts)-1] + flag.Parse() + + var err error + switch trigger { + case "post-delete": + err = cron.TriggerPostDelete() + case "post-deploy": + err = cron.TriggerPostDeploy() + case "report": + appName := flag.Arg(0) + err = cron.ReportSingleApp(appName, "") + default: + common.LogFail(fmt.Sprintf("Invalid plugin trigger call: %s", trigger)) + } + + if err != nil { + common.LogFail(err.Error()) + } +} diff --git a/plugins/cron/subcommands.go b/plugins/cron/subcommands.go new file mode 100644 index 000000000..040686189 --- /dev/null +++ b/plugins/cron/subcommands.go @@ -0,0 +1,47 @@ +package cron + +import ( + "fmt" + + "github.com/dokku/dokku/plugins/common" + "github.com/ryanuber/columnize" +) + +// CommandList lists all scheduled cron tasks for a given app +func CommandList(appName string) error { + if err := common.VerifyAppName(appName); err != nil { + return err + } + + entries, err := fetchCronEntries(appName) + if err != nil { + return err + } + + output := []string{"ID | Schedule | Command"} + for _, entry := range entries { + output = append(output, fmt.Sprintf("%s | %s | %s", entry.ID, entry.Schedule, entry.Command)) + } + result := columnize.SimpleFormat(output) + fmt.Println(result) + + return nil +} + +// CommandReport displays a cron report for one or more apps +func CommandReport(appName string, infoFlag string) error { + if len(appName) == 0 { + apps, err := common.DokkuApps() + if err != nil { + return err + } + for _, appName := range apps { + if err := ReportSingleApp(appName, infoFlag); err != nil { + return err + } + } + return nil + } + + return ReportSingleApp(appName, infoFlag) +} diff --git a/plugins/cron/templates/cron.tmpl b/plugins/cron/templates/cron.tmpl new file mode 100644 index 000000000..d1e32c148 --- /dev/null +++ b/plugins/cron/templates/cron.tmpl @@ -0,0 +1,6 @@ +PATH=/usr/local/bin:/usr/bin:/bin +SHELL=/bin/bash + +{{ range $entry := .Commands -}} +{{ $entry.Schedule }} dokku --rm run --cron-id {{ $entry.ID }} {{ $entry.App }} {{ $entry.Command }} +{{ end -}} diff --git a/plugins/cron/triggers.go b/plugins/cron/triggers.go new file mode 100644 index 000000000..34f75d070 --- /dev/null +++ b/plugins/cron/triggers.go @@ -0,0 +1,11 @@ +package cron + +// TriggerPostDelete updates the cron entries for all apps +func TriggerPostDelete() error { + return writeCronEntries() +} + +// TriggerPostDeploy updates the cron entries for all apps +func TriggerPostDeploy() error { + return writeCronEntries() +} diff --git a/plugins/network/report.go b/plugins/network/report.go index a1cc197b4..bfdca4095 100644 --- a/plugins/network/report.go +++ b/plugins/network/report.go @@ -6,7 +6,7 @@ import ( "github.com/dokku/dokku/plugins/common" ) -// ReportSingleApp is an internal function that displays the app report for one or more apps +// ReportSingleApp is an internal function that displays the network report for one or more apps func ReportSingleApp(appName, infoFlag string) error { if err := common.VerifyAppName(appName); err != nil { return err diff --git a/plugins/proxy/report.go b/plugins/proxy/report.go index 01f181307..5735dee1d 100644 --- a/plugins/proxy/report.go +++ b/plugins/proxy/report.go @@ -6,7 +6,7 @@ import ( "github.com/dokku/dokku/plugins/common" ) -// ReportSingleApp is an internal function that displays the app report for one or more apps +// ReportSingleApp is an internal function that displays the proxy report for one or more apps func ReportSingleApp(appName string, infoFlag string) error { if err := common.VerifyAppName(appName); err != nil { return err diff --git a/plugins/resource/resource.go b/plugins/resource/resource.go index 7f09e973f..b579faa2b 100644 --- a/plugins/resource/resource.go +++ b/plugins/resource/resource.go @@ -17,7 +17,7 @@ type Resource struct { NvidiaGPU string `json:"nvidia-gpu"` } -// ReportSingleApp is an internal function that displays the app report for one or more apps +// ReportSingleApp is an internal function that displays the resource report for one or more apps func ReportSingleApp(appName, infoFlag string) error { if err := common.VerifyAppName(appName); err != nil { return err diff --git a/plugins/scheduler-docker-local/scheduler-run b/plugins/scheduler-docker-local/scheduler-run index 680fc4fab..81cc35edd 100755 --- a/plugins/scheduler-docker-local/scheduler-run +++ b/plugins/scheduler-docker-local/scheduler-run @@ -37,6 +37,10 @@ trigger-scheduler-docker-local-scheduler-run() { DOCKER_ARGS+=$(: | plugn trigger docker-args-process-run "$APP" "$IMAGE_SOURCE_TYPE" "$IMAGE_TAG") DOCKER_ARGS+=" -e DYNO=run.$DYNO_NUMBER --name $APP.run.$DYNO_NUMBER" + if [[ -n "$DOKKU_CRON_ID" ]]; then + DOCKER_ARGS+=" --label=com.dokku.cron-id=$DOKKU_CRON_ID" + fi + declare -a ARG_ARRAY eval "ARG_ARRAY=($DOCKER_ARGS)" diff --git a/tests/apps/python/task.py b/tests/apps/python/task.py new file mode 100644 index 000000000..6d5436600 --- /dev/null +++ b/tests/apps/python/task.py @@ -0,0 +1,5 @@ +def main(args): + print(args) + +if __name__ == '__main__': + main(sys.argv) diff --git a/tests/unit/cron.bats b/tests/unit/cron.bats new file mode 100644 index 000000000..e198a32c8 --- /dev/null +++ b/tests/unit/cron.bats @@ -0,0 +1,206 @@ +#!/usr/bin/env bats + +load test_helper + +setup() { + global_setup + create_app +} + +teardown() { + destroy_app + global_teardown +} + +@test "(cron) cron:help" { + run /bin/bash -c "dokku cron" + echo "output: $output" + echo "status: $status" + assert_output_contains "Manage scheduled cron tasks" + help_output="$output" + + run /bin/bash -c "dokku cron:help" + echo "output: $output" + echo "status: $status" + assert_output_contains "Manage scheduled cron tasks" + assert_output "$help_output" +} + +@test "(cron) invalid [missing-keys]" { + run deploy_app python dokku@dokku.me:$TEST_APP template_cron_file_invalid + echo "output: $output" + echo "status: $status" + assert_failure +} + +@test "(cron) invalid [schedule]" { + run deploy_app python dokku@dokku.me:$TEST_APP template_cron_file_invalid_schedule + echo "output: $output" + echo "status: $status" + assert_failure +} + +@test "(cron) invalid [seconds]" { + run deploy_app python dokku@dokku.me:$TEST_APP template_cron_file_invalid_schedule_seconds + echo "output: $output" + echo "status: $status" + assert_failure +} + +@test "(cron) create [empty]" { + run deploy_app python dokku@dokku.me:$TEST_APP + echo "output: $output" + echo "status: $status" + assert_success + + run /bin/bash -c "cat /var/spool/cron/crontabs/dokku" + echo "output: $output" + echo "status: $status" + assert_failure +} + +@test "(cron) create [single-verbose]" { + run deploy_app python dokku@dokku.me:$TEST_APP template_cron_file_valid + echo "output: $output" + echo "status: $status" + assert_success + + run /bin/bash -c "cat /var/spool/cron/crontabs/dokku" + echo "output: $output" + echo "status: $status" + assert_success + assert_output_contains "python task.py schedule" +} + +@test "(cron) create [single-short]" { + run deploy_app python dokku@dokku.me:$TEST_APP template_cron_file_valid_short + echo "output: $output" + echo "status: $status" + assert_success + + run /bin/bash -c "cat /var/spool/cron/crontabs/dokku" + echo "output: $output" + echo "status: $status" + assert_success + assert_output_contains "python task.py daily" +} + +@test "(cron) create [multiple]" { + run deploy_app python dokku@dokku.me:$TEST_APP template_cron_file_valid_multiple + echo "output: $output" + echo "status: $status" + assert_success + + run /bin/bash -c "cat /var/spool/cron/crontabs/dokku" + echo "output: $output" + echo "status: $status" + assert_success + assert_output_contains "python task.py first" + assert_output_contains "python task.py second" +} + + +template_cron_file_invalid() { + local APP="$1" + local APP_REPO_DIR="$2" + [[ -z "$APP" ]] && local APP="$TEST_APP" + echo "injecting invalid cron app.json -> $APP_REPO_DIR/app.json" + cat <"$APP_REPO_DIR/app.json" +{ + "cron": [ + { + "key": "value" + } + ] +} +EOF +} + +template_cron_file_invalid_schedule() { + local APP="$1" + local APP_REPO_DIR="$2" + [[ -z "$APP" ]] && local APP="$TEST_APP" + echo "injecting invalid cron app.json -> $APP_REPO_DIR/app.json" + cat <"$APP_REPO_DIR/app.json" +{ + "cron": [ + { + "command": "python task.py", + "schedule": "@nonstandard" + } + ] +} +EOF +} + +template_cron_file_invalid_schedule_seconds() { + local APP="$1" + local APP_REPO_DIR="$2" + [[ -z "$APP" ]] && local APP="$TEST_APP" + echo "injecting invalid cron app.json -> $APP_REPO_DIR/app.json" + cat <"$APP_REPO_DIR/app.json" +{ + "cron": [ + { + "command": "python task.py", + "schedule": "0 5 * * * *" + } + ] +} +EOF +} + +template_cron_file_valid() { + local APP="$1" + local APP_REPO_DIR="$2" + [[ -z "$APP" ]] && local APP="$TEST_APP" + echo "injecting valid cron app.json -> $APP_REPO_DIR/app.json" + cat <"$APP_REPO_DIR/app.json" +{ + "cron": [ + { + "command": "python task.py schedule", + "schedule": "5 5 5 5 5" + } + ] +} +EOF +} + +template_cron_file_valid_short() { + local APP="$1" + local APP_REPO_DIR="$2" + [[ -z "$APP" ]] && local APP="$TEST_APP" + echo "injecting valid cron app.json -> $APP_REPO_DIR/app.json" + cat <"$APP_REPO_DIR/app.json" +{ + "cron": [ + { + "command": "python task.py daily", + "schedule": "@daily" + } + ] +} +EOF +} + +template_cron_file_valid_multiple() { + local APP="$1" + local APP_REPO_DIR="$2" + [[ -z "$APP" ]] && local APP="$TEST_APP" + echo "injecting valid cron app.json -> $APP_REPO_DIR/app.json" + cat <"$APP_REPO_DIR/app.json" +{ + "cron": [ + { + "command": "python task.py first", + "schedule": "5 5 5 5 5" + }, + { + "command": "python task.py second", + "schedule": "@daily" + } + ] +} +EOF +} diff --git a/tests/unit/init.bats b/tests/unit/init.bats index 80b1bbf33..5455e0626 100644 --- a/tests/unit/init.bats +++ b/tests/unit/init.bats @@ -28,7 +28,7 @@ teardown() { local APP="zombies-dockerfile-no-tini" deploy_app "$APP" local CIDS=$(get_app_container_ids "$APP") - + run "$DOCKER_BIN" container top "$CIDS" echo "output: $output" assert_output_contains "" "0"