mirror of
https://github.com/dokku/dokku.git
synced 2025-12-29 00:25:08 +01:00
Merge pull request #1056 from joshco/feature/retries
Review: Feature/retries
This commit is contained in:
Jose Diaz-Gonzalez
@@ -107,10 +107,14 @@ A check is a relative URL and may be followed by expected content from the page,
|
||||
/about Our Amazing Team
|
||||
```
|
||||
|
||||
Dokku will wait `DOKKU_CHECKS_WAIT` seconds (default: `5`) before running the checks to give server time to start. For shorter/longer wait, change the `DOKKU_CHECKS_WAIT` environment variable.
|
||||
Dokku will wait `DOKKU_CHECKS_WAIT` seconds (default: `5`) before running the checks to give server time to start. For shorter/longer wait, change the `DOKKU_CHECKS_WAIT` environment variable. This can be overridden in the CHECKS file by setting WAIT=nn.
|
||||
|
||||
Dokku will wait `DOKKU_WAIT_TO_RETIRE` seconds (default: `60`) before stopping the old container such that no existing connections to it are dropped.
|
||||
|
||||
Dokku will retry the checks DOKKU_CHECKS_ATTEMPTS times until the checks are successful or DOKKU_CHECKS_ATTEMPTS is exceeded. In the latter case, the deployment is considered failed. This can be overridden in the CHECKS file by setting ATTEMPTS=nn.
|
||||
|
||||
See [checks-examples.md](checks-examples.md) for examples and output.
|
||||
|
||||
# Removing a deployed app
|
||||
|
||||
SSH onto the server, then execute:
|
||||
|
||||
203
docs/checks-examples.md
Normal file
203
docs/checks-examples.md
Normal file
@@ -0,0 +1,203 @@
|
||||
# Checks Examples
|
||||
|
||||
The CHECKS file may contain empty lines, comments (lines starting with #),
|
||||
settings (NAME=VALUE) and check instructions.
|
||||
|
||||
The format of a check instruction is a path, optionally followed by the
|
||||
expected content. For example:
|
||||
|
||||
/ My Amazing App
|
||||
/stylesheets/index.css .body
|
||||
/scripts/index.js $(function()
|
||||
/images/logo.png
|
||||
|
||||
To check an application that supports multiple hostnames, use relative URLs
|
||||
that include the hostname, for example:
|
||||
|
||||
//admin.example.com Admin Dashboard
|
||||
//static.example.com/logo.png
|
||||
|
||||
You can also specify the protocol to explicitly check HTTPS requests.
|
||||
|
||||
The default behavior is to wait for 5 seconds before running the first check,
|
||||
and timeout each check to 30 minutes.
|
||||
|
||||
By default, checks will be attempted 5 times. (Retried 4 times)
|
||||
|
||||
You can change these by setting WAIT, TIMEOUT and ATTEMPTS to different values, for
|
||||
example:
|
||||
|
||||
WAIT=30 # Wait 1/2 minute
|
||||
TIMEOUT=60 # Timeout after a minute
|
||||
ATTEMPTS=10 # attempt checks 10 times
|
||||
|
||||
# Example: Successful Rails Deployment
|
||||
In this example, a rails applicaiton is successfully deployed to dokku. The initial round of checks fails while the server is starting, but once it starts they succeed and the deployment is successful.
|
||||
ATTEMPTS is set to 6, but the third attempt succeeds.
|
||||
|
||||
## CHECKS file
|
||||
|
||||
````
|
||||
WAIT=5
|
||||
ATTEMPTS=6
|
||||
/check.txt simple_check
|
||||
````
|
||||
|
||||
> check.txt is a text file returning the string 'simple_check'
|
||||
|
||||
## Deploy Output
|
||||
|
||||
> Note: The output has been trimmed for brevity
|
||||
|
||||
````
|
||||
git push dokku master
|
||||
|
||||
-----> Cleaning up...
|
||||
-----> Building myapp from buildstep...
|
||||
-----> Adding BUILD_ENV to build environment...
|
||||
-----> Ruby app detected
|
||||
-----> Compiling Ruby/Rails
|
||||
-----> Using Ruby version: ruby-2.0.0
|
||||
|
||||
.....
|
||||
|
||||
-----> Discovering process types
|
||||
Procfile declares types -> web
|
||||
-----> Releasing myapp...
|
||||
-----> Deploying myapp...
|
||||
-----> Running pre-flight checks
|
||||
-----> Attempt 1/6 Waiting for 5 seconds ...
|
||||
CHECKS expected result:
|
||||
http://localhost/check.txt => "simple_check"
|
||||
!
|
||||
curl: (7) Failed to connect to 172.17.0.155 port 5000: Connection refused
|
||||
! Check attempt 1/6 failed.
|
||||
-----> Attempt 2/6 Waiting for 5 seconds ...
|
||||
CHECKS expected result:
|
||||
http://localhost/check.txt => "simple_check"
|
||||
!
|
||||
curl: (7) Failed to connect to 172.17.0.155 port 5000: Connection refused
|
||||
! Check attempt 2/6 failed.
|
||||
-----> Attempt 3/6 Waiting for 5 seconds ...
|
||||
CHECKS expected result:
|
||||
http://localhost/check.txt => "simple_check"
|
||||
-----> All checks successful!
|
||||
=====> myapp container output:
|
||||
=> Booting Thin
|
||||
=> Rails 4.2.0 application starting in production on http://0.0.0.0:5000
|
||||
=> Run `rails server -h` for more startup options
|
||||
=> Ctrl-C to shutdown server
|
||||
Thin web server (v1.6.3 codename Protein Powder)
|
||||
Maximum connections set to 1024
|
||||
Listening on 0.0.0.0:5000, CTRL+C to stop
|
||||
=====> end myapp container output
|
||||
-----> Running post-deploy
|
||||
-----> Configuring myapp.dokku.example.com...
|
||||
-----> Creating http nginx.conf
|
||||
-----> Running nginx-pre-reload
|
||||
Reloading nginx
|
||||
-----> Shutting down old container in 60 seconds
|
||||
=====> Application deployed:
|
||||
http://myapp.dokku.example.com
|
||||
````
|
||||
|
||||
# Example: Failing Rails Deployment
|
||||
In this example, a Rails application fails to deploy. The reason for the failure is that the postgres database connection fails. The initial checks will fail while we wait for the server to start up, just like in the above example. However, once the server does start accepting connections, we will see an error 500 due to the postgres database connection failure.
|
||||
|
||||
Once the attempts have been exceeded, the deployment fails and we see the container output, which shows the Postgres connection errors.
|
||||
|
||||
## CHECKS file
|
||||
|
||||
````
|
||||
WAIT=5
|
||||
ATTEMPTS=6
|
||||
/
|
||||
````
|
||||
|
||||
> The check to the root url '/' would normally access the database.
|
||||
|
||||
## Deploy Output
|
||||
|
||||
> Note: The output has been trimmed for brevity
|
||||
|
||||
````
|
||||
git push dokku master
|
||||
|
||||
-----> Cleaning up...
|
||||
-----> Building myapp from buildstep...
|
||||
-----> Adding BUILD_ENV to build environment...
|
||||
-----> Ruby app detected
|
||||
-----> Compiling Ruby/Rails
|
||||
-----> Using Ruby version: ruby-2.0.0
|
||||
|
||||
.....
|
||||
|
||||
Discovering process types
|
||||
Procfile declares types -> web
|
||||
Releasing myapp...
|
||||
Deploying myapp...
|
||||
Running pre-flight checks
|
||||
-----> Attempt 1/6 Waiting for 5 seconds ...
|
||||
CHECKS expected result:
|
||||
http://localhost/ => ""
|
||||
!
|
||||
curl: (7) Failed to connect to 172.17.0.188 port 5000: Connection refused
|
||||
! Check attempt 1/6 failed.
|
||||
-----> Attempt 2/6 Waiting for 5 seconds ...
|
||||
CHECKS expected result:
|
||||
http://localhost/ => ""
|
||||
!
|
||||
curl: (7) Failed to connect to 172.17.0.188 port 5000: Connection refused
|
||||
! Check attempt 2/6 failed.
|
||||
-----> Attempt 3/6 Waiting for 5 seconds ...
|
||||
CHECKS expected result:
|
||||
http://localhost/ => ""
|
||||
!
|
||||
curl: (22) The requested URL returned error: 500 Internal Server Error
|
||||
! Check attempt 3/6 failed.
|
||||
-----> Attempt 4/6 Waiting for 5 seconds ...
|
||||
CHECKS expected result:
|
||||
http://localhost/ => ""
|
||||
!
|
||||
curl: (22) The requested URL returned error: 500 Internal Server Error
|
||||
! Check attempt 4/6 failed.
|
||||
-----> Attempt 5/6 Waiting for 5 seconds ...
|
||||
CHECKS expected result:
|
||||
http://localhost/ => ""
|
||||
!
|
||||
curl: (22) The requested URL returned error: 500 Internal Server Error
|
||||
! Check attempt 5/6 failed.
|
||||
-----> Attempt 6/6 Waiting for 5 seconds ...
|
||||
CHECKS expected result:
|
||||
http://localhost/ => ""
|
||||
!
|
||||
curl: (22) The requested URL returned error: 500 Internal Server Error
|
||||
Could not start due to 1 failed checks.
|
||||
! Check attempt 6/6 failed.
|
||||
=====> myapp container output:
|
||||
=> Booting Thin
|
||||
=> Rails 4.2.0 application starting in production on http://0.0.0.0:5000
|
||||
=> Run `rails server -h` for more startup options
|
||||
=> Ctrl-C to shutdown server
|
||||
Thin web server (v1.6.3 codename Protein Powder)
|
||||
Maximum connections set to 1024
|
||||
Listening on 0.0.0.0:5000, CTRL+C to stop
|
||||
Started GET "/" for 172.17.42.1 at 2015-03-26 21:36:47 +0000
|
||||
Is the server running on host "172.17.42.1" and accepting
|
||||
TCP/IP connections on port 5431?
|
||||
PG::ConnectionBad (could not connect to server: Connection refused
|
||||
Is the server running on host "172.17.42.1" and accepting
|
||||
TCP/IP connections on port 5431?
|
||||
):
|
||||
vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `initialize'
|
||||
vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `new'
|
||||
vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `connect'
|
||||
vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb:242:in `initialize'
|
||||
vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb:44:in `new'
|
||||
vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb:44:in `postgresql_connection
|
||||
=====> end myapp container output
|
||||
/usr/local/bin/dokku: line 49: 23409 Killed dokku deploy "$APP"
|
||||
To dokku@dokku.example.com:myapp
|
||||
! [remote rejected] dokku -> master (pre-receive hook declined)
|
||||
error: failed to push some refs to 'dokku@dokku.example.com:myapp'
|
||||
````
|
||||
@@ -24,11 +24,14 @@
|
||||
# The default behavior is to wait for 5 seconds before running the first check,
|
||||
# and timeout each check to 30 minutes.
|
||||
#
|
||||
# You can change these by setting WAIT and TIMEOUT to different values, for
|
||||
# By default, checks will be retried 5 times.
|
||||
|
||||
# You can change these by setting WAIT, TIMEOUT and ATTEMPTS to different values, for
|
||||
# example:
|
||||
#
|
||||
# WAIT=30 # Wait 1/2 minute
|
||||
# TIMEOUT=60 # Timeout after a minute
|
||||
# ATTEMPTS=10 # retry checks 10 times
|
||||
#
|
||||
|
||||
set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
|
||||
@@ -54,6 +57,8 @@ fi
|
||||
WAIT="${DOKKU_CHECKS_WAIT:-5}"
|
||||
# Wait this many seconds (default 30) for each response.
|
||||
TIMEOUT="${DOKKU_CHECKS_TIMEOUT:-30}"
|
||||
# use this number of retries for checks
|
||||
ATTEMPTS="${DOKKU_CHECKS_ATTEMPTS:-5}"
|
||||
|
||||
# try to copy CHECKS from container if not in APP dir & quit gracefully if it doesn't exist
|
||||
# docker cp exits with status 1 when run as non-root user when it tries to chown the file
|
||||
@@ -96,73 +101,92 @@ while read LINE ; do
|
||||
VALUE=${TRIM#*=}
|
||||
[[ "$NAME" = "WAIT" ]] && WAIT=$VALUE
|
||||
[[ "$NAME" = "TIMEOUT" ]] && TIMEOUT=$VALUE
|
||||
[[ "$NAME" = "ATTEMPTS" ]] && ATTEMPTS=$VALUE
|
||||
fi
|
||||
done
|
||||
|
||||
ATTEMPT=0
|
||||
|
||||
dokku_log_info1 "Waiting for $WAIT seconds ..."
|
||||
sleep $WAIT
|
||||
|
||||
|
||||
# -q do not use .curlrc (must come first)
|
||||
# --compressed Test compression handled correctly
|
||||
# --fail Fail on server errors (4xx, 5xx)
|
||||
# --location Follow redirects
|
||||
CURL_OPTIONS="-q --compressed --fail --location --max-time $TIMEOUT"
|
||||
|
||||
|
||||
exec < "$FILENAME"
|
||||
while read CHECK_URL EXPECTED ; do
|
||||
# Ignore empty lines and lines starting with #
|
||||
# shellcheck disable=SC1001
|
||||
[[ -z "$CHECK_URL" || "$CHECK_URL" =~ ^\# ]] && continue
|
||||
# Ignore variables
|
||||
[[ "$CHECK_URL" =~ ^.+= ]] && continue
|
||||
|
||||
if [[ "$CHECK_URL" =~ ^https?: ]] ; then
|
||||
URL_PROTOCOL=${CHECK_URL%:*}
|
||||
CHECK_URL=${CHECK_URL#*:}
|
||||
else
|
||||
URL_PROTOCOL="http"
|
||||
fi
|
||||
|
||||
if [[ "$CHECK_URL" =~ ^//.+ ]] ; then
|
||||
# To test a URL with specific host name, we still make request to localhost,
|
||||
# but we set Host header to $SEND_HOST.
|
||||
#
|
||||
# The pattern is
|
||||
# //SEND_HOST/PATHNAME
|
||||
UNPREFIXED=${CHECK_URL#//}
|
||||
URL_HOSTNAME=${UNPREFIXED%%/*}
|
||||
URL_PATHNAME=${UNPREFIXED#$URL_HOSTNAME}
|
||||
|
||||
HEADERS="-H Host:$URL_HOSTNAME"
|
||||
else
|
||||
URL_HOSTNAME=localhost
|
||||
URL_PATHNAME=$CHECK_URL
|
||||
fi
|
||||
|
||||
# This URL will show up in the messages
|
||||
LOG_URL="$URL_PROTOCOL://$URL_HOSTNAME$URL_PATHNAME"
|
||||
# And how we formulate the CURL request
|
||||
CURL_ARGS="$CURL_OPTIONS $URL_PROTOCOL://$DOKKU_APP_LISTEN_IP:$DOKKU_APP_LISTEN_PORT$URL_PATHNAME $HEADERS"
|
||||
|
||||
dokku_log_verbose "CHECKS expected result:"
|
||||
dokku_log_verbose "$LOG_URL => \"$EXPECTED\""
|
||||
[[ $DOKKU_TRACE ]] && dokku_log_verbose "$ curl $CURL_ARGS"
|
||||
|
||||
# Capture HTTP response or CURL error message
|
||||
if OUTPUT=$(curl -# $CURL_ARGS 2>&1) ; then
|
||||
# OUTPUT contains the HTTP response
|
||||
if [[ ! "$OUTPUT" =~ $EXPECTED ]] ; then
|
||||
dokku_log_fail "$LOG_URL: expected to but did not find: \"$EXPECTED\""
|
||||
exit 1
|
||||
until [[ $SUCCESS == 1 || $ATTEMPT -ge $ATTEMPTS ]]
|
||||
do
|
||||
FAILEDCHECKS=0
|
||||
ATTEMPT=$(( ATTEMPT + 1 ))
|
||||
dokku_log_info1 "Attempt $ATTEMPT/$ATTEMPTS Waiting for $WAIT seconds ..."
|
||||
sleep $WAIT
|
||||
|
||||
|
||||
# -q do not use .curlrc (must come first)
|
||||
# --compressed Test compression handled correctly
|
||||
# --fail Fail on server errors (4xx, 5xx)
|
||||
# --location Follow redirects
|
||||
CURL_OPTIONS="-q --compressed --fail --location --max-time $TIMEOUT"
|
||||
|
||||
|
||||
exec < "$FILENAME"
|
||||
while read CHECK_URL EXPECTED ; do
|
||||
# Ignore empty lines and lines starting with #
|
||||
# shellcheck disable=SC1001
|
||||
[[ -z "$CHECK_URL" || "$CHECK_URL" =~ ^\# ]] && continue
|
||||
# Ignore variables
|
||||
[[ "$CHECK_URL" =~ ^.+= ]] && continue
|
||||
|
||||
if [[ "$CHECK_URL" =~ ^https?: ]] ; then
|
||||
URL_PROTOCOL=${CHECK_URL%:*}
|
||||
CHECK_URL=${CHECK_URL#*:}
|
||||
else
|
||||
URL_PROTOCOL="http"
|
||||
fi
|
||||
|
||||
if [[ "$CHECK_URL" =~ ^//.+ ]] ; then
|
||||
# To test a URL with specific host name, we still make request to localhost,
|
||||
# but we set Host header to $SEND_HOST.
|
||||
#
|
||||
# The pattern is
|
||||
# //SEND_HOST/PATHNAME
|
||||
UNPREFIXED=${CHECK_URL#//}
|
||||
URL_HOSTNAME=${UNPREFIXED%%/*}
|
||||
URL_PATHNAME=${UNPREFIXED#$URL_HOSTNAME}
|
||||
|
||||
HEADERS="-H Host:$URL_HOSTNAME"
|
||||
else
|
||||
URL_HOSTNAME=localhost
|
||||
URL_PATHNAME=$CHECK_URL
|
||||
fi
|
||||
|
||||
# This URL will show up in the messages
|
||||
LOG_URL="$URL_PROTOCOL://$URL_HOSTNAME$URL_PATHNAME"
|
||||
# And how we formulate the CURL request
|
||||
CURL_ARGS="$CURL_OPTIONS $URL_PROTOCOL://$DOKKU_APP_LISTEN_IP:$DOKKU_APP_LISTEN_PORT$URL_PATHNAME $HEADERS"
|
||||
|
||||
dokku_log_verbose "CHECKS expected result:"
|
||||
dokku_log_verbose "$LOG_URL => \"$EXPECTED\""
|
||||
[[ $DOKKU_TRACE ]] && dokku_log_verbose "$ curl $CURL_ARGS"
|
||||
|
||||
# Capture HTTP response or CURL error message
|
||||
if OUTPUT=$(curl -# $CURL_ARGS 2>&1) ; then
|
||||
# OUTPUT contains the HTTP response
|
||||
if [[ ! "$OUTPUT" =~ $EXPECTED ]] ; then
|
||||
dokku_log_warn "$LOG_URL: expected to but did not find: \"$EXPECTED\""
|
||||
FAILEDCHECKS=$(( FAILEDCHECKS + 1 ))
|
||||
fi
|
||||
else
|
||||
# Failed to connect/no response, OUTPUT contains error message
|
||||
dokku_log_warn "$OUTPUT"
|
||||
FAILEDCHECKS=$(( FAILEDCHECKS + 1 ))
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $FAILEDCHECKS -gt 0 ]; then
|
||||
dokku_log_warn "Check attempt $ATTEMPT/$ATTEMPTS failed."
|
||||
SUCCESS=0
|
||||
else
|
||||
# Failed to connect/no response, OUTPUT contains error message
|
||||
dokku_log_fail "$OUTPUT"
|
||||
exit 2
|
||||
SUCCESS=1
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $FAILEDCHECKS -gt 0 ]; then
|
||||
dokku_log_fail "Could not start due to $FAILEDCHECKS failed checks."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
dokku_log_info1 "All checks successful!"
|
||||
|
||||
Reference in New Issue
Block a user