Handle crashing containers by using restart=on-failure policy

Previously a crashed container would stay down, regardless of exit status. In some cases, it may be useful to restart the container. For example, an application may not be correctly implementing their error handling, or the crash may be caused by a transient error.

By setting the restart policy to `on-failure:N` - where N is a number of max restarts - we can help developers guard against crashing applications. Note that this is not a replacement for proper error handling, nor does this include notifications to a developer when a container is restarted. Those patterns should be implemented application side, or via a feature request to docker.

The value is configurable at the app-level by setting DOKKU_RESTART_LIMIT to a number. By default, containers will be restarted a max of 10 times. If a container crashes during the check-deploy plugin trigger, then the deploy will be marked as a failure.

Closes #216
Closes #398
Closes #1327
This commit is contained in:
Jose Diaz-Gonzalez
2015-09-16 23:47:33 -07:00
parent 4ad955adcb
commit ab8eb5a727
2 changed files with 10 additions and 0 deletions

4
dokku
View File

@@ -77,6 +77,9 @@ case "$1" in
DOKKU_SCALE_FILE="$DOKKU_ROOT/$APP/DOKKU_SCALE"
oldids=$(get_app_container_ids $APP)
DOKKU_RESTART_LIMIT=$(dokku config:get $APP DOKKU_RESTART_LIMIT || echo 10)
DOKKU_RESTART_LIMIT=${DOKKU_RESTART_LIMIT:=10}
while read line || [[ -n "$line" ]]; do
TRIM=${line%#*}
PROC_TYPE=${TRIM%%=*}
@@ -91,6 +94,7 @@ case "$1" in
# start the app
DOCKER_ARGS=" -e DYNO='$PROC_TYPE.$CONTAINER_INDEX' "
DOCKER_ARGS=" --restart=on-failure:$DOKKU_RESTART_LIMIT"
DOCKER_ARGS+=$(: | plugn trigger docker-args-deploy $APP $IMAGE_TAG)
[[ "$DOKKU_TRACE" ]] && DOCKER_ARGS+=" -e TRACE=true "
BIND_EXTERNAL=$(plugn trigger bind-external-ip $APP)

View File

@@ -97,6 +97,12 @@ if [[ ! -s "${TMPDIR}/CHECKS" ]] || [[ "$DOKKU_APP_CONTAINER_TYPE" != "web" ]];
sleep $DOKKU_DEFAULT_CHECKS_WAIT
! (is_container_running $DOKKU_APP_CONTAINER_ID) && dokku_log_fail "App container failed to start!!"
container_restarts=$(docker inspect -f "{{ .RestartCount }}" $DOKKU_APP_CONTAINER_ID)
if [[ $container_restarts -ne 0 ]]; then
docker stop "$DOKKU_APP_CONTAINER_ID" || true
dokku_log_fail "App container failed to start!!"
fi
dokku_log_info1 "Default container check successful!" && exit 0
fi