Merge pull request #1455 from progrium/1356_mh-safe-stop-kill-event

continue but log an event if/when container retirement fails

docs/development/pluginhooks.md

@@ -664,3 +664,19 @@ APP="$1"; IMAGE_TAG="$2"
 
 some code to remove a docker hub tag because it's not implemented in the CLI....
 ```
+
+### `retire-container-failed`
+
+- Description: Allows you to run commands if/when retiring old containers has failed
+- Invoked by: `dokku deploy`
+- Arguments: `$APP`
+- Example:
+
+```shell
+#!/usr/bin/env bash
+
+set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
+APP="$1"; HOSTNAME=$(hostname -s)
+
+mail -s "$APP containers on $HOSTNAME failed to retire" ops@co.com
+```

dokku

@@ -181,8 +181,12 @@ case "$1" in
         trap '' INT HUP
         sleep $WAIT
         for oldid in $oldids; do
-          docker stop $oldid &> /dev/null
-          docker kill $oldid &> /dev/null  # force a kill as docker seems to not send SIGKILL as the docs would indicate
+          # Attempt to stop, if that fails, then force a kill as docker seems
+          # to not send SIGKILL as the docs would indicate. If that fails, move
+          # on to the next.
+          docker stop $oldid \
+          || docker kill $oldid \
+          || pluginhook retire-container-failed $APP # Trigger pluginhook for event logging
         done
       ) & disown -a
       # Use trap since disown/nohup don't seem to keep child alive

plugins/20_events/retire-container-failed

@@ -0,0 +1 @@
+hook