Restarting Docker containers that never started successfully
from Quail4789@lemmy.ml to selfhosted@lemmy.world on 03 Nov 07:53
https://lemmy.ml/post/22088904

I have some of my Docker containers bind to Tailscale IPs on the host. Even though I edited the Docker service to depend on tailscaled.service and it starts about 10 seconds after it, every once in a while Tailscale interface won’t be ready by the time the containers try to spin up after a reboot.

Since restart policies do not apply to containers that has never started and ran at least for a while, Docker gives me no choice but to restart either those containers or the Docker daemon manually.

Is there a way to have Docker daemon try to restart containers even though they failed on the first try?

A restart policy only takes effect after a container starts successfully. In this case, starting successfully means that the container is up for at least 10 seconds and Docker has started monitoring it. This prevents a container which doesn’t start at all from going into a restart loop.

https://docs.docker.com/engine/containers/start-containers-automatically

#selfhosted

threaded - newest

lemba@discuss.tchncs.de on 03 Nov 08:27 next collapse

I’m happy with autoheal: github.com/willfarrell/docker-autoheal

horse_battery_staple@lemmy.world on 03 Nov 08:27 next collapse

Delay the start of your containers with the tailscale dependency. Are you using required or depends_on in your docker-compose.yml

hatchjs.com/docker-compose-conditionally-start-se…

If you’re using kubernetes you can make the requirements at the pod level

stackoverflow.com/…/container-initialization-orde…

Edit: If using docker-compose.yml you can set a condition on a healthcheck

You can also specify a condition that must be met before the service is started. For example, the following configuration will start the web service only if the db service is healthy:

version: ‘3.7’
services:
web:
image: nginx
depends_on:
– db
condition: service_healthy

The service_healthy condition checks the health of the db service using the docker-compose ps command. If the db service is healthy, the web service will be started.

[deleted] on 03 Nov 14:29 collapse
.
danielquinn@lemmy.ca on 03 Nov 09:30 collapse

You probably want to look into Health Checks. I believe you can tell Docker to “start service B when service A is healthy”, so you can define your health check with a script that depends on Tailscale functioning.

Bakkoda@sh.itjust.works on 03 Nov 13:42 next collapse

I need to figure this out on my immich stack. VM reboot only starts two out of the 4 services. Ty for the info.

just_another_person@lemmy.world on 03 Nov 14:27 next collapse

This is the correct answer, but you need a few things to clarify:

  1. The issue isn’t the Docker system service. Don’t make that depend on Tailscale
  2. Add a healthcheck and restart policy to the container to make it fail when conditions aren’t met, and restart until they are successful
  3. Build in some time tolerance at the service level inside the container to prevent it from flailing if your Tailscale healthchecks don’t pass after they initially start. Don’t rely solely on container health checks to ensure it works properly as that might not always be possible.
horse_battery_staple@lemmy.world on 04 Nov 01:04 collapse

Considering it’s tailscale, one may want to have the service fail though as tailscale is sometimes not used for convenience but security concerns instead.

[deleted] on 03 Nov 14:29 collapse
.