Docker Image: new Exit and Restart Behavior for Controllers

[amurleopard]

The Jenkins project provides Docker images for controllers (and more). Beginning with Jenkins 2.344 released April 18, 2022 and Jenkins 2.332.3 released May 04, 2022, the behavior of the "Exit" and "Restart" lifecycle of the controller image jenkins/jenkins changed.

TL;DR; Ensure that you have a Container Restart Policy For quick readers: check the How to Add a Container Restart Policy section for immediate actions to be taken.

Application Lifecycle in a Container World

Jenkins previously restarted itself after upgrading plugins or via the /restart or /safeRestart endpoints by calling exec(2). This was fragile and exposed users to a variety of bugs.

The ExitLifecycle implementation allows the main process to exit normally before starting it again from scratch, relying on the exit code to signal to the container orchestrator that the container ought to be restarted rather than to remain shut down.

Avoiding exec(2) from Java code, with all the associated complexity of closing file handles and making low-level system calls via JNA, eliminates exposure to a category of bugs.

You can read more about this on the following issues: jenkinsci/jenkins#5899 and jenkinsci/docker#1268.

Delegate Restarts to the Container Engine

Since the Jenkins Weekly 2.344 and the Jenkins LTS 2.332.3, the Docker image for Jenkins controller uses the new lifecycle ExitLifecycle by default.

It means that when the Jenkins controller triggers any "restart" or "exit" event, then its container exits.

If you are running Jenkins in a controller as a container, you should add a "container restart policy" different than none to ensure that the container is restarted automatically.

If you are running Jenkins in production in a container, chances are great that you already have a restart policy set and the change will be transparent. But we updated the jenkinsci/docker documentation to mention that the flag --restart=on-failure is set when starting a Jenkins controller container.

How to Add a Container Restart Policy

If you do not have a container restart policy defined, please find the different following methods depending on your container orchestrator:

• Docker Engine, Docker Swarm: add the flag --restart=on-failure to the docker run <…​> jenkins/jenkins command.

  No need to create a new container: you can use the command docker update to update the restart policy of an existing container.

• Docker Compose: update your docker-compose.yml file to specify the restart: on-failure keyword (or the deploy.restart_policy if using docker compose deploy to a Docker Swarm cluster)

• Kubernetes:

  • If you are using the official Jenkins helm-chart, make sure that you are using at least the version 3.10.3 of the Helm chart and you’re good to go, as per jenkinsci/helm-charts#529

  • Otherwise, ensure that you configure the Deployment for Jenkins to specify the OnFailure restart policy for the pods. Please note that it’s the default policy if unspecified.

• Hashicorp’s Nomad: Ensure that you specify a non-empty restart stanza. Please note that the default stanza behavior behaves as expected: Jenkins is restarted.

https://www.jenkins.io/blog/2022/05/27/docker-image-new-lifecycle/