A journey to Kubernetes on Azure

amurleopard

With the ongoing migration to Azure, I would like to share my thoughts regarding one of the biggest challenges we have faced thus far: orchestrating container infrastructure. Many of the Jenkins project’s applications are run as Docker containers, making Kubernetes a logical choice as far as running our containers, but it presents its own set of challenges. For example, what would the workflow from development to production look like?

Before going deeper into the challenges, let’s review the requirements we started with:

<dl>

<dd>

We found it mandatory to keep track of all the infrastructure changes in Git repositories, including secrets, in order to facilitate reviewing, validation, rollback, etc of all infra changes.

</dd>

<dt class="hdlist1">Tests</dt>

<dd>

Infrastructure contributors are geographically distributed and in different timezones. Getting feedback can take time, so we heavily rely on a lot of tests before any changes can be merged.

</dd>

<dt class="hdlist1">Automation</dt>

<dd>

The change submitter is not necessarily the person who will deploy it. Repetitive tasks are error prone and a waste of time. For these reasons, all steps must be automated and stay as simple as possible.

</dd>

</dl>

A high level overview of our "infrastructure as code" workflow would look like:

Infrastructure as Code Workflow

<pre class="nowrap"> __________ _________ ______________ | | | | | | | Changes | ---->| Test |----->| Deployment | || || ^ || | ______________ | | | Validation | |_|</pre>

We identified two possible approaches for implementing our container orchestration with Kubernetes:

The Jenkins Way: Jenkins is triggered by a Git commit, runs the tests, and after validation, Jenkins deploys changes into production.
The Puppet Way: Jenkins is triggered by a Git commit, runs the tests, and after validation, it triggers Puppet to deploy into production.

Let’s discuss these two approaches in detail.

The Jenkins Way

Workflow

<pre class="nowrap"> _________________ ____________________ ______________ | | | | | | | Github: | | Jenkins: | | Jenkins: | | Commit trigger | ---->| Test & Validation | ---->| Deployment | || || |__________|</pre>

In this approach, Jenkins is used to test, validate, and deploy our Kubernetes configuration files. kubectl can be run on a directory and is idempotent. This means that we can run it as often as we want: the result will not change. Theoretically, this is the simplest way. The only thing needed is to run kubectl command each time Jenkins detects changes.

The following Jenkinsfile gives an example of this workflow.

Jenkinsfile

 pipeline { agent any stages { stage('Init'){ steps { sh 'curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl' } } stage('Test'){ steps { sh 'Run tests' } } stage('Deploy'){ steps { sh './kubectl apply -R true -f my_project' } } } }

The devil is in the details of course, and it was not as easy as it looked at first sight.

Order matters

Some resources needed to be deployed before others. A workaround was to use numbers as file names. But this added extra logic at file name level, for example:

project/00-nginx-ingress project/09-www.jenkins.io

Portability

The deployment environments needed to be the same across development machines and the Jenkins host. Although this a well-known problem, it was not easy to solve. The more the project grew, the more our scripts needed additional tools (make, bats, jq gpg, etc). The more tools we used, the more issues appeared because of the different versions used.

Another challenge that emerged when dealing with different environments was: how should we manage environment-specific configurations (dev, prod, etc)? Would it be better to define different configuration files per environment? Perhaps, but this means code duplication, or using file templates which would require more tools (sed, jinja2, erb), and more work.

There wasn’t a golden rule we discovered, and the answer is probably somewhere in between.

In any case, the good news is that a Jenkinsfile provides an easy way to execute tasks from a Docker image, and an image can contain all the necessary tools in our environment. We can even use different Docker images for each stage along the way.

In the following example, I use the my_env Docker image. It contains all the tools needed to test, validate, and deploy changes.

Jenkinsfile

pipeline{ agent { docker{ image 'my_env:1.0' } } options{ buildDiscarder(logRotator(numToKeepStr: '10')) disableConcurrentBuilds() timeout(time: 1, unit: 'HOURS') } triggers{ pollSCM('* * * * *') } stages{ stage('Init'){ steps{ // Init everything required to deploy our infra sh 'make init' } } stage('Test'){ steps{ // Run tests to validate changes sh 'make test' } } stage('Deploy'){ steps{ // Deploy changes in production sh 'make deploy' } } } post{ always { sh 'make notify' } } }

Secret credentials

Managing secrets is a big subject and brings with it many different requirements which are very hard to fulfill. For obvious reasons, we couldn’t publish the credentials used within the infra project. On the other hand, we needed to keep track and share them, particularly for the Jenkins node that deploys our cluster. This means that we needed a way to encrypt or decrypt those credentials depending on permissions, environments, etc. We analyzed two different approaches to handle this:

Storing secrets in a key management tool like Key Vault or Vault and use them like a Kubernetes "secret" type of resource.
→ Unfortunately, these tools are not yet integrated in Kubernetes. But we may come back to this option later. Kubernetes issue: 10439
Publishing and encrypting using a public GPG key.
This means that everybody can encrypt credentials for the infrastructure project but only the owner of the private key can decrypt credentials.
This solution implies:
- Scripting: as secrets need to be decrypted at deployment time.
- Templates: as secret values will change depending on the environment.
  → Each Jenkins node should only have the private key to decrypt secrets associated to its environment.

Scripting

Finally, the system we had built was hard to work with. Our initial Jenkinsfile which only ran one kubectl command slowly become a bunch of scripts to accomodate for:

Resources needing to be updated only in some situations.
Secrets needing to be encrypted/decrypted.
Tests needing to be run.

In the end, the amount of scripts required to deploy the Kubernetes resources started to become unwieldy and we began asking ourselves: "aren’t we re-inventing the wheel?"

The Puppet Way

The Jenkins project already uses Puppet, so we decided to look at using Puppet to orchestrate our container deployment with Kubernetes.

Workflow

<pre class="nowrap"> _________________ ____________________ _____________ | | | | | | | Github: | | Jenkins: | | Puppet: | | Commit trigger | ---->| Test & Validation | ---->| Deployment | || || |_________|</pre>

In this workflow, Puppet is used to template and deploy all Kubernetes configurations files needed to orchestrate our cluster. Puppet is also used to automate basic kubectl operations such as apply or remove for various resources based on file changes.

Puppet workflow

<pre class="nowrap">______________________ | | | Puppet Code: | | . | | ├── apply.pp | | ├── kubectl.pp | | ├── params.pp | | └── resources | | ├── lego.pp | | └── nginx.pp | || | _________________________________ | | | | | Host: Prod orchestrator | | | /home/k8s/ | | | . | | | └── resources | | Puppet generate workspace | ├── lego | └-------------------------------------->| │ ├── configmap.yaml | Puppet apply workspaces' resources on | │ ├── deployment.yaml | ----------------------------------------| │ └── namespace.yaml | | | └── nginx | v | ├── deployment.yaml | ______________ | ├── namespace.yaml | | Azure: | | └── service.yaml | | K8s Cluster | |___________| |_____________|</pre>

The main benefit of this approach is letting Puppet manage the environment and run common tasks. In the following example, we define a Puppet class for Datadog.

Puppet class for resource Datadog

# Deploy datadog resources on kubernetes cluster # Class: profile::kubernetes::resources::datadog # # This class deploy a datadog agent on each kubernetes node # # Parameters: # $apiKey: # Contain datadog api key. # Used in secret template class profile::kubernetes::resources::datadog ( $apiKey = base64('encode', $::datadog_agent::api_key, 'strict') ){ include ::stdlib include profile::kubernetes::params require profile::kubernetes::kubectl file { "${profile::kubernetes::params::resources}/datadog": ensure => 'directory', owner => $profile::kubernetes::params::user, } profile::kubernetes::apply { 'datadog/secret.yaml': parameters => { 'apiKey' => $apiKey }, } profile::kubernetes::apply { 'datadog/daemonset.yaml':} profile::kubernetes::apply { 'datadog/deployment.yaml':} # As secrets change do not trigger pods update, # we must reload pods 'manually' in order to use updated secrets. # If we delete a pod defined by a daemonset, # this daemonset will recreate pods automatically. exec { 'Reload datadog pods': path => ["${profile::kubernetes::params::bin}/"], command => 'kubectl delete pods -l app=datadog', refreshonly => true, environment => ["KUBECONFIG=${profile::kubernetes::params::home}/.kube/config"] , logoutput => true, subscribe => [ Exec['apply datadog/secret.yaml'], Exec['apply datadog/daemonset.yaml'], ], } }

→ More "resources" examples

Let’s compare the Puppet way with the challenges discovered with the Jenkins way.

Order Matters

With Puppet, it becomes easier to define priorities as Puppet provides relationship meta parameters and the function require (see also: Puppet relationships).

In our Datadog example, we can be sure that deployment will respect the following order:

<pre class="nowrap">datadog/secret.yaml -> datadog/daemonset.yaml -> datadog/deployment.yaml</pre>

Currently, our Puppet code only applies configuration when it detects file changes. It would be better to compare local files with the cluster configuration in order to trigger the required updates, but we haven’t found a good way to implement this yet.

Portability

As Puppet is used to configure working environments, it becomes easier to be sure that all tools are present and correctly configured. It’s also easier to replicate environments and run tests on them with tools like RSpec-puppet, Serverspec or Vagrant.

In our Datadog example, we can also easily change the Datadog API key depending on the environment with Hiera.

Secret credentials

As we were already using Hiera GPG with Puppet, we decided to continue to use it, making managing secrets for containers very simple.

Scripting

Of course the Puppet DSL is used, and even if it seems harder at the beginning, Puppet simplifies a lot the management of Kubernetes configuration files.

Conclusion

It was much easier to bootstrap the project with a full CI workflow within Jenkins as long as the Kubernetes project itself stays basic. But as soon as the project grew, and we started deploying different applications with different configurations per environment, it became easier to delegate Kubernetes management to Puppet.

If you have any comments feel free to send a message to Jenkins Infra mailing list.

Thanks

Thanks to Lindsay Vanheyste, Jean Marc Meessen, and Damien Duportal for their feedback.

http://feedproxy.google.com/~r/ContinuousBlog/~3/l0GRF3AYFHk/