Kubernetes News

• 

  Blog: Kubernetes Removals and Major Changes In v1.27
  • sorangutan  

  1
  1
  Posts
  28
  Views

  

  Author: Harshita Sao

  As Kubernetes develops and matures, features may be deprecated, removed, or replaced with better ones for the project's overall health. Based on the information available at this point in the v1.27 release process, which is still ongoing and can introduce additional changes, this article identifies and describes some of the planned changes for the Kubernetes v1.27 release.

  A note about the k8s.gcr.io redirect to registry.k8s.io

  To host its container images, the Kubernetes project uses a community-owned image registry called registry.k8s.io. On March 20th, all traffic from the out-of-date k8s.gcr.io registry will be redirected to registry.k8s.io. The deprecated k8s.gcr.io registry will eventually be phased out.

  What does this change mean?

  • If you are a subproject maintainer, you must update your manifests and Helm charts to use the new registry.

  • The v1.27 Kubernetes release will not be published to the old registry.

  • From April, patch releases for v1.24, v1.25, and v1.26 will no longer be published to the old registry.

  We have a blog post with all the information about this change and what to do if it impacts you.

  The Kubernetes API Removal and Deprecation process

  The Kubernetes project has a well-documented deprecation policy for features. This policy states that stable APIs may only be deprecated when a newer, stable version of that same API is available and that APIs have a minimum lifetime for each stability level. A deprecated API has been marked for removal in a future Kubernetes release, it will continue to function until removal (at least one year from the deprecation), but usage will result in a warning being displayed. Removed APIs are no longer available in the current version, at which point you must migrate to using the replacement.

  • Generally available (GA) or stable API versions may be marked as deprecated but must not be removed within a major version of Kubernetes.

  • Beta or pre-release API versions must be supported for 3 releases after the deprecation.

  • Alpha or experimental API versions may be removed in any release without prior deprecation notice.

  Whether an API is removed as a result of a feature graduating from beta to stable or because that API simply did not succeed, all removals comply with this deprecation policy. Whenever an API is removed, migration options are communicated in the documentation.

  API removals, and other changes for Kubernetes v1.27

  Removal of storage.k8s.io/v1beta1 from CSIStorageCapacity

  The CSIStorageCapacity API supports exposing currently available storage capacity via CSIStorageCapacity objects and enhances the scheduling of pods that use CSI volumes with late binding. The storage.k8s.io/v1beta1 API version of CSIStorageCapacity was deprecated in v1.24, and it will no longer be served in v1.27.

  Migrate manifests and API clients to use the storage.k8s.io/v1 API version, available since v1.24. All existing persisted objects are accessible via the new API.

  Refer to the Storage Capacity Constraints for Pod Scheduling KEP for more information.

  Kubernetes v1.27 is not removing any other APIs; however several other aspects are going to be removed. Read on for details.

  Support for deprecated seccomp annotations

  In Kubernetes v1.19, the seccomp (secure computing mode) support graduated to General Availability (GA). This feature can be used to increase the workload security by restricting the system calls for a Pod (applies to all containers) or single containers.

  The support for the alpha seccomp annotations seccomp.security.alpha.kubernetes.io/pod and container.seccomp.security.alpha.kubernetes.io were deprecated since v1.19, now have been completely removed. The seccomp fields are no longer auto-populated when pods with seccomp annotations are created. Pods should use the corresponding pod or container securityContext.seccompProfile field instead.

  Removal of several feature gates for volume expansion

  The following feature gates for volume expansion GA features will be removed and must no longer be referenced in --feature-gates flags:

  ExpandCSIVolumes

  Enable expanding of CSI volumes.

  ExpandInUsePersistentVolumes

  Enable expanding in-use PVCs.

  ExpandPersistentVolumes

  Enable expanding of persistent volumes.

  Removal of --master-service-namespace command line argument

  The kube-apiserver accepts a deprecated command line argument, --master-service-namespace, that specified where to create the Service named kubernetes to represent the API server. Kubernetes v1.27 will remove that argument, which has been deprecated since the v1.26 release.

  Removal of the ControllerManagerLeaderMigration feature gate

  Leader Migration provides a mechanism in which HA clusters can safely migrate "cloud-specific" controllers between the kube-controller-manager and the cloud-controller-manager via a shared resource lock between the two components while upgrading the replicated control plane.

  The ControllerManagerLeaderMigration feature, GA since v1.24, is unconditionally enabled and for the v1.27 release the feature gate option will be removed. If you're setting this feature gate explicitly, you'll need to remove that from command line arguments or configuration files.

  Removal of --enable-taint-manager command line argument

  The kube-controller-manager command line argument --enable-taint-manager is deprecated, and will be removed in Kubernetes v1.27. The feature that it supports, taint based eviction, is already enabled by default and will continue to be implicitly enabled when the flag is removed.

  Removal of --pod-eviction-timeout command line argument

  The deprecated command line argument --pod-eviction-timeout will be removed from the kube-controller-manager.

  Removal of the CSI Migration feature gate

  The CSI migration programme allows moving from in-tree volume plugins to out-of-tree CSI drivers. CSI migration is generally available since Kubernetes v1.16, and the associated CSIMigration feature gate will be removed in v1.27.

  Removal of CSIInlineVolume feature gate

  The CSI Ephemeral Volume feature allows CSI volumes to be specified directly in the pod specification for ephemeral use cases. They can be used to inject arbitrary states, such as configuration, secrets, identity, variables or similar information, directly inside pods using a mounted volume. This feature graduated to GA in v1.25. Hence, the feature gate CSIInlineVolume will be removed in the v1.27 release.

  Removal of EphemeralContainers feature gate

  Ephemeral containers graduated to GA in v1.25. These are containers with a temporary duration that executes within namespaces of an existing pod. Ephemeral containers are typically initiated by a user in order to observe the state of other pods and containers for troubleshooting and debugging purposes. For Kubernetes v1.27, API support for ephemeral containers is unconditionally enabled; the EphemeralContainers feature gate will be removed.

  Removal of LocalStorageCapacityIsolation feature gate

  The Local Ephemeral Storage Capacity Isolation feature moved to GA in v1.25. The feature provides support for capacity isolation of local ephemeral storage between pods, such as emptyDir volumes, so that a pod can be hard limited in its consumption of shared resources. The kubelet will evicting Pods if consumption of local ephemeral storage exceeds the configured limit. The feature gate, LocalStorageCapacityIsolation, will be removed in the v1.27 release.

  Removal of NetworkPolicyEndPort feature gate

  The v1.25 release of Kubernetes promoted endPort in NetworkPolicy to GA. NetworkPolicy providers that support the endPort field that can be used to specify a range of ports to apply a NetworkPolicy. Previously, each NetworkPolicy could only target a single port. So the feature gate NetworkPolicyEndPort will be removed in this release.

  Please be aware that endPort field must be supported by the Network Policy provider. If your provider does not support endPort, and this field is specified in a Network Policy, the Network Policy will be created covering only the port field (single port).

  Removal of StatefulSetMinReadySeconds feature gate

  For a pod that is part of a StatefulSet, Kubernetes can mark the Pod ready only if Pod is available (and passing checks) for at least the period you specify in minReadySeconds. The feature became generally available in Kubernetes v1.25, and the StatefulSetMinReadySeconds feature gate will be locked to true and removed in the v1.27 release.

  Removal of IdentifyPodOS feature gate

  You can specify the operating system for a Pod, and the feature support for that is stable since the v1.25 release. The IdentifyPodOS feature gate will be removed for Kubernetes v1.27.

  Removal of DaemonSetUpdateSurge feature gate

  The v1.25 release of Kubernetes also stabilised surge support for DaemonSet pods, implemented in order to minimize DaemonSet downtime during rollouts. The DaemonSetUpdateSurge feature gate will be removed in Kubernetes v1.27.

  Looking ahead

  The official list of API removals planned for Kubernetes v1.29 includes:

  • The flowcontrol.apiserver.k8s.io/v1beta2 API version of FlowSchema and PriorityLevelConfiguration will no longer be served in v1.29.

  Want to know more?

  Deprecations are announced in the Kubernetes release notes. You can see the announcements of pending deprecations in the release notes for:

  • Kubernetes v1.23

  • Kubernetes v1.24

  • Kubernetes v1.25

  • Kubernetes v1.26

  We will formally announce the deprecations that come with Kubernetes v1.27 as part of the CHANGELOG for that release.

  For information on the process of deprecation and removal, check out the official Kubernetes deprecation policy document.

  
  
  https://kubernetes.io/blog/2023/03/17/upcoming-changes-in-kubernetes-v1-27/
• 

  Blog: k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know
  • sorangutan  

  1
  1
  Posts
  17
  Views

  

  Authors: Bob Killen (Google), Davanum Srinivas (AWS), Chris Short (AWS), Frederico Muñoz (SAS Institute), Tim Bannister (The Scale Factory), Ricky Sadowski (AWS), Grace Nguyen (Expo), Mahamed Ali (Rackspace Technology), Mars Toktonaliev (independent), Laura Santamaria (Dell), Kat Cosgrove (Dell)

  On Monday, March 20th, the k8s.gcr.io registry will be redirected to the community owned registry, registry.k8s.io .

  TL;DR: What you need to know about this change

  • On Monday, March 20th, traffic from the older k8s.gcr.io registry will be redirected to registry.k8s.io with the eventual goal of sunsetting k8s.gcr.io.
  • If you run in a restricted environment, and apply strict domain name or IP address access policies limited to k8s.gcr.io, the image pulls will not function after k8s.gcr.io starts redirecting to the new registry. 
  • A small subset of non-standard clients do not handle HTTP redirects by image registries, and will need to be pointed directly at registry.k8s.io.
  • The redirect is a stopgap to assist users in making the switch. The deprecated k8s.gcr.io registry will be phased out at some point. Please update your manifests as soon as possible to point to registry.k8s.io.
  • If you host your own image registry, you can copy images you need there as well to reduce traffic to community owned registries.

  If you think you may be impacted, or would like to know more about this change, please keep reading.

  Why did Kubernetes change to a different image registry?

  k8s.gcr.io is hosted on a custom Google Container Registry (GCR) domain that was set up solely for the Kubernetes project. This has worked well since the inception of the project, and we thank Google for providing these resources, but today, there are other cloud providers and vendors that would like to host images to provide a better experience for the people on their platforms. In addition to Google’s renewed commitment to donate $3 million to support the project's infrastructure last year, Amazon Web Services announced a matching donation during their Kubecon NA 2022 keynote in Detroit. This will provide a better experience for users (closer servers = faster downloads) and will reduce the egress bandwidth and costs from GCR at the same time.

  For more details on this change, check out registry.k8s.io: faster, cheaper and Generally Available (GA).

  Why is a redirect being put in place?

  The project switched to registry.k8s.io last year with the 1.25 release; however, most of the image pull traffic is still directed at the old endpoint k8s.gcr.io. This has not been sustainable for us as a project, as it is not utilizing the resources that have been donated to the project from other providers, and we are in the danger of running out of funds due to the cost of serving this traffic.

  A redirect will enable the project to take advantage of these new resources, significantly reducing our egress bandwidth costs. We only expect this change to impact a small subset of users running in restricted environments or using very old clients that do not respect redirects properly.

  What images will be impacted?

  ALL images on k8s.gcr.io will be impacted by this change. k8s.gcr.io hosts many images beyond Kubernetes releases. A large number of Kubernetes subprojects host their images there as well. Some examples include the dns/k8s-dns-node-cache, ingress-nginx/controller, and node-problem-detector/node-problem-detector images.

  What will happen to k8s.gcr.io?

  Separate from the the redirect, k8s.gcr.io will be frozen and will not be updated with new images after April 3rd, 2023. k8s.gcr.io will not get any new releases, patches, or security updates. It will continue to remain available to help people migrate, but it WILL be phased out entirely in the future.

  I run in a restricted environment. What should I do?

  For impacted users that run in a restricted environment, the best option is to copy over the required images to a private registry or configure a pull-through cache in their registry.

  There are several tools to copy images between registries; crane is one of those tools, and images can be copied to a private registry by using crane copy SRC DST. There are also vendor-specific tools, like e.g. Google’s gcrane, that perform a similar function but are streamlined for their platform.

  How can I check registry.k8s.io is accessible from my cluster?

  To test connectivity to registry.k8s.io and being able to pull images from there, here is a sample command that can be executed in the namespace of your choosing:

  kubectl run hello-world --tty --rm -i --image=registry.k8s.io/busybox:latest sh
  

  When you run the command above, here’s what to expect when things work correctly:

  $ kubectl run hello-world --tty --rm -i --image=registry.k8s.io/busybox:latest sh
  If you don't see a command prompt, try pressing enter.
  / # exit
  Session ended, resume using 'kubectl attach hello-world -c hello-world -i -t' command when the pod is running
  pod "hello-world" deleted
  

  What kind of errors will I see if I’m impacted?

  Errors may depend on what kind of container runtime you are using, and what endpoint you are routed to, but it should present such as ErrImagePull, ImagePullBackOff, or a container failing to be created with the warning FailedCreatePodSandBox.

  Below is an example error message showing a proxied deployment failing to pull due to an unknown certificate:

  FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = Error response from daemon: Head “https://us-west1-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: x509: certificate signed by unknown authority
  

  How can I find which images are using the legacy registry, and fix them?

  Option 1: See the one line kubectl command in our earlier blog post:

  kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" |\
  tr -s '[[:space:]]' '\n' |\
  sort |\
  uniq -c
  

  Option 2: A kubectl krew plugin has been developed called community-images, that will scan and report any images using the k8s.gcr.io endpoint.

  If you have krew installed, you can install it with:

  kubectl krew install community-images
  

  and generate a report with:

  kubectl community-images
  

  For alternate methods of install and example output, check out the repo: kubernetes-sigs/community-images.

  Option 3: If you do not have access to a cluster directly, or manage many clusters - the best way is to run a search over your manifests and charts for "k8s.gcr.io".

  Option 4: If you wish to prevent k8s.gcr.io based images from running in your cluster, example policies for Gatekeeper and Kyverno are available in the AWS EKS Best Practices repository that will block them from being pulled. You can use these third-party policies with any Kubernetes cluster.

  Option 5: As a LAST possible option, you can use a Mutating Admission Webhook to change the image address dynamically. This should only be considered a stopgap till your manifests have been updated. You can find a (third party) Mutating Webhook and Kyverno policy in k8s-gcr-quickfix.

  I still have questions, where should I go?

  For more information on registry.k8s.io and why it was developed, see registry.k8s.io: faster, cheaper and Generally Available.

  If you would like to know more about the image freeze and the last images that will be available there, see the blog post: k8s.gcr.io Image Registry Will Be Frozen From the 3rd of April 2023.

  Information on the architecture of registry.k8s.io and its request handling decision tree can be found in the kubernetes/registry.k8s.io repo.

  If you believe you have encountered a bug with the new registry or the redirect, please open an issue in the kubernetes/registry.k8s.io repo. Please check if there is an issue already open similar to what you are seeing before you create a new issue.

  
  
  https://kubernetes.io/blog/2023/03/10/image-registry-redirect/
• 

  Blog: Forensic container analysis
  • sorangutan  

  1
  1
  Posts
  23
  Views

  

  Authors: Adrian Reber (Red Hat)

  In my previous article, Forensic container checkpointing in Kubernetes, I introduced checkpointing in Kubernetes and how it has to be setup and how it can be used. The name of the feature is Forensic container checkpointing, but I did not go into any details how to do the actual analysis of the checkpoint created by Kubernetes. In this article I want to provide details how the checkpoint can be analyzed.

  Checkpointing is still an alpha feature in Kubernetes and this article wants to provide a preview how the feature might work in the future.

  Preparation

  Details about how to configure Kubernetes and the underlying CRI implementation to enable checkpointing support can be found in my Forensic container checkpointing in Kubernetes article.

  As an example I prepared a container image (quay.io/adrianreber/counter:blog) which I want to checkpoint and then analyze in this article. This container allows me to create files in the container and also store information in memory which I later want to find in the checkpoint.

  To run that container I need a pod, and for this example I am using the following Pod manifest:

  apiVersion: v1
  kind: Pod
  metadata:
   name: counters
  spec:
   containers:
   - name: counter
   image: quay.io/adrianreber/counter:blog
  

  This results in a container called counter running in a pod called counters.

  Once the container is running I am performing following actions with that container:

  $ kubectl get pod counters --template '{{.status.podIP}}'
  10.88.0.25
  $ curl 10.88.0.25:8088/create?test-file
  $ curl 10.88.0.25:8088/secret?RANDOM_1432_KEY
  $ curl 10.88.0.25:8088
  

  The first access creates a file called test-file with the content test-file in the container and the second access stores my secret information (RANDOM_1432_KEY) somewhere in the container's memory. The last access just adds an additional line to the internal log file.

  The last step before I can analyze the checkpoint it to tell Kubernetes to create the checkpoint. As described in the previous article this requires access to the kubelet only checkpoint API endpoint.

  For a container named counter in a pod named counters in a namespace named default the kubelet API endpoint is reachable at:

  # run this on the node where that Pod is executing
  curl -X POST "https://localhost:10250/checkpoint/default/counters/counter"
  

  For completeness the following curl command-line options are necessary to have curl accept the kubelet's self signed certificate and authorize the use of the kubelet checkpoint API:

  --insecure --cert /var/run/kubernetes/client-admin.crt --key /var/run/kubernetes/client-admin.key
  

  Once the checkpointing has finished the checkpoint should be available at /var/lib/kubelet/checkpoints/checkpoint-<pod-name>_<namespace-name>-<container-name>-<timestamp>.tar

  In the following steps of this article I will use the name checkpoint.tar when analyzing the checkpoint archive.

  Checkpoint archive analysis using checkpointctl

  To get some initial information about the checkpointed container I am using the tool checkpointctl like this:

  $ checkpointctl show checkpoint.tar --print-stats
  +-----------+----------------------------------+--------------+---------+---------------------+--------+------------+------------+-------------------+
  | CONTAINER | IMAGE | ID | RUNTIME | CREATED | ENGINE | IP | CHKPT SIZE | ROOT FS DIFF SIZE |
  +-----------+----------------------------------+--------------+---------+---------------------+--------+------------+------------+-------------------+
  | counter | quay.io/adrianreber/counter:blog | 059a219a22e5 | runc | 2023-03-02T06:06:49 | CRI-O | 10.88.0.23 | 8.6 MiB | 3.0 KiB |
  +-----------+----------------------------------+--------------+---------+---------------------+--------+------------+------------+-------------------+
  CRIU dump statistics
  +---------------+-------------+--------------+---------------+---------------+---------------+
  | FREEZING TIME | FROZEN TIME | MEMDUMP TIME | MEMWRITE TIME | PAGES SCANNED | PAGES WRITTEN |
  +---------------+-------------+--------------+---------------+---------------+---------------+
  | 100809 us | 119627 us | 11602 us | 7379 us | 7800 | 2198 |
  +---------------+-------------+--------------+---------------+---------------+---------------+
  

  This gives me already some information about the checkpoint in that checkpoint archive. I can see the name of the container, information about the container runtime and container engine. It also lists the size of the checkpoint (CHKPT SIZE). This is mainly the size of the memory pages included in the checkpoint, but there is also information about the size of all changed files in the container (ROOT FS DIFF SIZE).

  The additional parameter --print-stats decodes information in the checkpoint archive and displays them in the second table (CRIU dump statistics). This information is collected during checkpoint creation and gives an overview how much time CRIU needed to checkpoint the processes in the container and how many memory pages were analyzed and written during checkpoint creation.

  Digging deeper

  With the help of checkpointctl I am able to get some high level information about the checkpoint archive. To be able to analyze the checkpoint archive further I have to extract it. The checkpoint archive is a tar archive and can be extracted with the help of tar xf checkpoint.tar.

  Extracting the checkpoint archive will result in following files and directories:

  • bind.mounts - this file contains information about bind mounts and is needed during restore to mount all external files and directories at the right location
  • checkpoint/ - this directory contains the actual checkpoint as created by CRIU
  • config.dump and spec.dump - these files contain metadata about the container which is needed during restore
  • dump.log - this file contains the debug output of CRIU created during checkpointing
  • stats-dump - this file contains the data which is used by checkpointctl to display dump statistics (--print-stats)
  • rootfs-diff.tar - this file contains all changed files on the container's file-system

  File-system changes - rootfs-diff.tar

  The first step to analyze the container's checkpoint further is to look at the files that have changed in my container. This can be done by looking at the file rootfs-diff.tar:

  $ tar xvf rootfs-diff.tar
  home/counter/logfile
  home/counter/test-file
  

  Now the files that changed in the container can be studied:

  $ cat home/counter/logfile
  10.88.0.1 - - [02/Mar/2023 06:07:29] "GET /create?test-file HTTP/1.1" 200 -
  10.88.0.1 - - [02/Mar/2023 06:07:40] "GET /secret?RANDOM_1432_KEY HTTP/1.1" 200 -
  10.88.0.1 - - [02/Mar/2023 06:07:43] "GET / HTTP/1.1" 200 -
  $ cat home/counter/test-file
  test-file 
  

  Compared to the container image (quay.io/adrianreber/counter:blog) this container is based on, I can see that the file logfile contains information about all access to the service the container provides and the file test-file was created just as expected.

  With the help of rootfs-diff.tar it is possible to inspect all files that were created or changed compared to the base image of the container.

  Analyzing the checkpointed processes - checkpoint/

  The directory checkpoint/ contains data created by CRIU while checkpointing the processes in the container. The content in the directory checkpoint/ consists of different image files which can be analyzed with the help of the tool CRIT which is distributed as part of CRIU.

  First lets get an overview of the processes inside of the container:

  $ crit show checkpoint/pstree.img | jq .entries[].pid
  1
  7
  8
  

  This output means that I have three processes inside of the container's PID namespace with the PIDs: 1, 7, 8

  This is only the view from the inside of the container's PID namespace. During restore exactly these PIDs will be recreated. From the outside of the container's PID namespace the PIDs will change after restore.

  The next step is to get some additional information about these three processes:

  $ crit show checkpoint/core-1.img | jq .entries[0].tc.comm
  "bash"
  $ crit show checkpoint/core-7.img | jq .entries[0].tc.comm
  "counter.py"
  $ crit show checkpoint/core-8.img | jq .entries[0].tc.comm
  "tee"
  

  This means the three processes in my container are bash, counter.py (a Python interpreter) and tee. For details about the parent child relations of these processes there is more data to be analyzed in checkpoint/pstree.img.

  Let's compare the so far collected information to the still running container:

  $ crictl inspect --output go-template --template "{{(index .info.pid)}}" 059a219a22e56
  722520
  $ ps auxf | grep -A 2 722520
  fedora 722520 \_ bash -c /home/counter/counter.py 2>&1 | tee /home/counter/logfile
  fedora 722541 \_ /usr/bin/python3 /home/counter/counter.py
  fedora 722542 \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /home/counter/logfile
  $ cat /proc/722520/comm
  bash
  $ cat /proc/722541/comm
  counter.py
  $ cat /proc/722542/comm
  tee
  

  In this output I am first retrieving the PID of the first process in the container and then I am looking for that PID and child processes on the system where the container is running. I am seeing three processes and the first one is "bash" which is PID 1 inside of the containers PID namespace. Then I am looking at /proc/<PID>/comm and I can find the exact same value as in the checkpoint image.

  Important to remember is that the checkpoint will contain the view from within the container's PID namespace because that information is important to restore the processes.

  One last example of what crit can tell us about the container is the information about the UTS namespace:

  $ crit show checkpoint/utsns-12.img
  {
   "magic": "UTSNS",
   "entries": [
   {
   "nodename": "counters",
   "domainname": "(none)"
   }
   ]
  }
  

  This tells me that the hostname inside of the UTS namespace is counters.

  For every resource CRIU collected during checkpointing the checkpoint/ directory contains corresponding image files which can be analyzed with the help of crit.

  Looking at the memory pages

  In addition to the information from CRIU that can be decoded with the help of CRIT, there are also files containing the raw memory pages written by CRIU to disk:

  $ ls checkpoint/pages-*
  checkpoint/pages-1.img checkpoint/pages-2.img checkpoint/pages-3.img
  

  When I initially used the container I stored a random key (RANDOM_1432_KEY) somewhere in the memory. Let see if I can find it:

  $ grep -ao RANDOM_1432_KEY checkpoint/pages-*
  checkpoint/pages-2.img:RANDOM_1432_KEY
  

  And indeed, there is my data. This way I can easily look at the content of all memory pages of the processes in the container, but it is also important to remember that anyone that can access the checkpoint archive has access to all information that was stored in the memory of the container's processes.

  Using gdb for further analysis

  Another possibility to look at the checkpoint images is gdb. The CRIU repository contains the script coredump which can convert a checkpoint into a coredump file:

  $ /home/criu/coredump/coredump-python3
  $ ls -al core*
  core.1 core.7 core.8
  

  Running the coredump-python3 script will convert the checkpoint images into one coredump file for each process in the container. Using gdb I can also look at the details of the processes:

  $ echo info registers | gdb --core checkpoint/core.1 -q
  
  [New LWP 1]
  
  Core was generated by `bash -c /home/counter/counter.py 2>&1 | tee /home/counter/logfile'.
  
  #0 0x00007fefba110198 in ?? ()
  (gdb)
  rax 0x3d 61
  rbx 0x8 8
  rcx 0x7fefba11019a 140667595587994
  rdx 0x0 0
  rsi 0x7fffed9c1110 140737179816208
  rdi 0xffffffff 4294967295
  rbp 0x1 0x1
  rsp 0x7fffed9c10e8 0x7fffed9c10e8
  r8 0x1 1
  r9 0x0 0
  r10 0x0 0
  r11 0x246 582
  r12 0x0 0
  r13 0x7fffed9c1170 140737179816304
  r14 0x0 0
  r15 0x0 0
  rip 0x7fefba110198 0x7fefba110198
  eflags 0x246 [ PF ZF IF ]
  cs 0x33 51
  ss 0x2b 43
  ds 0x0 0
  es 0x0 0
  fs 0x0 0
  gs 0x0 0
  

  In this example I can see the value of all registers as they were during checkpointing and I can also see the complete command-line of my container's PID 1 process: bash -c /home/counter/counter.py 2>&1 | tee /home/counter/logfile

  Summary

  With the help of container checkpointing, it is possible to create a checkpoint of a running container without stopping the container and without the container knowing that it was checkpointed. The result of checkpointing a container in Kubernetes is a checkpoint archive; using different tools like checkpointctl, tar, crit and gdb the checkpoint can be analyzed. Even with simple tools like grep it is possible to find information in the checkpoint archive.

  The different examples I have shown in this article how to analyze a checkpoint are just the starting point. Depending on your requirements it is possible to look at certain things in much more detail, but this article should give you an introduction how to start the analysis of your checkpoint.

  How do I get involved?

  You can reach SIG Node by several means:

  • Slack: #sig-node
  • Slack: #sig-security
  • Mailing list

  
  
  https://kubernetes.io/blog/2023/03/10/forensic-container-analysis/
• 

  Blog: Introducing KWOK: Kubernetes WithOut Kubelet
  • sorangutan  

  1
  1
  Posts
  18
  Views

  

  Author: Shiming Zhang (DaoCloud), Wei Huang (Apple), Yibo Zhuang (Apple)

  

  Have you ever wondered how to set up a cluster of thousands of nodes just in seconds, how to simulate real nodes with a low resource footprint, and how to test your Kubernetes controller at scale without spending much on infrastructure?

  If you answered "yes" to any of these questions, then you might be interested in KWOK, a toolkit that enables you to create a cluster of thousands of nodes in seconds.

  What is KWOK?

  KWOK stands for Kubernetes WithOut Kubelet. So far, it provides two tools:

  kwok

  kwok is the cornerstone of this project, responsible for simulating the lifecycle of fake nodes, pods, and other Kubernetes API resources.

  kwokctl

  kwokctl is a CLI tool designed to streamline the creation and management of clusters, with nodes simulated by kwok.

  Why use KWOK?

  KWOK has several advantages:

  • Speed: You can create and delete clusters and nodes almost instantly, without waiting for boot or provisioning.
  • Compatibility: KWOK works with any tools or clients that are compliant with Kubernetes APIs, such as kubectl, helm, kui, etc.
  • Portability: KWOK has no specific hardware or software requirements. You can run it using pre-built images, once Docker or Nerdctl is installed. Alternatively, binaries are also available for all platforms and can be easily installed.
  • Flexibility: You can configure different node types, labels, taints, capacities, conditions, etc., and you can configure different pod behaviors, status, etc. to test different scenarios and edge cases.
  • Performance: You can simulate thousands of nodes on your laptop without significant consumption of CPU or memory resources.

  What are the use cases?

  KWOK can be used for various purposes:

  • Learning: You can use KWOK to learn about Kubernetes concepts and features without worrying about resource waste or other consequences.
  • Development: You can use KWOK to develop new features or tools for Kubernetes without accessing to a real cluster or requiring other components.
  • Testing:
    • You can measure how well your application or controller scales with different numbers of nodes and(or) pods.
    • You can generate high loads on your cluster by creating many pods or services with different resource requests or limits.
    • You can simulate node failures or network partitions by changing node conditions or randomly deleting nodes.
    • You can test how your controller interacts with other components or features of Kubernetes by enabling different feature gates or API versions.

  What are the limitations?

  KWOK is not intended to replace others completely. It has some limitations that you should be aware of:

  • Functionality: KWOK is not a kubelet and may exhibit different behaviors in areas such as pod lifecycle management, volume mounting, and device plugins. Its primary function is to simulate updates of node and pod status.
  • Accuracy: It's important to note that KWOK doesn't accurately reflect the performance or behavior of real nodes under various workloads or environments. Instead, it approximates some behaviors using simple formulas.
  • Security: KWOK does not enforce any security policies or mechanisms on simulated nodes. It assumes that all requests from the kube-apiserver are authorized and valid.

  Getting started

  If you are interested in trying out KWOK, please check its documents for more details.

  

  Using kwokctl to manage simulated clusters

  Getting Involved

  If you're interested in participating in future discussions or development related to KWOK, there are several ways to get involved:

  • Slack: #kwok for general usage discussion, #kwok-dev for development discussion. (visit slack.k8s.io for a workspace invitation)
  • Open Issues/PRs/Discussions in sigs.k8s.io/kwok

  We welcome feedback and contributions from anyone who wants to join us in this exciting project.

  
  
  https://kubernetes.io/blog/2023/03/01/introducing-kwok/
• 

  Blog: Free Katacoda Kubernetes Tutorials Are Shutting Down
  • sorangutan  

  1
  1
  Posts
  18
  Views

  

  Author: Natali Vlatko, SIG Docs Co-Chair for Kubernetes

  Katacoda, the popular learning platform from O’Reilly that has been helping people learn all about Java, Docker, Kubernetes, Python, Go, C++, and more, shut down for public use in June 2022. However, tutorials specifically for Kubernetes, linked from the Kubernetes website for our project’s users and contributors, remained available and active after this change. Unfortunately, this will no longer be the case, and Katacoda tutorials for learning Kubernetes will cease working after March 31st, 2023.

  The Kubernetes Project wishes to thank O'Reilly Media for the many years it has supported the community via the Katacoda learning platform. You can read more about the decision to shutter katacoda.com on O'Reilly's own site. With this change, we’ll be focusing on the work needed to remove links to their various tutorials. We have a general issue tracking this topic at #33936 and GitHub discussion. We’re also interested in researching what other learning platforms could be beneficial for the Kubernetes community, replacing Katacoda with a link to a platform or service that has a similar user experience. However, this research will take time, so we’re actively looking for volunteers to help with this work. If a replacement is found, it will need to be supported by Kubernetes leadership, specifically, SIG Contributor Experience, SIG Docs, and the Kubernetes Steering Committee.

  The Katacoda shutdown affects 25 tutorial pages, their localizations, as well as the Katacoda Scenario repository: github.com/katacoda-scenarios/kubernetes-bootcamp-scenarios. We recommend that any links, guides, or documentation you have that points to the Katacoda learning platform be updated immediately to reflect this change. While we have yet to find a replacement learning solution, the Kubernetes website contains a lot of helpful documentation to support your continued learning and growth. You can find all of our available documentation tutorials for Kubernetes at https://k8s.io/docs/tutorials/.

  If you have any questions regarding the Katacoda shutdown, or subsequent link removal from Kubernetes tutorial pages, please feel free to comment on the general issue tracking the shutdown, or visit the #sig-docs channel on the Kubernetes Slack.

  
  
  https://kubernetes.io/blog/2023/02/14/kubernetes-katacoda-tutorials-stop-from-2023-03-31/
• 

  Blog: k8s.gcr.io image registry will be frozen from the 3rd of April 2023
  • sorangutan  

  1
  1
  Posts
  18
  Views

  

  Authors: Mahamed Ali (Rackspace Technology)

  The Kubernetes project runs a community-owned image registry called registry.k8s.io to host its container images. On the 3rd of April 2023, the old registry k8s.gcr.io will be frozen and no further images for Kubernetes and related subprojects will be pushed to the old registry.

  This registry registry.k8s.io replaced the old one and has been generally available for several months. We have published a blog post about its benefits to the community and the Kubernetes project. This post also announced that future versions of Kubernetes will not be available in the old registry. Now that time has come.

  What does this change mean for contributors:

  • If you are a maintainer of a subproject, you will need to update your manifests and Helm charts to use the new registry.

  What does this change mean for end users:

  • 1.27 Kubernetes release will not be published to the old registry.
  • Patch releases for 1.24, 1.25, and 1.26 will no longer be published to the old registry from April. Please read the timelines below for details of the final patch releases in the old registry.
  • Starting in 1.25, the default image registry has been set to registry.k8s.io. This value is overridable in kubeadm and kubelet but setting it to k8s.gcr.io will fail for new releases after April as they won’t be present in the old registry.
  • If you want to increase the reliability of your cluster and remove dependency on the community-owned registry or you are running Kubernetes in networks where external traffic is restricted, you should consider hosting local image registry mirrors. Some cloud vendors may offer hosted solutions for this.

  Timeline of the Changes:

  • k8s.gcr.io will be frozen on the 3rd of April 2023
  • 1.27 is expected to be released on the 12th of April 2023
  • The last 1.23 release on k8s.gcr.io will be 1.23.18 (1.23 goes EoL before the freeze)
  • The last 1.24 release on k8s.gcr.io will be 1.24.12
  • The last 1.25 release on k8s.gcr.io will be 1.25.8
  • The last 1.26 release on k8s.gcr.io will be 1.26.3

  What's next

  Please make sure your cluster does not have dependencies on old image registry. For example, you can run this command to list the images used by pods:

  kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" |\
  tr -s '[[:space:]]' '\n' |\
  sort |\
  uniq -c
  

  There may be other dependencies on the old image registry. Make sure you review any potential dependencies to keep your cluster healthy and up to date.

  Acknowledgments

  Change is hard, and evolving our image-serving platform is needed to ensure a sustainable future for the project. We strive to make things better for everyone using Kubernetes. Many contributors from all corners of our community have been working long and hard to ensure we are making the best decisions possible, executing plans, and doing our best to communicate those plans.

  Thanks to Aaron Crickenberger, Arnaud Meukam, Benjamin Elder, Caleb Woodbine, Davanum Srinivas, Mahamed Ali, and Tim Hockin from SIG K8s Infra, Brian McQueen, and Sergey Kanzhelev from SIG Node, Lubomir Ivanov from SIG Cluster Lifecycle, Adolfo García Veytia, Jeremy Rickard, Sascha Grunert, and Stephen Augustus from SIG Release, Bob Killen and Kaslin Fields from SIG Contribex, Tim Allclair from the Security Response Committee. Also a big thank you to our friends acting as liaisons with our cloud provider partners: Jay Pipes from Amazon and Jon Johnson Jr. from Google.

  
  
  https://kubernetes.io/blog/2023/02/06/k8s-gcr-io-freeze-announcement/
• 

  Blog: Spotlight on SIG Instrumentation
  • sorangutan  

  1
  1
  Posts
  20
  Views

  

  Author: Imran Noor Mohamed (Delivery Hero)

  Observability requires the right data at the right time for the right consumer (human or piece of software) to make the right decision. In the context of Kubernetes, having best practices for cluster observability across all Kubernetes components is crucial.

  SIG Instrumentation helps to address this issue by providing best practices and tools that all other SIGs use to instrument Kubernetes components-like the Api server, scheduler, kubelet and kube-controller-manager.

  In this SIG Instrumentation spotlight, Imran Noor Mohamed, SIG ContribEx-Comms tech lead talked with Elana Hashman, and Han Kang, chairs of SIG Instrumentation, on how the SIG is organized, what are the current challenges and how anyone can get involved and contribute.

  About SIG Instrumentation

  Imran (INM): Hello, thank you for the opportunity of learning more about SIG Instrumentation. Could you tell us a bit about yourself, your role, and how you got involved in SIG Instrumentation?

  Han (HK): I started in SIG Instrumentation in 2018, and became a chair in 2020. I primarily got involved with SIG instrumentation due to a number of upstream issues with metrics which ended up affecting GKE in bad ways. As a result, we ended up launching an initiative to stabilize our metrics and make metrics a proper API.

  Elana (EH): I also joined SIG Instrumentation in 2018 and became a chair at the same time as Han. I was working as a site reliability engineer (SRE) on bare metal Kubernetes clusters and was working to build out our observability stack. I encountered some issues with label joins where Kubernetes metrics didn’t match kube-state-metrics (KSM) and started participating in SIG meetings to improve things. I helped test performance improvements to kube-state-metrics and ultimately coauthored a KEP for overhauling metrics in the 1.14 release to improve usability.

  Imran (INM): Interesting! Does that mean SIG Instrumentation involves a lot of plumbing?

  Han (HK): I wouldn’t say it involves a ton of plumbing, though it does touch basically every code base. We have our own dedicated directories for our metrics, logs, and tracing frameworks which we tend to work out of primarily. We do have to interact with other SIGs in order to propagate our changes which makes us more of a horizontal SIG.

  Imran (INM): Speaking about interaction and coordination with other SIG could you describe how the SIGs is organized?

  Elana (EH): In SIG Instrumentation, we have two chairs, Han and myself, as well as two tech leads, David Ashpole and Damien Grisonnet. We all work together as the SIG’s leads in order to run meetings, triage issues and PRs, review and approve KEPs, plan for each release, present at KubeCon and community meetings, and write our annual report. Within the SIG we also have a number of important subprojects, each of which is stewarded by its subproject owners. For example, Marek Siarkowicz is a subproject owner of metrics-server.

  Because we’re a horizontal SIG, some of our projects have a wide scope and require coordination from a dedicated group of contributors. For example, in order to guide the Kubernetes migration to structured logging, we chartered the Structured Logging Working Group (WG), organized by Marek and Patrick Ohly. The WG doesn’t own any code, but helps with various components such as the kubelet, scheduler, etc. in migrating their code to use structured logs.

  Imran (INM): Walking through the charter alone it’s clear that SIG Instrumentation has a lot of sub-projects. Could you highlight some important ones?

  Han (HK): We have many different sub-projects and we are in dire need of people who can come and help shepherd them. Our most important projects in-tree (that is, within the kubernetes/kubernetes repo) are metrics, tracing, and, structured logging. Our most important projects out-of-tree are (a) KSM (kube-state-metrics) and (b) metrics-server.

  Elana (EH): Echoing this, we would love to bring on more maintainers for kube-state-metrics and metrics-server. Our friends at WG Structured Logging are also looking for contributors. Other subprojects include klog, prometheus-adapter, and a new subproject that we just launched for collecting high-fidelity, scalable utilization metrics called usage-metrics-collector. All are seeking new contributors!

  Current status and ongoing challenges

  Imran (INM): For release 1.26 we can see that there are a relevant number of metrics, logs, and tracing KEPs in the pipeline. Would you like to point out important things for last release (maybe alpha & stable milestone candidates?)

  Han (HK): We can now generate documentation for every single metric in the main Kubernetes code base! We have a pretty fancy static analysis pipeline that enables this functionality. We’ve also added feature metrics so that you can look at your metrics to determine which features are enabled in your cluster at a given time. Lastly, we added a component-sli endpoint, which should make it easy for people to create availability SLOs for control-plane components.

  Elana (EH): We’ve also been working on tracing KEPs for both the API server and kubelet, though neither graduated in 1.26. I’m also really excited about the work Han is doing with WG Reliability to extend and improve our metrics stability framework.

  Imran (INM): What do you think are the Kubernetes-specific challenges tackled by the SIG Instrumentation? What are the future efforts to solve them?

  Han (HK): SIG instrumentation suffered a bit in the past from being a horizontal SIG. We did not have an obvious location to put our code and did not have a good mechanism to audit metrics that people would randomly add. We’ve fixed this over the years and now we have dedicated spots for our code and a reliable mechanism for auditing new metrics. We also now offer stability guarantees for metrics. We hope to have full-blown tracing up and down the kubernetes stack, and metric support via exemplars.

  Elana (EH): I think SIG Instrumentation is a really interesting SIG because it poses different kinds of opportunities to get involved than in other SIGs. You don’t have to be a software developer to contribute to our SIG! All of our components and subprojects are focused on better understanding Kubernetes and its performance in production, which allowed me to get involved as one of the few SIG Chairs working as an SRE at that time. I like that we provide opportunities for newcomers to contribute through using, testing, and providing feedback on our subprojects, which is a lower barrier to entry. Because many of these projects are out-of-tree, I think one of our challenges is to figure out what’s in scope for core Kubernetes SIGs instrumentation subprojects, what’s missing, and then fill in the gaps.

  Community and contribution

  Imran (INM): Kubernetes values community over products. Any recommendation for anyone looking into getting involved in SIG Instrumentation work? Where should they start (new contributor-friendly areas within SIG?)

  Han(HK) and Elana (EH): Come to our bi-weekly triage meetings! They aren’t recorded and are a great place to ask questions and learn about our ongoing work. We strive to be a friendly community and one of the easiest SIGs to get started with. You can check out our latest KubeCon NA 2022 SIG Instrumentation Deep Dive to get more insight into our work. We also invite you to join our Slack channel #sig-instrumentation and feel free to reach out to any of our SIG leads or subproject owners directly.

  Thank you so much for your time and insights into the workings of SIG Instrumentation!

  
  
  https://kubernetes.io/blog/2023/02/03/sig-instrumentation-spotlight-2023/
• 

  Blog: Consider All Microservices Vulnerable — And Monitor Their Behavior
  • sorangutan  

  1
  1
  Posts
  31
  Views

  

  Author: David Hadas (IBM Research Labs)

  This post warns Devops from a false sense of security. Following security best practices when developing and configuring microservices do not result in non-vulnerable microservices. The post shows that although all deployed microservices are vulnerable, there is much that can be done to ensure microservices are not exploited. It explains how analyzing the behavior of clients and services from a security standpoint, named here "Security-Behavior Analysis", can protect the deployed vulnerable microservices. It points to Guard, an open source project offering security-behavior monitoring and control of Kubernetes microservices presumed vulnerable.

  As cyber attacks continue to intensify in sophistication, organizations deploying cloud services continue to grow their cyber investments aiming to produce safe and non-vulnerable services. However, the year-by-year growth in cyber investments does not result in a parallel reduction in cyber incidents. Instead, the number of cyber incidents continues to grow annually. Evidently, organizations are doomed to fail in this struggle - no matter how much effort is made to detect and remove cyber weaknesses from deployed services, it seems offenders always have the upper hand.

  Considering the current spread of offensive tools, sophistication of offensive players, and ever-growing cyber financial gains to offenders, any cyber strategy that relies on constructing a non-vulnerable, weakness-free service in 2023 is clearly too naïve. It seems the only viable strategy is to:

  ➥ Admit that your services are vulnerable!

  In other words, consciously accept that you will never create completely invulnerable services. If your opponents find even a single weakness as an entry-point, you lose! Admitting that in spite of your best efforts, all your services are still vulnerable is an important first step. Next, this post discusses what you can do about it...

  How to protect microservices from being exploited

  Being vulnerable does not necessarily mean that your service will be exploited. Though your services are vulnerable in some ways unknown to you, offenders still need to identify these vulnerabilities and then exploit them. If offenders fail to exploit your service vulnerabilities, you win! In other words, having a vulnerability that can’t be exploited, represents a risk that can’t be realized.

  

  Figure 1. An Offender gaining foothold in a vulnerable service

  The above diagram shows an example in which the offender does not yet have a foothold in the service; that is, it is assumed that your service does not run code controlled by the offender on day 1. In our example the service has vulnerabilities in the API exposed to clients. To gain an initial foothold the offender uses a malicious client to try and exploit one of the service API vulnerabilities. The malicious client sends an exploit that triggers some unplanned behavior of the service.

  More specifically, let’s assume the service is vulnerable to an SQL injection. The developer failed to sanitize the user input properly, thereby allowing clients to send values that would change the intended behavior. In our example, if a client sends a query string with key “username” and value of “tom or 1=1”, the client will receive the data of all users. Exploiting this vulnerability requires the client to send an irregular string as the value. Note that benign users will not be sending a string with spaces or with the equal sign character as a username, instead they will normally send legal usernames which for example may be defined as a short sequence of characters a-z. No legal username can trigger service unplanned behavior.

  In this simple example, one can already identify several opportunities to detect and block an attempt to exploit the vulnerability (un)intentionally left behind by the developer, making the vulnerability unexploitable. First, the malicious client behavior differs from the behavior of benign clients, as it sends irregular requests. If such a change in behavior is detected and blocked, the exploit will never reach the service. Second, the service behavior in response to the exploit differs from the service behavior in response to a regular request. Such behavior may include making subsequent irregular calls to other services such as a data store, taking irregular time to respond, and/or responding to the malicious client with an irregular response (for example, containing much more data than normally sent in case of benign clients making regular requests). Service behavioral changes, if detected, will also allow blocking the exploit in different stages of the exploitation attempt.

  More generally:

  • Monitoring the behavior of clients can help detect and block exploits against service API vulnerabilities. In fact, deploying efficient client behavior monitoring makes many vulnerabilities unexploitable and others very hard to achieve. To succeed, the offender needs to create an exploit undetectable from regular requests.

  • Monitoring the behavior of services can help detect services as they are being exploited regardless of the attack vector used. Efficient service behavior monitoring limits what an attacker may be able to achieve as the offender needs to ensure the service behavior is undetectable from regular service behavior.

  Combining both approaches may add a protection layer to the deployed vulnerable services, drastically decreasing the probability for anyone to successfully exploit any of the deployed vulnerable services. Next, let us identify four use cases where you need to use security-behavior monitoring.

  Use cases

  One can identify the following four different stages in the life of any service from a security standpoint. In each stage, security-behavior monitoring is required to meet different challenges:

  Service State	Use case	What do you need in order to cope with this use case?
  Normal	No known vulnerabilities: The service owner is normally not aware of any known vulnerabilities in the service image or configuration. Yet, it is reasonable to assume that the service has weaknesses.	Provide generic protection against any unknown, zero-day, service vulnerabilities - Detect/block irregular patterns sent as part of incoming client requests that may be used as exploits.
  Vulnerable	An applicable CVE is published: The service owner is required to release a new non-vulnerable revision of the service. Research shows that in practice this process of removing a known vulnerability may take many weeks to accomplish (2 months on average).	Add protection based on the CVE analysis - Detect/block incoming requests that include specific patterns that may be used to exploit the discovered vulnerability. Continue to offer services, although the service has a known vulnerability.
  Exploitable	A known exploit is published: The service owner needs a way to filter incoming requests that contain the known exploit.	Add protection based on a known exploit signature - Detect/block incoming client requests that carry signatures identifying the exploit. Continue to offer services, although the presence of an exploit.
  Misused	An offender misuses pods backing the service: The offender can follow an attack pattern enabling him/her to misuse pods. The service owner needs to restart any compromised pods while using non compromised pods to continue offering the service. Note that once a pod is restarted, the offender needs to repeat the attack pattern before he/she may again misuse it.	Identify and restart instances of the component that is being misused - At any given time, some backing pods may be compromised and misused, while others behave as designed. Detect/remove the misused pods while allowing other pods to continue servicing client requests.

  Fortunately, microservice architecture is well suited to security-behavior monitoring as discussed next.

  Security-Behavior of microservices versus monoliths

  Kubernetes is often used to support workloads designed with microservice architecture. By design, microservices aim to follow the UNIX philosophy of "Do One Thing And Do It Well". Each microservice has a bounded context and a clear interface. In other words, you can expect the microservice clients to send relatively regular requests and the microservice to present a relatively regular behavior as a response to these requests. Consequently, a microservice architecture is an excellent candidate for security-behavior monitoring.

  

  Figure 2. Microservices are well suited for security-behavior monitoring

  The diagram above clarifies how dividing a monolithic service to a set of microservices improves our ability to perform security-behavior monitoring and control. In a monolithic service approach, different client requests are intertwined, resulting in a diminished ability to identify irregular client behaviors. Without prior knowledge, an observer of the intertwined client requests will find it hard to distinguish between types of requests and their related characteristics. Further, internal client requests are not exposed to the observer. Lastly, the aggregated behavior of the monolithic service is a compound of the many different internal behaviors of its components, making it hard to identify irregular service behavior.

  In a microservice environment, each microservice is expected by design to offer a more well-defined service and serve better defined type of requests. This makes it easier for an observer to identify irregular client behavior and irregular service behavior. Further, a microservice design exposes the internal requests and internal services which offer more security-behavior data to identify irregularities by an observer. Overall, this makes the microservice design pattern better suited for security-behavior monitoring and control.

  Security-Behavior monitoring on Kubernetes

  Kubernetes deployments seeking to add Security-Behavior may use Guard, developed under the CNCF project Knative. Guard is integrated into the full Knative automation suite that runs on top of Kubernetes. Alternatively, you can deploy Guard as a standalone tool to protect any HTTP-based workload on Kubernetes.

  See:

  • Guard on Github, for using Guard as a standalone tool.
  • The Knative automation suite - Read about Knative, in the blog post Opinionated Kubernetes which describes how Knative simplifies and unifies the way web services are deployed on Kubernetes.
  • You may contact Guard maintainers on the SIG Security Slack channel or on the Knative community security Slack channel. The Knative community channel will move soon to the CNCF Slack under the name #knative-security.

  The goal of this post is to invite the Kubernetes community to action and introduce Security-Behavior monitoring and control to help secure Kubernetes based deployments. Hopefully, the community as a follow up will:

  1. Analyze the cyber challenges presented for different Kubernetes use cases
  2. Add appropriate security documentation for users on how to introduce Security-Behavior monitoring and control.
  3. Consider how to integrate with tools that can help users monitor and control their vulnerable services.

  Getting involved

  You are welcome to get involved and join the effort to develop security behavior monitoring and control for Kubernetes; to share feedback and contribute to code or documentation; and to make or suggest improvements of any kind.

  
  
  https://kubernetes.io/blog/2023/01/20/security-behavior-analysis/
• 

  Blog: Protect Your Mission-Critical Pods From Eviction With PriorityClass
  • sorangutan  

  1
  1
  Posts
  22
  Views

  

  Author: Sunny Bhambhani (InfraCloud Technologies)

  Kubernetes has been widely adopted, and many organizations use it as their de-facto orchestration engine for running workloads that need to be created and deleted frequently.

  Therefore, proper scheduling of the pods is key to ensuring that application pods are up and running within the Kubernetes cluster without any issues. This article delves into the use cases around resource management by leveraging the PriorityClass object to protect mission-critical or high-priority pods from getting evicted and making sure that the application pods are up, running, and serving traffic.

  Resource management in Kubernetes

  The control plane consists of multiple components, out of which the scheduler (usually the built-in kube-scheduler) is one of the components which is responsible for assigning a node to a pod.

  Whenever a pod is created, it enters a "pending" state, after which the scheduler determines which node is best suited for the placement of the new pod.

  In the background, the scheduler runs as an infinite loop looking for pods without a nodeName set that are ready for scheduling. For each Pod that needs scheduling, the scheduler tries to decide which node should run that Pod.

  If the scheduler cannot find any node, the pod remains in the pending state, which is not ideal.

  Note: To name a few, nodeSelector , taints and tolerations , nodeAffinity , the rank of nodes based on available resources (for example, CPU and memory), and several other criteria are used to determine the pod's placement.

  The below diagram, from point number 1 through 4, explains the request flow:

  

  Scheduling in Kubernetes

  Typical use cases

  Below are some real-life scenarios where control over the scheduling and eviction of pods may be required.

  1. Let's say the pod you plan to deploy is critical, and you have some resource constraints. An example would be the DaemonSet of an infrastructure component like Grafana Loki. The Loki pods must run before other pods can on every node. In such cases, you could ensure resource availability by manually identifying and deleting the pods that are not required or by adding a new node to the cluster. Both these approaches are unsuitable since the former would be tedious to execute, and the latter could involve an expenditure of time and money.

  2. Another use case could be a single cluster that holds the pods for the below environments with associated priorities:

     • Production (prod): top priority
     • Preproduction (preprod): intermediate priority
     • Development (dev): least priority

  In the event of high resource consumption in the cluster, there is competition for CPU and memory resources on the nodes. While cluster-level autoscaling may add more nodes, it takes time. In the interim, if there are no further nodes to scale the cluster, some Pods could remain in a Pending state, or the service could be degraded as they compete for resources. If the kubelet does evict a Pod from the node, that eviction would be random because the kubelet doesn’t have any special information about which Pods to evict and which to keep.

  3. A third example could be a microservice backed by a queuing application or a database running into a resource crunch and the queue or database getting evicted. In such a case, all the other services would be rendered useless until the database can serve traffic again.

  There can also be other scenarios where you want to control the order of scheduling or order of eviction of pods.

  PriorityClasses in Kubernetes

  PriorityClass is a cluster-wide API object in Kubernetes and part of the scheduling.k8s.io/v1 API group. It contains a mapping of the PriorityClass name (defined in .metadata.name) and an integer value (defined in .value). This represents the value that the scheduler uses to determine Pod's relative priority.

  Additionally, when you create a cluster using kubeadm or a managed Kubernetes service (for example, Azure Kubernetes Service), Kubernetes uses PriorityClasses to safeguard the pods that are hosted on the control plane nodes. This ensures that critical cluster components such as CoreDNS and kube-proxy can run even if resources are constrained.

  This availability of pods is achieved through the use of a special PriorityClass that ensures the pods are up and running and that the overall cluster is not affected.

  $ kubectl get priorityclass
  NAME VALUE GLOBAL-DEFAULT AGE
  system-cluster-critical 2000000000 false 82m
  system-node-critical 2000001000 false 82m
  

  The diagram below shows exactly how it works with the help of an example, which will be detailed in the upcoming section.

  

  Pod scheduling and preemption

  Pod priority and preemption

  Pod preemption is a Kubernetes feature that allows the cluster to preempt pods (removing an existing Pod in favor of a new Pod) on the basis of priority. Pod priority indicates the importance of a pod relative to other pods while scheduling. If there aren't enough resources to run all the current pods, the scheduler tries to evict lower-priority pods over high-priority ones.

  Also, when a healthy cluster experiences a node failure, typically, lower-priority pods get preempted to create room for higher-priority pods on the available node. This happens even if the cluster can bring up a new node automatically since pod creation is usually much faster than bringing up a new node.

  PriorityClass requirements

  Before you set up PriorityClasses, there are a few things to consider.

  1. Decide which PriorityClasses are needed. For instance, based on environment, type of pods, type of applications, etc.
  2. The default PriorityClass resource for your cluster. The pods without a priorityClassName will be treated as priority 0.
  3. Use a consistent naming convention for all PriorityClasses.
  4. Make sure that the pods for your workloads are running with the right PriorityClass.

  PriorityClass hands-on example

  Let’s say there are 3 application pods: one for prod, one for preprod, and one for development. Below are three sample YAML manifest files for each of those.

  ---
  # development
  apiVersion: v1
  kind: Pod
  metadata:
   name: dev-nginx
   labels:
   env: dev
  spec:
   containers:
   - name: dev-nginx
   image: nginx
   resources:
   requests:
   memory: "256Mi"
   cpu: "0.2"
   limits:
   memory: ".5Gi"
   cpu: "0.5"
  

  ---
  # preproduction
  apiVersion: v1
  kind: Pod
  metadata:
   name: preprod-nginx
   labels:
   env: preprod
  spec:
   containers:
   - name: preprod-nginx
   image: nginx
   resources:
   requests:
   memory: "1.5Gi"
   cpu: "1.5"
   limits:
   memory: "2Gi"
   cpu: "2"
  

  ---
  # production
  apiVersion: v1
  kind: Pod
  metadata:
   name: prod-nginx
   labels:
   env: prod
  spec:
   containers:
   - name: prod-nginx
   image: nginx
   resources:
   requests:
   memory: "2Gi"
   cpu: "2"
   limits:
   memory: "2Gi"
   cpu: "2"
  

  You can create these pods with the kubectl create -f <FILE.yaml> command, and then check their status using the kubectl get pods command. You can see if they are up and look ready to serve traffic:

  $ kubectl get pods --show-labels
  NAME READY STATUS RESTARTS AGE LABELS
  dev-nginx 1/1 Running 0 55s env=dev
  preprod-nginx 1/1 Running 0 55s env=preprod
  prod-nginx 0/1 Pending 0 55s env=prod
  

  Bad news. The pod for the Production environment is still Pending and isn't serving any traffic.

  Let's see why this is happening:

  $ kubectl get events
  ...
  ...
  5s Warning FailedScheduling pod/prod-nginx 0/2 nodes are available: 1 Insufficient cpu, 2 Insufficient memory.
  

  In this example, there is only one worker node, and that node has a resource crunch.

  Now, let's look at how PriorityClass can help in this situation since prod should be given higher priority than the other environments.

  PriorityClass API

  Before creating PriorityClasses based on these requirements, let's see what a basic manifest for a PriorityClass looks like and outline some prerequisites:

  apiVersion: scheduling.k8s.io/v1
  kind: PriorityClass
  metadata:
   name: PRIORITYCLASS_NAME
  value: 0 # any integer value between -1000000000 to 1000000000 
  description: >-
    (Optional) description goes here!
  globalDefault: false # or true. Only one PriorityClass can be the global default.
  

  Below are some prerequisites for PriorityClasses:

  • The name of a PriorityClass must be a valid DNS subdomain name.
  • When you make your own PriorityClass, the name should not start with system-, as those names are reserved by Kubernetes itself (for example, they are used for two built-in PriorityClasses).
  • Its absolute value should be between -1000000000 to 1000000000 (1 billion).
  • Larger numbers are reserved by PriorityClasses such as system-cluster-critical (this Pod is critically important to the cluster) and system-node-critical (the node critically relies on this Pod). system-node-critical is a higher priority than system-cluster-critical, because a cluster-critical Pod can only work well if the node where it is running has all its node-level critical requirements met.
  • There are two optional fields:
    • globalDefault: When true, this PriorityClass is used for pods where a priorityClassName is not specified. Only one PriorityClass with globalDefault set to true can exist in a cluster.
      If there is no PriorityClass defined with globalDefault set to true, all the pods with no priorityClassName defined will be treated with 0 priority (i.e. the least priority).
    • description: A string with a meaningful value so that people know when to use this PriorityClass.

  Note: Adding a PriorityClass with globalDefault set to true does not mean it will apply the same to the existing pods that are already running. This will be applicable only to the pods that came into existence after the PriorityClass was created.

  PriorityClass in action

  Here's an example. Next, create some environment-specific PriorityClasses:

  apiVersion: scheduling.k8s.io/v1
  kind: PriorityClass
  metadata:
   name: dev-pc
  value: 1000000
  globalDefault: false
  description: >-
    (Optional) This priority class should only be used for all development pods.
  

  apiVersion: scheduling.k8s.io/v1
  kind: PriorityClass
  metadata:
   name: preprod-pc
  value: 2000000
  globalDefault: false
  description: >-
    (Optional) This priority class should only be used for all preprod pods.
  

  apiVersion: scheduling.k8s.io/v1
  kind: PriorityClass
  metadata:
   name: prod-pc
  value: 4000000
  globalDefault: false
  description: >-
    (Optional) This priority class should only be used for all prod pods.
  

  Use kubectl create -f <FILE.YAML> command to create a pc and kubectl get pc to check its status.

  $ kubectl get pc
  NAME VALUE GLOBAL-DEFAULT AGE
  dev-pc 1000000 false 3m13s
  preprod-pc 2000000 false 2m3s
  prod-pc 4000000 false 7s
  system-cluster-critical 2000000000 false 82m
  system-node-critical 2000001000 false 82m
  

  The new PriorityClasses are in place now. A small change is needed in the pod manifest or pod template (in a ReplicaSet or Deployment). In other words, you need to specify the priority class name at .spec.priorityClassName (which is a string value).

  First update the previous production pod manifest file to have a PriorityClass assigned, then delete the Production pod and recreate it. You can't edit the priority class for a Pod that already exists.

  In my cluster, when I tried this, here's what happened. First, that change seems successful; the status of pods has been updated:

  $ kubectl get pods --show-labels
  NAME READY STATUS RESTARTS AGE LABELS
  dev-nginx 1/1 Terminating 0 55s env=dev
  preprod-nginx 1/1 Running 0 55s env=preprod
  prod-nginx 0/1 Pending 0 55s env=prod
  

  The dev-nginx pod is getting terminated. Once that is successfully terminated and there are enough resources for the prod pod, the control plane can schedule the prod pod:

  Warning FailedScheduling pod/prod-nginx 0/2 nodes are available: 1 Insufficient cpu, 2 Insufficient memory.
  Normal Preempted pod/dev-nginx by default/prod-nginx on node node01
  Normal Killing pod/dev-nginx Stopping container dev-nginx
  Normal Scheduled pod/prod-nginx Successfully assigned default/prod-nginx to node01
  Normal Pulling pod/prod-nginx Pulling image "nginx"
  Normal Pulled pod/prod-nginx Successfully pulled image "nginx"
  Normal Created pod/prod-nginx Created container prod-nginx
  Normal Started pod/prod-nginx Started container prod-nginx
  

  Enforcement

  When you set up PriorityClasses, they exist just how you defined them. However, people (and tools) that make changes to your cluster are free to set any PriorityClass, or to not set any PriorityClass at all. However, you can use other Kubernetes features to make sure that the priorities you wanted are actually applied.

  As an alpha feature, you can define a ValidatingAdmissionPolicy and a ValidatingAdmissionPolicyBinding so that, for example, Pods that go into the prod namespace must use the prod-pc PriorityClass. With another ValidatingAdmissionPolicyBinding you ensure that the preprod namespace uses the preprod-pc PriorityClass, and so on. In any cluster, you can enforce similar controls using external projects such as Kyverno or Gatekeeper, through validating admission webhooks.

  However you do it, Kubernetes gives you options to make sure that the PriorityClasses are used how you wanted them to be, or perhaps just to warn users when they pick an unsuitable option.

  Summary

  The above example and its events show you what this feature of Kubernetes brings to the table, along with several scenarios where you can use this feature. To reiterate, this helps ensure that mission-critical pods are up and available to serve the traffic and, in the case of a resource crunch, determines cluster behavior.

  It gives you some power to decide the order of scheduling and order of preemption for Pods. Therefore, you need to define the PriorityClasses sensibly. For example, if you have a cluster autoscaler to add nodes on demand, make sure to run it with the system-cluster-critical PriorityClass. You don't want to get in a situation where the autoscaler has been preempted and there are no new nodes coming online.

  If you have any queries or feedback, feel free to reach out to me on LinkedIn.

  
  
  https://kubernetes.io/blog/2023/01/12/protect-mission-critical-pods-priorityclass/
• 

  Blog: Kubernetes 1.26: Eviction policy for unhealthy pods guarded by PodDisruptionBudgets
  • sorangutan  

  1
  1
  Posts
  20
  Views

  

  Authors: Filip Křepinský (Red Hat), Morten Torkildsen (Google), Ravi Gudimetla (Apple)

  Ensuring the disruptions to your applications do not affect its availability isn't a simple task. Last month's release of Kubernetes v1.26 lets you specify an unhealthy pod eviction policy for PodDisruptionBudgets (PDBs) to help you maintain that availability during node management operations. In this article, we will dive deeper into what modifications were introduced for PDBs to give application owners greater flexibility in managing disruptions.

  What problems does this solve?

  API-initiated eviction of pods respects PodDisruptionBudgets (PDBs). This means that a requested voluntary disruption via an eviction to a Pod, should not disrupt a guarded application and .status.currentHealthy of a PDB should not fall below .status.desiredHealthy. Running pods that are Unhealthy do not count towards the PDB status, but eviction of these is only possible in case the application is not disrupted. This helps disrupted or not yet started application to achieve availability as soon as possible without additional downtime that would be caused by evictions.

  Unfortunately, this poses a problem for cluster administrators that would like to drain nodes without any manual interventions. Misbehaving applications with pods in CrashLoopBackOff state (due to a bug or misconfiguration) or pods that are simply failing to become ready make this task much harder. Any eviction request will fail due to violation of a PDB, when all pods of an application are unhealthy. Draining of a node cannot make any progress in that case.

  On the other hand there are users that depend on the existing behavior, in order to:

  • prevent data-loss that would be caused by deleting pods that are guarding an underlying resource or storage
  • achieve the best availability possible for their application

  Kubernetes 1.26 introduced a new experimental field to the PodDisruptionBudget API: .spec.unhealthyPodEvictionPolicy. When enabled, this field lets you support both of those requirements.

  How does it work?

  API-initiated eviction is the process that triggers graceful pod termination. The process can be initiated either by calling the API directly, by using a kubectl drain command, or other actors in the cluster. During this process every pod removal is consulted with appropriate PDBs, to ensure that a sufficient number of pods is always running in the cluster.

  The following policies allow PDB authors to have a greater control how the process deals with unhealthy pods.

  There are two policies IfHealthyBudget and AlwaysAllow to choose from.

  The former, IfHealthyBudget, follows the existing behavior to achieve the best availability that you get by default. Unhealthy pods can be disrupted only if their application has a minimum available .status.desiredHealthy number of pods.

  By setting the spec.unhealthyPodEvictionPolicy field of your PDB to AlwaysAllow, you are choosing the best effort availability for your application. With this policy it is always possible to evict unhealthy pods. This will make it easier to maintain and upgrade your clusters.

  We think that AlwaysAllow will often be a better choice, but for some critical workloads you may still prefer to protect even unhealthy Pods from node drains or other forms of API-initiated eviction.

  How do I use it?

  This is an alpha feature, which means you have to enable the PDBUnhealthyPodEvictionPolicy feature gate, with the command line argument --feature-gates=PDBUnhealthyPodEvictionPolicy=true to the kube-apiserver.

  Here's an example. Assume that you've enabled the feature gate in your cluster, and that you already defined a Deployment that runs a plain webserver. You labelled the Pods for that Deployment with app: nginx. You want to limit avoidable disruption, and you know that best effort availability is sufficient for this app. You decide to allow evictions even if those webserver pods are unhealthy. You create a PDB to guard this application, with the AlwaysAllow policy for evicting unhealthy pods:

  apiVersion: policy/v1
  kind: PodDisruptionBudget
  metadata:
   name: nginx-pdb
  spec:
   selector:
   matchLabels:
   app: nginx
   maxUnavailable: 1
   unhealthyPodEvictionPolicy: AlwaysAllow
  

  How can I learn more?

  • Read the KEP: Unhealthy Pod Eviction Policy for PDBs
  • Read the documentation: Unhealthy Pod Eviction Policy for PodDisruptionBudgets
  • Review the Kubernetes documentation for PodDisruptionBudgets, draining of Nodes and evictions

  How do I get involved?

  If you have any feedback, please reach out to us in the #sig-apps channel on Slack (visit https://slack.k8s.io/ for an invitation if you need one), or on the SIG Apps mailing list: [email protected]

  
  
  https://kubernetes.io/blog/2023/01/06/unhealthy-pod-eviction-policy-for-pdbs/
• 

  Blog: Kubernetes 1.26: Retroactive Default StorageClass
  • sorangutan  

  1
  1
  Posts
  13
  Views

  

  Author: Roman Bednář (Red Hat)

  The v1.25 release of Kubernetes introduced an alpha feature to change how a default StorageClass was assigned to a PersistentVolumeClaim (PVC). With the feature enabled, you no longer need to create a default StorageClass first and PVC second to assign the class. Additionally, any PVCs without a StorageClass assigned can be updated later. This feature was graduated to beta in Kubernetes 1.26.

  You can read retroactive default StorageClass assignment in the Kubernetes documentation for more details about how to use that, or you can read on to learn about why the Kubernetes project is making this change.

  Why did StorageClass assignment need improvements

  Users might already be familiar with a similar feature that assigns default StorageClasses to new PVCs at the time of creation. This is currently handled by the admission controller.

  But what if there wasn't a default StorageClass defined at the time of PVC creation? Users would end up with a PVC that would never be assigned a class. As a result, no storage would be provisioned, and the PVC would be somewhat "stuck" at this point. Generally, two main scenarios could result in "stuck" PVCs and cause problems later down the road. Let's take a closer look at each of them.

  Changing default StorageClass

  With the alpha feature enabled, there were two options admins had when they wanted to change the default StorageClass:

  1. Creating a new StorageClass as default before removing the old one associated with the PVC. This would result in having two defaults for a short period. At this point, if a user were to create a PersistentVolumeClaim with storageClassName set to null (implying default StorageClass), the newest default StorageClass would be chosen and assigned to this PVC.

  2. Removing the old default first and creating a new default StorageClass. This would result in having no default for a short time. Subsequently, if a user were to create a PersistentVolumeClaim with storageClassName set to null (implying default StorageClass), the PVC would be in Pending state forever. The user would have to fix this by deleting the PVC and recreating it once the default StorageClass was available.

  Resource ordering during cluster installation

  If a cluster installation tool needed to create resources that required storage, for example, an image registry, it was difficult to get the ordering right. This is because any Pods that required storage would rely on the presence of a default StorageClass and would fail to be created if it wasn't defined.

  What changed

  We've changed the PersistentVolume (PV) controller to assign a default StorageClass to any unbound PersistentVolumeClaim that has the storageClassName set to null. We've also modified the PersistentVolumeClaim admission within the API server to allow the change of values from an unset value to an actual StorageClass name.

  Null storageClassName versus storageClassName: "" - does it matter?

  Before this feature was introduced, those values were equal in terms of behavior. Any PersistentVolumeClaim with the storageClassName set to null or "" would bind to an existing PersistentVolume resource with storageClassName also set to null or "".

  With this new feature enabled we wanted to maintain this behavior but also be able to update the StorageClass name. With these constraints in mind, the feature changes the semantics of null. If a default StorageClass is present, null would translate to "Give me a default" and "" would mean "Give me PersistentVolume that also has "" StorageClass name." In the absence of a StorageClass, the behavior would remain unchanged.

  Summarizing the above, we've changed the semantics of null so that its behavior depends on the presence or absence of a definition of default StorageClass.

  The tables below show all these cases to better describe when PVC binds and when its StorageClass gets updated.

  PVC binding behavior with Retroactive default StorageClass

  		PVC storageClassName = ""	PVC storageClassName = null
  Without default class	PV storageClassName = ""	binds	binds
  	PV without storageClassName	binds	binds
  With default class	PV storageClassName = ""	binds	class updates
  	PV without storageClassName	binds	class updates

  How to use it

  If you want to test the feature whilst it's alpha, you need to enable the relevant feature gate in the kube-controller-manager and the kube-apiserver. Use the --feature-gates command line argument:

  --feature-gates="...,RetroactiveDefaultStorageClass=true"
  

  Test drive

  If you would like to see the feature in action and verify it works fine in your cluster here's what you can try:

  1. Define a basic PersistentVolumeClaim:

  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
   name: pvc-1
  spec:
   accessModes:
   - ReadWriteOnce
   resources:
   requests:
   storage: 1Gi
  

  2. Create the PersistentVolumeClaim when there is no default StorageClass. The PVC won't provision or bind (unless there is an existing, suitable PV already present) and will remain in Pending state.

  $ kc get pvc
  NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
  pvc-1 Pending
  

  3. Configure one StorageClass as default.

  $ kc patch sc -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
  storageclass.storage.k8s.io/my-storageclass patched
  

  4. Verify that PersistentVolumeClaims is now provisioned correctly and was updated retroactively with new default StorageClass.

  $ kc get pvc
  NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
  pvc-1 Bound pvc-06a964ca-f997-4780-8627-b5c3bf5a87d8 1Gi RWO my-storageclass 87m
  

  New metrics

  To help you see that the feature is working as expected we also introduced a new retroactive_storageclass_total metric to show how many times that the PV controller attempted to update PersistentVolumeClaim, and retroactive_storageclass_errors_total to show how many of those attempts failed.

  Getting involved

  We always welcome new contributors so if you would like to get involved you can join our Kubernetes Storage Special-Interest-Group (SIG).

  If you would like to share feedback, you can do so on our public Slack channel.

  Special thanks to all the contributors that provided great reviews, shared valuable insight and helped implement this feature (alphabetical order):

  • Deep Debroy (ddebroy)
  • Divya Mohan (divya-mohan0209)
  • Jan Šafránek (jsafrane)
  • Joe Betz (jpbetz)
  • Jordan Liggitt (liggitt)
  • Michelle Au (msau42)
  • Seokho Son (seokho-son)
  • Shannon Kularathna (shannonxtreme)
  • Tim Bannister (sftim)
  • Tim Hockin (thockin)
  • Wojciech Tyczynski (wojtek-t)
  • Xing Yang (xing-yang)

  
  
  https://kubernetes.io/blog/2023/01/05/retroactive-default-storage-class/
• 

  Blog: Kubernetes v1.26: Alpha support for cross-namespace storage data sources
  • sorangutan  

  1
  1
  Posts
  28
  Views

  

  Author: Takafumi Takahashi (Hitachi Vantara)

  Kubernetes v1.26, released last month, introduced an alpha feature that lets you specify a data source for a PersistentVolumeClaim, even where the source data belong to a different namespace. With the new feature enabled, you specify a namespace in the dataSourceRef field of a new PersistentVolumeClaim. Once Kubernetes checks that access is OK, the new PersistentVolume can populate its data from the storage source specified in that other namespace. Before Kubernetes v1.26, provided your cluster had the AnyVolumeDataSource feature enabled, you could already provision new volumes from a data source in the same namespace. However, that only worked for the data source in the same namespace, therefore users couldn't provision a PersistentVolume with a claim in one namespace from a data source in other namespace. To solve this problem, Kubernetes v1.26 added a new alpha namespace field to dataSourceRef field in PersistentVolumeClaim the API.

  How it works

  Once the csi-provisioner finds that a data source is specified with a dataSourceRef that has a non-empty namespace name, it checks all reference grants within the namespace that's specified by the.spec.dataSourceRef.namespace field of the PersistentVolumeClaim, in order to see if access to the data source is allowed. If any ReferenceGrant allows access, the csi-provisioner provisions a volume from the data source.

  Trying it out

  The following things are required to use cross namespace volume provisioning:

  • Enable the AnyVolumeDataSource and CrossNamespaceVolumeDataSource feature gates for the kube-apiserver and kube-controller-manager
  • Install a CRD for the specific VolumeSnapShot controller
  • Install the CSI Provisioner controller and enable the CrossNamespaceVolumeDataSource feature gate
  • Install the CSI driver
  • Install a CRD for ReferenceGrants

  Putting it all together

  To see how this works, you can install the sample and try it out. This sample do to create PVC in dev namespace from VolumeSnapshot in prod namespace. That is a simple example. For real world use, you might want to use a more complex approach.

  Assumptions for this example

  • Your Kubernetes cluster was deployed with AnyVolumeDataSource and CrossNamespaceVolumeDataSource feature gates enabled
  • There are two namespaces, dev and prod
  • CSI driver is being deployed
  • There is an existing VolumeSnapshot named new-snapshot-demo in the prod namespace
  • The ReferenceGrant CRD (from the Gateway API project) is already deployed

  Grant ReferenceGrants read permission to the CSI Provisioner

  Access to ReferenceGrants is only needed when the CSI driver has the CrossNamespaceVolumeDataSource controller capability. For this example, the external-provisioner needs get, list, and watch permissions for referencegrants (API group gateway.networking.k8s.io).

   - apiGroups: ["gateway.networking.k8s.io"]
   resources: ["referencegrants"]
   verbs: ["get", "list", "watch"]
  

  Enable the CrossNamespaceVolumeDataSource feature gate for the CSI Provisioner

  Add --feature-gates=CrossNamespaceVolumeDataSource=true to the csi-provisioner command line. For example, use this manifest snippet to redefine the container:

   - args:
   - -v=5
   - --csi-address=/csi/csi.sock
   - --feature-gates=Topology=true
   - --feature-gates=CrossNamespaceVolumeDataSource=true
   image: csi-provisioner:latest
   imagePullPolicy: IfNotPresent
   name: csi-provisioner
  

  Create a ReferenceGrant

  Here's a manifest for an example ReferenceGrant.

  apiVersion: gateway.networking.k8s.io/v1beta1
  kind: ReferenceGrant
  metadata:
   name: allow-prod-pvc
   namespace: prod
  spec:
   from:
   - group: ""
   kind: PersistentVolumeClaim
   namespace: dev
   to:
   - group: snapshot.storage.k8s.io
   kind: VolumeSnapshot
   name: new-snapshot-demo
  

  Create a PersistentVolumeClaim by using cross namespace data source

  Kubernetes creates a PersistentVolumeClaim on dev and the CSI driver populates the PersistentVolume used on dev from snapshots on prod.

  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
   name: example-pvc
   namespace: dev
  spec:
   storageClassName: example
   accessModes:
   - ReadWriteOnce
   resources:
   requests:
   storage: 1Gi
   dataSourceRef:
   apiGroup: snapshot.storage.k8s.io
   kind: VolumeSnapshot
   name: new-snapshot-demo
   namespace: prod
   volumeMode: Filesystem
  

  How can I learn more?

  The enhancement proposal, Provision volumes from cross-namespace snapshots, includes lots of detail about the history and technical implementation of this feature.

  Please get involved by joining the Kubernetes Storage Special Interest Group (SIG) to help us enhance this feature. There are a lot of good ideas already and we'd be thrilled to have more!

  Acknowledgments

  It takes a wonderful group to make wonderful software. Special thanks to the following people for the insightful reviews, thorough consideration and valuable contribution to the CrossNamespaceVolumeDataSouce feature:

  • Michelle Au (msau42)
  • Xing Yang (xing-yang)
  • Masaki Kimura (mkimuram)
  • Tim Hockin (thockin)
  • Ben Swartzlander (bswartz)
  • Rob Scott (robscott)
  • John Griffith (j-griffith)
  • Michael Henriksen (mhenriks)
  • Mustafa Elbehery (Elbehery)

  It’s been a joy to work with y'all on this.

  
  
  https://kubernetes.io/blog/2023/01/02/cross-namespace-data-sources-alpha/
• 

  Blog: Kubernetes v1.26: Advancements in Kubernetes Traffic Engineering
  • sorangutan  

  1
  1
  Posts
  14
  Views

  

  Authors: Andrew Sy Kim (Google)

  Kubernetes v1.26 includes significant advancements in network traffic engineering with the graduation of two features (Service internal traffic policy support, and EndpointSlice terminating conditions) to GA, and a third feature (Proxy terminating endpoints) to beta. The combination of these enhancements aims to address short-comings in traffic engineering that people face today, and unlock new capabilities for the future.

  Traffic Loss from Load Balancers During Rolling Updates

  Prior to Kubernetes v1.26, clusters could experience loss of traffic from Service load balancers during rolling updates when setting the externalTrafficPolicy field to Local. There are a lot of moving parts at play here so a quick overview of how Kubernetes manages load balancers might help!

  In Kubernetes, you can create a Service with type: LoadBalancer to expose an application externally with a load balancer. The load balancer implementation varies between clusters and platforms, but the Service provides a generic abstraction representing the load balancer that is consistent across all Kubernetes installations.

  apiVersion: v1
  kind: Service
  metadata:
   name: my-service
  spec:
   selector:
   app.kubernetes.io/name: my-app
   ports:
   - protocol: TCP
   port: 80
   targetPort: 9376
   type: LoadBalancer
  

  Under the hood, Kubernetes allocates a NodePort for the Service, which is then used by kube-proxy to provide a network data path from the NodePort to the Pod. A controller will then add all available Nodes in the cluster to the load balancer’s backend pool, using the designated NodePort for the Service as the backend target port.

  

  Figure 1: Overview of Service load balancers

  Oftentimes it is beneficial to set externalTrafficPolicy: Local for Services, to avoid extra hops between Nodes that are not running healthy Pods backing that Service. When using externalTrafficPolicy: Local, an additional NodePort is allocated for health checking purposes, such that Nodes that do not contain healthy Pods are excluded from the backend pool for a load balancer.

  

  Figure 2: Load balancer traffic to a healthy Node, when externalTrafficPolicy is Local

  One such scenario where traffic can be lost is when a Node loses all Pods for a Service, but the external load balancer has not probed the health check NodePort yet. The likelihood of this situation is largely dependent on the health checking interval configured on the load balancer. The larger the interval, the more likely this will happen, since the load balancer will continue to send traffic to a node even after kube-proxy has removed forwarding rules for that Service. This also occurrs when Pods start terminating during rolling updates. Since Kubernetes does not consider terminating Pods as “Ready”, traffic can be loss when there are only terminating Pods on any given Node during a rolling update.

  

  Figure 3: Load balancer traffic to terminating endpoints, when externalTrafficPolicy is Local

  Starting in Kubernetes v1.26, kube-proxy enables the ProxyTerminatingEndpoints feature by default, which adds automatic failover and routing to terminating endpoints in scenarios where the traffic would otherwise be dropped. More specifically, when there is a rolling update and a Node only contains terminating Pods, kube-proxy will route traffic to the terminating Pods based on their readiness. In addition, kube-proxy will actively fail the health check NodePort if there are only terminating Pods available. By doing so, kube-proxy alerts the external load balancer that new connections should not be sent to that Node but will gracefully handle requests for existing connections.

  

  Figure 4: Load Balancer traffic to terminating endpoints with ProxyTerminatingEndpoints enabled, when externalTrafficPolicy is Local

  EndpointSlice Conditions

  In order to support this new capability in kube-proxy, the EndpointSlice API introduced new conditions for endpoints: serving and terminating.

  

  Figure 5: Overview of EndpointSlice conditions

  The serving condition is semantically identical to ready, except that it can be true or false while a Pod is terminating, unlike ready which will always be false for terminating Pods for compatibility reasons. The terminating condition is true for Pods undergoing termination (non-empty deletionTimestamp), false otherwise.

  The addition of these two conditions enables consumers of this API to understand Pod states that were previously not possible. For example, we can now track "ready" and "not ready" Pods that are also terminating.

  

  Figure 6: EndpointSlice conditions with a terminating Pod

  Consumers of the EndpointSlice API, such as Kube-proxy and Ingress Controllers, can now use these conditions to coordinate connection draining events, by continuing to forward traffic for existing connections but rerouting new connections to other non-terminating endpoints.

  Optimizing Internal Node-Local Traffic

  Similar to how Services can set externalTrafficPolicy: Local to avoid extra hops for externally sourced traffic, Kubernetes now supports internalTrafficPolicy: Local, to enable the same optimization for traffic originating within the cluster, specifically for traffic using the Service Cluster IP as the destination address. This feature graduated to Beta in Kubernetes v1.24 and is graduating to GA in v1.26.

  Services default the internalTrafficPolicy field to Cluster, where traffic is randomly distributed to all endpoints.

  

  Figure 7: Service routing when internalTrafficPolicy is Cluster

  When internalTrafficPolicy is set to Local, kube-proxy will forward internal traffic for a Service only if there is an available endpoint that is local to the same Node.

  

  Figure 8: Service routing when internalTrafficPolicy is Local

  Caution: When using internalTrafficPoliy: Local, traffic will be dropped by kube-proxy when no local endpoints are available.

  Getting Involved

  If you're interested in future discussions on Kubernetes traffic engineering, you can get involved in SIG Network through the following ways:

  • Slack: #sig-network
  • Mailing list
  • Open Community Issues/PRs
  • Biweekly meetings

  
  
  https://kubernetes.io/blog/2022/12/30/advancements-in-kubernetes-traffic-engineering/
• 

  Blog: Kubernetes 1.26: Job Tracking, to Support Massively Parallel Batch Workloads, Is Generally Available
  • sorangutan  

  1
  1
  Posts
  41
  Views

  

  Authors: Aldo Culquicondor (Google)

  The Kubernetes 1.26 release includes a stable implementation of the Job controller that can reliably track a large amount of Jobs with high levels of parallelism. SIG Apps and WG Batch have worked on this foundational improvement since Kubernetes 1.22. After multiple iterations and scale verifications, this is now the default implementation of the Job controller.

  Paired with the Indexed completion mode, the Job controller can handle massively parallel batch Jobs, supporting up to 100k concurrent Pods.

  The new implementation also made possible the development of Pod failure policy, which is in beta in the 1.26 release.

  How do I use this feature?

  To use Job tracking with finalizers, upgrade to Kubernetes 1.25 or newer and create new Jobs. You can also use this feature in v1.23 and v1.24, if you have the ability to enable the JobTrackingWithFinalizers feature gate.

  If your cluster runs Kubernetes 1.26, Job tracking with finalizers is a stable feature. For v1.25, it's behind that feature gate, and your cluster administrators may have explicitly disabled it - for example, if you have a policy of not using beta features.

  Jobs created before the upgrade will still be tracked using the legacy behavior. This is to avoid retroactively adding finalizers to running Pods, which might introduce race conditions.

  For maximum performance on large Jobs, the Kubernetes project recommends using the Indexed completion mode. In this mode, the control plane is able to track Job progress with less API calls.

  If you are a developer of operator(s) for batch, HPC, AI, ML or related workloads, we encourage you to use the Job API to delegate accurate progress tracking to Kubernetes. If there is something missing in the Job API that forces you to manage plain Pods, the Working Group Batch welcomes your feedback and contributions.

  Deprecation notices

  During the development of the feature, the control plane added the annotation batch.kubernetes.io/job-tracking to the Jobs that were created when the feature was enabled. This allowed a safe transition for older Jobs, but it was never meant to stay.

  In the 1.26 release, we deprecated the annotation batch.kubernetes.io/job-tracking and the control plane will stop adding it in Kubernetes 1.27. Along with that change, we will remove the legacy Job tracking implementation. As a result, the Job controller will track all Jobs using finalizers and it will ignore Pods that don't have the aforementioned finalizer.

  Before you upgrade your cluster to 1.27, we recommend that you verify that there are no running Jobs that don't have the annotation, or you wait for those jobs to complete. Otherwise, you might observe the control plane recreating some Pods. We expect that this shouldn't affect any users, as the feature is enabled by default since Kubernetes 1.25, giving enough buffer for old jobs to complete.

  What problem does the new implementation solve?

  Generally, Kubernetes workload controllers, such as ReplicaSet or StatefulSet, rely on the existence of Pods or other objects in the API to determine the status of the workload and whether replacements are needed. For example, if a Pod that belonged to a ReplicaSet terminates or ceases to exist, the ReplicaSet controller needs to create a replacement Pod to satisfy the desired number of replicas (.spec.replicas).

  Since its inception, the Job controller also relied on the existence of Pods in the API to track Job status. A Job has completion and failure handling policies, requiring the end state of a finished Pod to determine whether to create a replacement Pod or mark the Job as completed or failed. As a result, the Job controller depended on Pods, even terminated ones, to remain in the API in order to keep track of the status.

  This dependency made the tracking of Job status unreliable, because Pods can be deleted from the API for a number of reasons, including:

  • The garbage collector removing orphan Pods when a Node goes down.
  • The garbage collector removing terminated Pods when they reach a threshold.
  • The Kubernetes scheduler preempting a Pod to accomodate higher priority Pods.
  • The taint manager evicting a Pod that doesn't tolerate a NoExecute taint.
  • External controllers, not included as part of Kubernetes, or humans deleting Pods.

  The new implementation

  When a controller needs to take an action on objects before they are removed, it should add a finalizer to the objects that it manages. A finalizer prevents the objects from being deleted from the API until the finalizers are removed. Once the controller is done with the cleanup and accounting for the deleted object, it can remove the finalizer from the object and the control plane removes the object from the API.

  This is what the new Job controller is doing: adding a finalizer during Pod creation, and removing the finalizer after the Pod has terminated and has been accounted for in the Job status. However, it wasn't that simple.

  The main challenge is that there are at least two objects involved: the Pod and the Job. While the finalizer lives in the Pod object, the accounting lives in the Job object. There is no mechanism to atomically remove the finalizer in the Pod and update the counters in the Job status. Additionally, there could be more than one terminated Pod at a given time.

  To solve this problem, we implemented a three staged approach, each translating to an API call.

  1. For each terminated Pod, add the unique ID (UID) of the Pod into short-lived lists stored in the .status of the owning Job (.status.uncountedTerminatedPods).
  2. Remove the finalizer from the Pods(s).
  3. Atomically do the following operations:
     • remove UIDs from the short-lived lists
     • increment the overall succeeded and failed counters in the status of the Job.

  Additional complications come from the fact that the Job controller might receive the results of the API changes in steps 1 and 2 out of order. We solved this by adding an in-memory cache for removed finalizers.

  Still, we faced some issues during the beta stage, leaving some pods stuck with finalizers in some conditions (#108645, #109485, and #111646). As a result, we decided to switch that feature gate to be disabled by default for the 1.23 and 1.24 releases.

  Once resolved, we re-enabled the feature for the 1.25 release. Since then, we have received reports from our customers running tens of thousands of Pods at a time in their clusters through the Job API. Seeing this success, we decided to graduate the feature to stable in 1.26, as part of our long term commitment to make the Job API the best way to run large batch Jobs in a Kubernetes cluster.

  To learn more about the feature, you can read the KEP.

  Acknowledgments

  As with any Kubernetes feature, multiple people contributed to getting this done, from testing and filing bugs to reviewing code.

  On behalf of SIG Apps, I would like to especially thank Jordan Liggitt (Google) for helping me debug and brainstorm solutions for more than one race condition and Maciej Szulik (Red Hat) for his thorough reviews.

  
  
  https://kubernetes.io/blog/2022/12/29/scalable-job-tracking-ga/
• 

  Blog: Kubernetes v1.26: CPUManager goes GA
  • sorangutan  

  1
  1
  Posts
  17
  Views

  

  Author: Francesco Romani (Red Hat)

  The CPU Manager is a part of the kubelet, the Kubernetes node agent, which enables the user to allocate exclusive CPUs to containers. Since Kubernetes v1.10, where it graduated to Beta, the CPU Manager proved itself reliable and fulfilled its role of allocating exclusive CPUs to containers, so adoption has steadily grown making it a staple component of performance-critical and low-latency setups. Over time, most changes were about bugfixes or internal refactoring, with the following noteworthy user-visible changes:

  • support explicit reservation of CPUs: it was already possible to request to reserve a given number of CPUs for system resources, including the kubelet itself, which will not be used for exclusive CPU allocation. Now it is possible to also explicitly select which CPUs to reserve instead of letting the kubelet pick them up automatically.
  • report the exclusively allocated CPUs to containers, much like is already done for devices, using the kubelet-local PodResources API.
  • optimize the usage of system resources, eliminating unnecessary sysfs changes.

  The CPU Manager reached the point on which it "just works", so in Kubernetes v1.26 it has graduated to generally available (GA).

  Customization options for CPU Manager

  The CPU Manager supports two operation modes, configured using its policies. With the none policy, the CPU Manager allocates CPUs to containers without any specific constraint except the (optional) quota set in the Pod spec. With the static policy, then provided that the pod is in the Guaranteed QoS class and every container in that Pod requests an integer amount of vCPU cores, then the CPU Manager allocates CPUs exclusively. Exclusive assignment means that other containers (whether from the same Pod, or from a different Pod) do not get scheduled onto that CPU.

  This simple operational model served the user base pretty well, but as the CPU Manager matured more and more, users started to look at more elaborate use cases and how to better support them.

  Rather than add more policies, the community realized that pretty much all the novel use cases are some variation of the behavior enabled by the static CPU Manager policy. Hence, it was decided to add options to tune the behavior of the static policy. The options have a varying degree of maturity, like any other Kubernetes feature, and in order to be accepted, each new option provides a backward compatible behavior when disabled, and to document how to interact with each other, should they interact at all.

  This enabled the Kubernetes project to graduate to GA the CPU Manager core component and core CPU allocation algorithms to GA, while also enabling a new age of experimentation in this area. In Kubernetes v1.26, the CPU Manager supports three different policy options:

  full-pcpus-only

  restrict the CPU Manager core allocation algorithm to full physical cores only, reducing noisy neighbor issues from hardware technologies that allow sharing cores.

  distribute-cpus-across-numa

  drive the CPU Manager to evenly distribute CPUs across NUMA nodes, for cases where more than one NUMA node is required to satisfy the allocation.

  align-by-socket

  change how the CPU Manager allocates CPUs to a container: consider CPUs to be aligned at the socket boundary, instead of NUMA node boundary.

  Further development

  After graduating the main CPU Manager feature, each existing policy option will follow their graduation process, independent from CPU Manager and from each other option. There is room for new options to be added, but there's also a growing demand for even more flexibility than what the CPU Manager, and its policy options, currently grant.

  Conversations are in progress in the community about splitting the CPU Manager and the other resource managers currently part of the kubelet executable into pluggable, independent kubelet plugins. If you are interested in this effort, please join the conversation on SIG Node communication channels (Slack, mailing list, weekly meeting).

  Further reading

  Please check out the Control CPU Management Policies on the Node task page to learn more about the CPU Manager, and how it fits in relation to the other node-level resource managers.

  Getting involved

  This feature is driven by the SIG Node community. Please join us to connect with the community and share your ideas and feedback around the above feature and beyond. We look forward to hearing from you!

  
  
  https://kubernetes.io/blog/2022/12/27/cpumanager-ga/
• 

  Blog: Kubernetes 1.26: Pod Scheduling Readiness
  • sorangutan  

  1
  1
  Posts
  22
  Views

  

  Author: Wei Huang (Apple), Abdullah Gharaibeh (Google)

  Kubernetes 1.26 introduced a new Pod feature: scheduling gates. In Kubernetes, scheduling gates are keys that tell the scheduler when a Pod is ready to be considered for scheduling.

  What problem does it solve?

  When a Pod is created, the scheduler will continuously attempt to find a node that fits it. This infinite loop continues until the scheduler either finds a node for the Pod, or the Pod gets deleted.

  Pods that remain unschedulable for long periods of time (e.g., ones that are blocked on some external event) waste scheduling cycles. A scheduling cycle may take ≅20ms or more depending on the complexity of the Pod's scheduling constraints. Therefore, at scale, those wasted cycles significantly impact the scheduler's performance. See the arrows in the "scheduler" box below.

  graph LR; pod((New Pod))-->queue subgraph Scheduler queue(scheduler queue) sched_cycle[/scheduling cycle/] schedulable{schedulable?} queue==>|Pop out|sched_cycle sched_cycle==>schedulable schedulable==>|No|queue subgraph note [Cycles wasted on keep rescheduling 'unready' Pods] end end classDef plain fill:#ddd,stroke:#fff,stroke-width:1px,color:#000; classDef k8s fill:#326ce5,stroke:#fff,stroke-width:1px,color:#fff; classDef Scheduler fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; classDef note fill:#edf2ae,stroke:#fff,stroke-width:1px; class queue,sched_cycle,schedulable k8s; class pod plain; class note note; class Scheduler Scheduler;

  Scheduling gates helps address this problem. It allows declaring that newly created Pods are not ready for scheduling. When scheduling gates are present on a Pod, the scheduler ignores the Pod and therefore saves unnecessary scheduling attempts. Those Pods will also be ignored by Cluster Autoscaler if you have it installed in the cluster.

  Clearing the gates is the responsibility of external controllers with knowledge of when the Pod should be considered for scheduling (e.g., a quota manager).

  graph LR; pod((New Pod))-->queue subgraph Scheduler queue(scheduler queue) sched_cycle[/scheduling cycle/] schedulable{schedulable?} popout{Pop out?} queue==>|PreEnqueue check|popout popout-->|Yes|sched_cycle popout==>|No|queue sched_cycle-->schedulable schedulable-->|No|queue subgraph note [A knob to gate Pod's scheduling] end end classDef plain fill:#ddd,stroke:#fff,stroke-width:1px,color:#000; classDef k8s fill:#326ce5,stroke:#fff,stroke-width:1px,color:#fff; classDef Scheduler fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; classDef note fill:#edf2ae,stroke:#fff,stroke-width:1px; classDef popout fill:#f96,stroke:#fff,stroke-width:1px; class queue,sched_cycle,schedulable k8s; class pod plain; class note note; class popout popout; class Scheduler Scheduler;

  How does it work?

  Scheduling gates in general works very similar to Finalizers. Pods with a non-empty spec.schedulingGates field will show as status SchedulingGated and be blocked from scheduling. Note that more than one gate can be added, but they all should be added upon Pod creation (e.g., you can add them as part of the spec or via a mutating webhook).

  NAME READY STATUS RESTARTS AGE
  test-pod 0/1 SchedulingGated 0 10s
  

  To clear the gates, you update the Pod by removing all of the items from the Pod's schedulingGates field. The gates do not need to be removed all at once, but only when all the gates are removed the scheduler will start to consider the Pod for scheduling.

  Under the hood, scheduling gates are implemented as a PreEnqueue scheduler plugin, a new scheduler framework extension point that is invoked at the beginning of each scheduling cycle.

  Use Cases

  An important use case this feature enables is dynamic quota management. Kubernetes supports ResourceQuota, however the API Server enforces quota at the time you attempt Pod creation. For example, if a new Pod exceeds the CPU quota, it gets rejected. The API Server doesn't queue the Pod; therefore, whoever created the Pod needs to continuously attempt to recreate it again. This either means a delay between resources becoming available and the Pod actually running, or it means load on the API server and Scheduler due to constant attempts.

  Scheduling gates allows an external quota manager to address the above limitation of ResourceQuota. Specifically, the manager could add a example.com/quota-check scheduling gate to all Pods created in the cluster (using a mutating webhook). The manager would then remove the gate when there is quota to start the Pod.

  Whats next?

  To use this feature, the PodSchedulingReadiness feature gate must be enabled in the API Server and scheduler. You're more than welcome to test it out and tell us (SIG Scheduling) what you think!

  Additional resources

  • Pod Scheduling Readiness in the Kubernetes documentation
  • Kubernetes Enhancement Proposal

  
  
  https://kubernetes.io/blog/2022/12/26/pod-scheduling-readiness-alpha/
• 

  Blog: Kubernetes 1.26: Support for Passing Pod fsGroup to CSI Drivers At Mount Time
  • sorangutan  

  1
  1
  Posts
  9
  Views

  

  Authors: Fabio Bertinatto (Red Hat), Hemant Kumar (Red Hat)

  Delegation of fsGroup to CSI drivers was first introduced as alpha in Kubernetes 1.22, and graduated to beta in Kubernetes 1.25. For Kubernetes 1.26, we are happy to announce that this feature has graduated to General Availability (GA).

  In this release, if you specify a fsGroup in the security context, for a (Linux) Pod, all processes in the pod's containers are part of the additional group that you specified.

  In previous Kubernetes releases, the kubelet would always apply the fsGroup ownership and permission changes to files in the volume according to the policy you specified in the Pod's .spec.securityContext.fsGroupChangePolicy field.

  Starting with Kubernetes 1.26, CSI drivers have the option to apply the fsGroup settings during volume mount time, which frees the kubelet from changing the permissions of files and directories in those volumes.

  How does it work?

  CSI drivers that support this feature should advertise the VOLUME_MOUNT_GROUP node capability.

  After recognizing this information, the kubelet passes the fsGroup information to the CSI driver during pod startup. This is done through the NodeStageVolumeRequest and NodePublishVolumeRequest CSI calls.

  Consequently, the CSI driver is expected to apply the fsGroup to the files in the volume using a mount option. As an example, Azure File CSIDriver utilizes the gid mount option to map the fsGroup information to all the files in the volume.

  It should be noted that in the example above the kubelet refrains from directly applying the permission changes into the files and directories in that volume files. Additionally, two policy definitions no longer have an effect: neither .spec.fsGroupPolicy for the CSIDriver object, nor .spec.securityContext.fsGroupChangePolicy for the Pod.

  For more details about the inner workings of this feature, check out the enhancement proposal and the CSI Driver fsGroup Support in the CSI developer documentation.

  Why is it important?

  Without this feature, applying the fsGroup information to files is not possible in certain storage environments.

  For instance, Azure File does not support a concept of POSIX-style ownership and permissions of files. The CSI driver is only able to set the file permissions at the volume level.

  How do I use it?

  This feature should be mostly transparent to users. If you maintain a CSI driver that should support this feature, read CSI Driver fsGroup Support for more information on how to support this feature in your CSI driver.

  Existing CSI drivers that do not support this feature will continue to work as usual: they will not receive any fsGroup information from the kubelet. In addition to that, the kubelet will continue to perform the ownership and permissions changes to files for those volumes, according to the policies specified in .spec.fsGroupPolicy for the CSIDriver and .spec.securityContext.fsGroupChangePolicy for the relevant Pod.

  
  
  https://kubernetes.io/blog/2022/12/23/kubernetes-12-06-fsgroup-on-mount/
• 

  Blog: Kubernetes v1.26: GA Support for Kubelet Credential Providers
  • sorangutan  

  1
  1
  Posts
  43
  Views

  

  Authors: Andrew Sy Kim (Google), Dixita Narang (Google)

  Kubernetes v1.26 introduced generally available (GA) support for kubelet credential provider plugins, offering an extensible plugin framework to dynamically fetch credentials for any container image registry.

  Background

  Kubernetes supports the ability to dynamically fetch credentials for a container registry service. Prior to Kubernetes v1.20, this capability was compiled into the kubelet and only available for Amazon Elastic Container Registry, Azure Container Registry, and Google Cloud Container Registry.

  

  Figure 1: Kubelet built-in credential provider support for Amazon Elastic Container Registry, Azure Container Registry, and Google Cloud Container Registry.

  Kubernetes v1.20 introduced alpha support for kubelet credential providers plugins, which provides a mechanism for the kubelet to dynamically authenticate and pull images for arbitrary container registries - whether these are public registries, managed services, or even a self-hosted registry. In Kubernetes v1.26, this feature is now GA

  

  Figure 2: Kubelet credential provider overview

  Why is it important?

  Prior to Kubernetes v1.20, if you wanted to dynamically fetch credentials for image registries other than ACR (Azure Container Registry), ECR (Elastic Container Registry), or GCR (Google Container Registry), you needed to modify the kubelet code. The new plugin mechanism can be used in any cluster, and lets you authenticate to new registries without any changes to Kubernetes itself. Any cloud provider or vendor can publish a plugin that lets you authenticate with their image registry.

  How it works

  The kubelet and the exec plugin binary communicate through stdio (stdin, stdout, and stderr) by sending and receiving json-serialized api-versioned types. If the exec plugin is enabled and the kubelet requires authentication information for an image that matches against a plugin, the kubelet will execute the plugin binary, passing the CredentialProviderRequest API via stdin. Then the exec plugin communicates with the container registry to dynamically fetch the credentials and returns the credentials in an encoded response of the CredentialProviderResponse API to the kubelet via stdout.

  

  Figure 3: Kubelet credential provider plugin flow

  On receiving credentials from the kubelet, the plugin can also indicate how long credentials can be cached for, to prevent unnecessary execution of the plugin by the kubelet for subsequent image pull requests to the same registry. In cases where the cache duration is not specified by the plugin, a default cache duration can be specified by the kubelet (more details below).

  {
   "apiVersion": "kubelet.k8s.io/v1",
   "kind": "CredentialProviderResponse",
   "auth": {
   "cacheDuration": "6h",
   "private-registry.io/my-app": {
   "username": "exampleuser",
   "password": "token12345"
   }
   }
  }
  

  In addition, the plugin can specify the scope in which cached credentials are valid for. This is specified through the cacheKeyType field in CredentialProviderResponse. When the value is Image, the kubelet will only use cached credentials for future image pulls that exactly match the image of the first request. When the value is Registry, the kubelet will use cached credentials for any subsequent image pulls destined for the same registry host but using different paths (for example, gcr.io/foo/bar and gcr.io/bar/foo refer to different images from the same registry). Lastly, when the value is Global, the kubelet will use returned credentials for all images that match against the plugin, including images that can map to different registry hosts (for example, gcr.io vs k8s.gcr.io). The cacheKeyType field is required by plugin implementations.

  {
   "apiVersion": "kubelet.k8s.io/v1",
   "kind": "CredentialProviderResponse",
   "auth": {
   "cacheKeyType": "Registry",
   "private-registry.io/my-app": {
   "username": "exampleuser",
   "password": "token12345"
   }
   }
  }
  

  Using kubelet credential providers

  You can configure credential providers by installing the exec plugin(s) into a local directory accessible by the kubelet on every node. Then you set two command line arguments for the kubelet:

  • --image-credential-provider-config: the path to the credential provider plugin config file.
  • --image-credential-provider-bin-dir: the path to the directory where credential provider plugin binaries are located.

  The configuration file passed into --image-credential-provider-config is read by the kubelet to determine which exec plugins should be invoked for a container image used by a Pod. Note that the name of each provider must match the name of the binary located in the local directory specified in --image-credential-provider-bin-dir, otherwise the kubelet cannot locate the path of the plugin to invoke.

  kind: CredentialProviderConfig
  apiVersion: kubelet.config.k8s.io/v1
  providers:
  - name: auth-provider-gcp
   apiVersion: credentialprovider.kubelet.k8s.io/v1
   matchImages:
   - "container.cloud.google.com"
   - "gcr.io"
   - "*.gcr.io"
   - "*.pkg.dev"
   args:
   - get-credentials
   - --v=3
   defaultCacheDuration: 1m
  

  Below is an overview of how the Kubernetes project is using kubelet credential providers for end-to-end testing.

  

  Figure 4: Kubelet credential provider configuration used for Kubernetes e2e testing

  For more configuration details, see Kubelet Credential Providers.

  Getting Involved

  Come join SIG Node if you want to report bugs or have feature requests for the Kubelet Credential Provider. You can reach us through the following ways:

  • Slack: #sig-node
  • Mailing list
  • Open Community Issues/PRs
  • Biweekly meetings

  
  
  https://kubernetes.io/blog/2022/12/22/kubelet-credential-providers/
• 

  Blog: Kubernetes 1.26: Introducing Validating Admission Policies
  • sorangutan  

  1
  1
  Posts
  21
  Views

  

  Authors: Joe Betz (Google), Cici Huang (Google)

  In Kubernetes 1.26, the 1st alpha release of validating admission policies is available!

  Validating admission policies use the Common Expression Language (CEL) to offer a declarative, in-process alternative to validating admission webhooks.

  CEL was first introduced to Kubernetes for the Validation rules for CustomResourceDefinitions. This enhancement expands the use of CEL in Kubernetes to support a far wider range of admission use cases.

  Admission webhooks can be burdensome to develop and operate. Webhook developers must implement and maintain a webhook binary to handle admission requests. Also, admission webhooks are complex to operate. Each webhook must be deployed, monitored and have a well defined upgrade and rollback plan. To make matters worse, if a webhook times out or becomes unavailable, the Kubernetes control plane can become unavailable. This enhancement avoids much of this complexity of admission webhooks by embedding CEL expressions into Kubernetes resources instead of calling out to a remote webhook binary.

  For example, to set a limit on how many replicas a Deployment can have. Start by defining a validation policy:

  apiVersion: admissionregistration.k8s.io/v1alpha1
  kind: ValidatingAdmissionPolicy
  metadata:
   name: "demo-policy.example.com"
  spec:
   matchConstraints:
   resourceRules:
   - apiGroups: ["apps"]
   apiVersions: ["v1"]
   operations: ["CREATE", "UPDATE"]
   resources: ["deployments"]
   validations:
   - expression: "object.spec.replicas <= 5"
  

  The expression field contains the CEL expression that is used to validate admission requests. matchConstraints declares what types of requests this ValidatingAdmissionPolicy is may validate.

  Next bind the policy to the appropriate resources:

  apiVersion: admissionregistration.k8s.io/v1alpha1
  kind: ValidatingAdmissionPolicyBinding
  metadata:
   name: "demo-binding-test.example.com"
  spec:
   policy: "demo-policy.example.com"
   matchResources:
   namespaceSelector:
   - key: environment,
   operator: In,
   values: ["test"]
  

  This ValidatingAdmissionPolicyBinding resource binds the above policy only to namespaces where the environment label is set to test. Once this binding is created, the kube-apiserver will begin enforcing this admission policy.

  To emphasize how much simpler this approach is than admission webhooks, if this example were instead implemented with a webhook, an entire binary would need to be developed and maintained just to perform a <= check. In our review of a wide range of admission webhooks used in production, the vast majority performed relatively simple checks, all of which can easily be expressed using CEL.

  Validation admission policies are highly configurable, enabling policy authors to define policies that can be parameterized and scoped to resources as needed by cluster administrators.

  For example, the above admission policy can be modified to make it configurable:

  apiVersion: admissionregistration.k8s.io/v1alpha1
  kind: ValidatingAdmissionPolicy
  metadata:
   name: "demo-policy.example.com"
  spec:
   paramKind:
   apiVersion: rules.example.com/v1 # You also need a CustomResourceDefinition for this API
   kind: ReplicaLimit
   matchConstraints:
   resourceRules:
   - apiGroups: ["apps"]
   apiVersions: ["v1"]
   operations: ["CREATE", "UPDATE"]
   resources: ["deployments"]
   validations:
   - expression: "object.spec.replicas <= params.maxReplicas"
  

  Here, paramKind defines the resources used to configure the policy and the expression uses the params variable to access the parameter resource.

  This allows multiple bindings to be defined, each configured differently. For example:

  apiVersion: admissionregistration.k8s.io/v1alpha1
  kind: ValidatingAdmissionPolicyBinding
  metadata:
   name: "demo-binding-production.example.com"
  spec:
   policy: "demo-policy.example.com"
   paramsRef:
   name: "demo-params-production.example.com"
   matchResources:
   namespaceSelector:
   - key: environment,
   operator: In,
   values: ["production"]
  

  apiVersion: rules.example.com/v1 # defined via a CustomResourceDefinition
  kind: ReplicaLimit
  metadata:
   name: "demo-params-production.example.com"
  maxReplicas: 1000
  

  This binding and parameter resource pair limit deployments in namespaces with the environment label set to production to a max of 1000 replicas.

  You can then use a separate binding and parameter pair to set a different limit for namespaces in the test environment.

  I hope this has given you a glimpse of what is possible with validating admission policies! There are many features that we have not yet touched on.

  To learn more, read Validating Admission Policy.

  We are working hard to add more features to admission policies and make the enhancement easier to use. Try it out, send us your feedback and help us build a simpler alternative to admission webhooks!

  How do I get involved?

  If you want to get involved in development of admission policies, discuss enhancement roadmaps, or report a bug, you can get in touch with developers at SIG API Machinery.

  
  
  https://kubernetes.io/blog/2022/12/20/validating-admission-policies-alpha/
• 

  Blog: Kubernetes 1.26: Device Manager graduates to GA
  • sorangutan  

  1
  1
  Posts
  13
  Views

  

  Author: Swati Sehgal (Red Hat)

  The Device Plugin framework was introduced in the Kubernetes v1.8 release as a vendor independent framework to enable discovery, advertisement and allocation of external devices without modifying core Kubernetes. The feature graduated to Beta in v1.10. With the recent release of Kubernetes v1.26, Device Manager is now generally available (GA).

  Within the kubelet, the Device Manager facilitates communication with device plugins using gRPC through Unix sockets. Device Manager and Device plugins both act as gRPC servers and clients by serving and connecting to the exposed gRPC services respectively. Device plugins serve a gRPC service that kubelet connects to for device discovery, advertisement (as extended resources) and allocation. Device Manager connects to the Registration gRPC service served by kubelet to register itself with kubelet.

  Please refer to the documentation for an example on how a pod can request a device exposed to the cluster by a device plugin.

  Here are some example implementations of device plugins:

  • AMD GPU device plugin
  • Collection of Intel device plugins for Kubernetes
  • NVIDIA device plugin for Kubernetes
  • SRIOV network device plugin for Kubernetes

  Noteworthy developments since Device Plugin framework introduction

  Kubelet APIs moved to kubelet staging repo

  External facing deviceplugin API packages moved from k8s.io/kubernetes/pkg/kubelet/apis/ to k8s.io/kubelet/pkg/apis/ in v1.17. Refer to Move external facing kubelet apis to staging for more details on the rationale behind this change.

  Device Plugin API updates

  Additional gRPC endpoints introduced:

  1. GetDevicePluginOptions is used by device plugins to communicate options to the DeviceManager in order to indicate if PreStartContainer, GetPreferredAllocation or other future optional calls are supported and can be called before making devices available to the container.
  2. GetPreferredAllocation allows a device plugin to forward allocation preferrence to the DeviceManager so it can incorporate this information into its allocation decisions. The DeviceManager will call out to a plugin at pod admission time asking for a preferred device allocation of a given size from a list of available devices to make a more informed decision. E.g. Specifying inter-device constraints to indicate preferrence on best-connected set of devices when allocating devices to a container.
  3. PreStartContainer is called before each container start if indicated by device plugins during registration phase. It allows Device Plugins to run device specific operations on the Devices requested. E.g. reconfiguring or reprogramming FPGAs before the container starts running.

  Pull Requests that introduced these changes are here:

  1. Invoke preStart RPC call before container start, if desired by plugin
  2. Add GetPreferredAllocation() call to the v1beta1 device plugin API

  With introduction of the above endpoints the interaction between Device Manager in kubelet and Device Manager can be shown as below:

  

  Device Plugin framework Overview

  Change in semantics of device plugin registration process

  Device plugin code was refactored to separate 'plugin' package under the devicemanager package to lay the groundwork for introducing a v1beta2 device plugin API. This would allow adding support in devicemanager to service multiple device plugin APIs at the same time.

  With this refactoring work, it is now mandatory for a device plugin to start serving its gRPC service before registering itself with kubelet. Previously, these two operations were asynchronous and device plugin could register itself before starting its gRPC server which is no longer the case. For more details, refer to PR #109016 and Issue #112395.

  Dynamic resource allocation

  In Kubernetes 1.26, inspired by how Persistent Volumes are handled in Kubernetes, Dynamic Resource Allocation has been introduced to cater to devices that have more sophisticated resource requirements like:

  1. Decouple device initialization and allocation from the pod lifecycle.
  2. Facilitate dynamic sharing of devices between containers and pods.
  3. Support custom resource-specific parameters
  4. Enable resource-specific setup and cleanup actions
  5. Enable support for Network-attached resources, not just node-local resources

  Is the Device Plugin API stable now?

  No, the Device Plugin API is still not stable; the latest Device Plugin API version available is v1beta1. There are plans in the community to introduce v1beta2 API to service multiple plugin APIs at once. A per-API call with request/response types would allow adding support for newer API versions without explicitly bumping the API.

  In addition to that, there are existing proposals in the community to introduce additional endpoints KEP-3162: Add Deallocate and PostStopContainer to Device Manager API.

  
  
  https://kubernetes.io/blog/2022/12/19/devicemanager-ga/