Kubernetes News

• 

  Blog: Kubernetes 1.14: Production-level support for Windows Nodes, Kubectl Updates, Persistent Local Volumes GA
  • sorangutan  

  1
  1
  Posts
  29
  Views

  

  Authors: The 1.14 Release Team

  We’re pleased to announce the delivery of Kubernetes 1.14, our first release of 2019!

  Kubernetes 1.14 consists of 31 enhancements: 10 moving to stable, 12 in beta, and 7 net new. The main themes of this release are extensibility and supporting more workloads on Kubernetes with three major features moving to general availability, and an important security feature moving to beta.

  More enhancements graduated to stable in this release than any prior Kubernetes release. This represents an important milestone for users and operators in terms of setting support expectations. In addition, there are notable Pod and RBAC enhancements in this release, which are discussed in the “additional notable features” section below.

  Let’s dive into the key features of this release:

  Production-level Support for Windows Nodes

  Up until now Windows Node support in Kubernetes has been in beta, allowing many users to experiment and see the value of Kubernetes for Windows containers. Kubernetes now officially supports adding Windows nodes as worker nodes and scheduling Windows containers, enabling a vast ecosystem of Windows applications to leverage the power of our platform. Enterprises with investments in Windows-based applications and Linux-based applications don’t have to look for separate orchestrators to manage their workloads, leading to increased operational efficiencies across their deployments, regardless of operating system.

  Some of the key features of enabling Windows containers in Kubernetes include:

  • Support for Windows Server 2019 for worker nodes and containers
  • Support for out of tree networking with Azure-CNI, OVN-Kubernetes, and Flannel
  • Improved support for pods, service types, workload controllers, and metrics/quotas to closely match the capabilities offered for Linux containers

  Notable Kubectl Updates

  New Kubectl Docs and Logo

  The documentation for kubectl has been rewritten from the ground up with a focus on managing Resources using declarative Resource Config. The documentation has been published as a standalone site with the format of a book, and it is linked from the main k8s.io documentation (available at https://kubectl.docs.kubernetes.io).

  The new kubectl logo and mascot (pronounced kubee-cuddle) are shown on the new docs site logo.

  Kustomize Integration

  The declarative Resource Config authoring capabilities of kustomize are now available in kubectl through the -k flag (e.g. for commands like apply, get) and the kustomize subcommand. Kustomize helps users author and reuse Resource Config using Kubernetes native concepts. Users can now apply directories with kustomization.yaml to a cluster using kubectl apply -k dir/. Users can also emit customized Resource Config to stdout without applying them via kubectl kustomize dir/. The new capabilities are documented in the new docs at https://kubectl.docs.kubernetes.io

  The kustomize subcommand will continue to be developed in the Kubernetes owned kustomize repo. The latest kustomize features will be available from a standalone kustomize binary (published to the kustomize repo) at a frequent release cadence, and will be updated in kubectl prior to each Kubernetes releases.

  kubectl Plugin Mechanism Graduating to Stable

  The kubectl plugin mechanism allows developers to publish their own custom kubectl subcommands in the form of standalone binaries. This may be used to extend kubectl with new higher-level functionality and with additional porcelain (e.g. adding a set-ns command).

  Plugins must have the kubectl- name prefix and exist on the user’s $PATH. The plugin mechanics have been simplified significantly for GA, and are similar to the git plugin system.

  Persistent Local Volumes are Now GA

  This feature, graduating to stable, makes locally attached storage available as a persistent volume source. Distributed file systems and databases are the primary use cases for persistent local storage due performance and cost. On cloud providers, local SSDs give better performance than remote disks. On bare metal, in addition to performance, local storage is typically cheaper and using it is a necessity to provision distributed file systems.

  PID Limiting is Moving to Beta

  Process IDs (PIDs) are a fundamental resource on Linux hosts. It is trivial to hit the task limit without hitting any other resource limits and cause instability to a host machine. Administrators require mechanisms to ensure that user pods cannot induce PID exhaustion that prevents host daemons (runtime, kubelet, etc) from running. In addition, it is important to ensure that PIDs are limited among pods in order to ensure they have limited impact to other workloads on the node.

  Administrators are able to provide pod-to-pod PID isolation by defaulting the number of PIDs per pod as a beta feature. In addition, administrators can enable node-to-pod PID isolation as an alpha feature by reserving a number of allocatable PIDs to user pods via node allocatable. The community hopes to graduate this feature to beta in the next release.

  Additional Notable Feature Updates

  Pod priority and preemption enables Kubernetes scheduler to schedule more important Pods first and when cluster is out of resources, it removes less important pods to create room for more important ones. The importance is specified by priority.

  Pod Readiness Gates introduce an extension point for external feedback on pod readiness.

  Harden the default RBAC discovery clusterrolebindings removes discovery from the set of APIs which allow for unauthenticated access by default, improving privacy for CRDs and the default security posture of default clusters in general.

  Availability

  Kubernetes 1.14 is available for download on GitHub. To get started with Kubernetes, check out these interactive tutorials. You can also easily install 1.14 using kubeadm.

  Features Blog Series

  If you’re interested in exploring these features more in depth, check back next week for our 5 Days of Kubernetes series where we’ll highlight detailed walkthroughs of the following features:

  • Day 1 - Windows Server Containers
  • Day 2 - Harden the default RBAC discovery clusterrolebindings
  • Day 3 - Pod Priority and Preemption in Kubernetes
  • Day 4 - PID Limiting
  • Day 5 - Persistent Local Volumes

  Release Team

  This release is made possible through the efforts of hundreds of individuals who contributed both technical and non-technical content. Special thanks to the release team led by Aaron Crickenberger, Senior Test Engineer at Google. The 43 individuals on the release team coordinated many aspects of the release, from documentation to testing, validation, and feature completeness.

  As the Kubernetes community has grown, our release process represents an amazing demonstration of collaboration in open source software development. Kubernetes continues to gain new users at a rapid clip. This growth creates a positive feedback cycle where more contributors commit code creating a more vibrant ecosystem. Kubernetes has had over 28,000 individual contributors to date and an active community of more than 57,000 people.

  Project Velocity

  The CNCF has continued refining DevStats, an ambitious project to visualize the myriad contributions that go into the project. K8s DevStats illustrates the breakdown of contributions from major company contributors, as well as an impressive set of preconfigured reports on everything from individual contributors to pull request lifecycle times. On average over the past year, 381 different companies and over 2,458 individuals contribute to Kubernetes each month. Check out DevStats to learn more about the overall velocity of the Kubernetes project and community.

  User Highlights

  Established, global organizations are using Kubernetes in production at massive scale. Recently published user stories from the community include:

  • NetEase moving to Kubernetes has increased their R&D efficiency by more than 100% and deployment efficiency by 280%.
  • VSCO found that moving to continuous integration, containerization, and Kubernetes, velocity was increased dramatically. The time from code-complete to deployment in production on real infrastructure went from 1-2 weeks to 2-4 hours for a typical service.
  • NAV’s move to Kubernetes saved the company 50% in infrastructure costs, deployments increased 5x from 10 a day to 50 a day and resource utilization increased from 1% to 40%.

  Is Kubernetes helping your team? Share your story with the community.

  Ecosystem Updates

  • Kubernetes is participating in GSoC 2019 under the CNCF. Check out the full list of project ideas for 2019 on CNCF’s GitHub page.
  • The CNCF Annual report 2018 provides a look back at the growth of the Kubernetes and the foundation, community engagement, ecosystem tools, test conformance projects, KubeCon and more.
  • Out of 8,000 attendees at KubeCon + CloudNativeCon North America 2018, 73% were first-time KubeCon-ers, highlighting massive growth and new interest in Kubernetes and cloud native technologies. Conference transparency report.

  KubeCon

  The world’s largest Kubernetes gathering, KubeCon + CloudNativeCon is coming to Barcelona from May 20-23, 2019 and Shanghai (co-located with Open Source Summit) from June 24-26, 2019. These conferences will feature technical sessions, case studies, developer deep dives, salons, and more! Register today!

  Webinar

  Join members of the Kubernetes 1.14 release team on April 23rd at 10am PDT to learn about the major features in this release. Register here.

  Get Involved

  The simplest way to get involved with Kubernetes is by joining one of the many Special Interest Groups (SIGs) that align with your interests. Have something you’d like to broadcast to the Kubernetes community? Share your voice at our weekly community meeting, and through the channels below.

  Thank you for your continued feedback and support.

  • Post questions (or answer questions) on Stack Overflow
  • Join the community discussion on Discuss
  • Follow us on Twitter @Kubernetesio for latest updates
  • Join the community on Slack
  • Share your Kubernetes story

  
  
  https://kubernetes.io/blog/2019/03/25/kubernetes-1-14-release-announcement/
• 

  Blog: Kubernetes End-to-end Testing for Everyone
  • sorangutan  

  1
  1
  Posts
  39
  Views

  

  Author: Patrick Ohly (Intel)

  More and more components that used to be part of Kubernetes are now being developed outside of Kubernetes. For example, storage drivers used to be compiled into Kubernetes binaries, then were moved into stand-alone Flexvolume binaries on the host, and now are delivered as Container Storage Interface (CSI) drivers that get deployed in pods inside the Kubernetes cluster itself.

  This poses a challenge for developers who work on such components: how can end-to-end (E2E) testing on a Kubernetes cluster be done for such external components? The E2E framework that is used for testing Kubernetes itself has all the necessary functionality. However, trying to use it outside of Kubernetes was difficult and only possible by carefully selecting the right versions of a large number of dependencies. E2E testing has become a lot simpler in Kubernetes 1.13.

  This blog post summarizes the changes that went into Kubernetes 1.13. For CSI driver developers, it will cover the ongoing effort to also make the storage tests available for testing of third-party CSI drivers. How to use them will be shown based on two Intel CSI drivers:

  • Open Infrastructure Manager (OIM)
  • PMEM-CSI

  Testing those drivers was the main motivation behind most of these enhancements.

  E2E overview

  E2E testing consists of several phases:

  • Implementing a test suite. This is the main focus of this blog post. The Kubernetes E2E framework is written in Go. It relies on Ginkgo for managing tests and Gomega for assertions. These tools support “behavior driven development”, which describes expected behavior in “specs”. In this blog post, “test” is used to reference an individual Ginkgo.It spec. Tests interact with the Kubernetes cluster using client-go.
  • Bringing up a test cluster. Tools like kubetest can help here.
  • Running an E2E test suite against that cluster. Ginkgo test suites can be run with the ginkgo tool or as a normal Go test with go test. Without any parameters, a Kubernetes E2E test suite will connect to the default cluster based on environment variables like KUBECONFIG, exactly like kubectl. Kubetest also knows how to run the Kubernetes E2E suite.

  E2E framework enhancements in Kubernetes 1.13

  All of the following enhancements follow the same basic pattern: they make the E2E framework more useful and easier to use outside of Kubernetes, without changing the behavior of the original Kubernetes e2e.test binary.

  Splitting out provider support

  The main reason why using the E2E framework from Kubernetes <= 1.12 was difficult were the dependencies on provider-specific SDKs, which pulled in a large number of packages. Just getting it compiled was non-trivial.

  Many of these packages are only needed for certain tests. For example, testing the mounting of a pre-provisioned volume must first provision such a volume the same way as an administrator would, by talking directly to a specific storage backend via some non-Kubernetes API.

  There is an effort to remove cloud provider-specific tests from core Kubernetes. The approach taken in PR #68483 can be seen as an incremental step towards that goal: instead of ripping out the code immediately and breaking all tests that depend on it, all cloud provider-specific code was moved into optional packages under test/e2e/framework/providers. The E2E framework then accesses it via an interface that gets implemented separately by each vendor package.

  The author of a E2E test suite decides which of these packages get imported into the test suite. The vendor support is then activated via the --provider command line flag. The Kubernetes e2e.test binary in 1.13 and 1.14 still contains support for the same providers as in 1.12. It is also okay to include no packages, which means that only the generic providers will be available:

  • “skeleton”: cluster is accessed via the Kubernetes API and nothing else
  • “local”: like “skeleton”, but in addition the scripts in kubernetes/kubernetes/cluster can retrieve logs via ssh after a test suite is run

  External files

  Tests may have to read additional files at runtime, like .yaml manifests. But the Kubernetes e2e.test binary is supposed to be usable and entirely stand-alone because that simplifies shipping and running it. The solution in the Kubernetes build system is to link all files under test/e2e/testing-manifests into the binary with go-bindata. The E2E framework used to have a hard dependency on the output of go-bindata, now bindata support is optional. When accessing a file via the testfiles package, files will be retrieved from different sources:

  • relative to the directory specified with --repo-root parameter
  • zero or more bindata chunks

  Test parameters

  The e2e.test binary takes additional parameters which control test execution. In 2016, an effort was started to replace all E2E command line parameters with a Viper configuration file. But that effort stalled, which left developers without clear guidance how they should handle test-specific parameters.

  The approach in v1.12 was to add all flags to the central test/e2e/framework/test_context.go, which does not work for tests developed independently from the framework. Since PR #69105 the recommendation has been to use the normal flag package to define its parameters, in its own source code. Flag names must be hierarchical with dots separating different levels, for example my.test.parameter, and must be unique. Uniqueness is enforced by the flag package which panics when registering a flag a second time. The new config package simplifies the definition of multiple options, which are stored in a single struct.

  To summarize, this is how parameters are handled now:

  • The init code in test packages defines tests and parameters. The actual parameter values are not available yet, so test definitions cannot use them.
  • The init code of the test suite parses parameters and (optionally) the configuration file.
  • The tests run and now can use parameter values.

  However, recently it was pointed out that it is desirable and was possible to not expose test settings as command line flags and only set them via a configuration file. There is an open bug and a pending PR about this.

  Viper support has been enhanced. Like the provider support, it is completely optional. It gets pulled into a e2e.test binary by importing the viperconfig package and calling it after parsing the normal command line flags. This has been implemented so that all variables which can be set via command line flags are also set when the flag appears in a Viper config file. For example, the Kubernetes v1.13 e2e.test binary accepts --viper-config=/tmp/my-config.yaml and that file will set the my.test.parameter to value when it has this content: my: test: parameter: value

  In older Kubernetes releases, that option could only load a file from the current directory, the suffix had to be left out, and only a few parameters actually could be set this way. Beware that one limitation of Viper still exists: it works by matching config file entries against known flags, without warning about unknown config file entries and thus leaving typos undetected. A better config file parser for Kubernetes is still work in progress.

  Creating items from .yaml manifests

  In Kubernetes 1.12, there was some support for loading individual items from a .yaml file, but then creating that item had to be done by hand-written code. Now the framework has new methods for loading a .yaml file that has multiple items, patching those items (for example, setting the namespace created for the current test), and creating them. This is currently used to deploy CSI drivers anew for each test from exactly the same .yaml files that are also used for deployment via kubectl. If the CSI driver supports running under different names, then tests are completely independent and can run in parallel.

  However, redeploying a driver slows down test execution and it does not cover concurrent operations against the driver. A more realistic test scenario is to deploy a driver once when bringing up the test cluster, then run all tests against that deployment. Eventually the Kubernetes E2E testing will move to that model, once it is clearer how test cluster bringup can be extended such that it also includes installing additional entities like CSI drivers.

  Upcoming enhancements in Kubernetes 1.14

  Reusing storage tests

  Being able to use the framework outside of Kubernetes enables building a custom test suite. But a test suite without tests is still useless. Several of the existing tests, in particular for storage, can also be applied to out-of-tree components. Thanks to the work done by Masaki Kimura, storage tests in Kubernetes 1.13 are defined such that they can be instantiated multiple times for different drivers.

  But history has a habit of repeating itself. As with providers, the package defining these tests also pulled in driver definitions for all in-tree storage backends, which in turn pulled in more additional packages than were needed. This has been fixed for the upcoming Kubernetes 1.14.

  Skipping unsupported tests

  Some of the storage tests depend on features of the cluster (like running on a host that supports XFS) or of the driver (like supporting block volumes). These conditions are checked while the test runs, leading to skipped tests when they are not satisfied. The good thing is that this records an explanation why the test did not run.

  Starting a test is slow, in particular when it must first deploy the CSI driver, but also in other scenarios. Creating the namespace for a test has been measured at 5 seconds on a fast cluster, and it produces a lot of noisy test output. It would have been possible to address that by skipping the definition of unsupported tests, but then reporting why a test isn’t even part of the test suite becomes tricky. This approach has been dropped in favor of reorganizing the storage test suite such that it first checks conditions before doing the more expensive test setup steps.

  More readable test definitions

  The same PR also rewrites the tests to operate like conventional Ginkgo tests, with test cases and their local variables in a single function.

  Testing external drivers

  Building a custom E2E test suite is still quite a bit of work. The e2e.test binary that will get distributed in the Kubernetes 1.14 test archive will have the ability to test already installed storage drivers without rebuilding the test suite. See this README for further instructions.

  E2E test suite HOWTO

  Test suite initialization

  The first step is to set up the necessary boilerplate code that defines the test suite. In Kubernetes E2E, this is done in the e2e.go and e2e_test.go files. It could also be done in a single e2e_test.go file. Kubernetes imports all of the various providers, in-tree tests, Viper configuration support, and bindata file lookup in e2e_test.go. e2e.go controls the actual execution, including some cluster preparations and metrics collection.

  A simpler starting point are the e2e_[test].go files from PMEM-CSI. It doesn’t use any providers, no Viper, no bindata, and imports just the storage tests.

  Like PMEM-CSI, OIM drops all of the extra features, but is a bit more complex because it integrates a custom cluster startup directly into the test suite, which was useful in this case because some additional components have to run on the host side. By running them directly in the E2E binary, interactive debugging with dlv becomes easier.

  Both CSI drivers follow the Kubernetes example and use the test/e2e directory for their test suites, but any other directory and other file names would also work.

  Adding E2E storage tests

  Tests are defined by packages that get imported into a test suite. The only thing specific to E2E tests is that they instantiate a framework.Framework pointer (usually called f) with framework.NewDefaultFramework. This variable gets initialized anew in a BeforeEach for each test and freed in an AfterEach. It has a f.ClientSet and f.Namespace at runtime (and only at runtime!) which can be used by a test.

  The PMEM-CSI storage test imports the Kubernetes storage test suite and sets up one instance of the provisioning tests for a PMEM-CSI driver which must be already installed in the test cluster. The storage test suite changes the storage class to run tests with different filesystem types. Because of this requirement, the storage class is created from a .yaml file.

  Explaining all the various utility methods available in the framework is out of scope for this blog post. Reading existing tests and the source code of the framework is a good way to get started.

  Vendoring

  Vendoring Kubernetes code is still not trivial, even after eliminating many of the unnecessary dependencies. k8s.io/kubernetes is not meant to be included in other projects and does not define its dependencies in a way that is understood by tools like dep. The other k8s.io packages are meant to be included, but don’t follow semantic versioning yet or don’t tag any releases (k8s.io/kube-openapi, k8s.io/utils).

  PMEM-CSI uses dep. It’s Gopkg.toml file is a good starting point. It enables pruning (not enabled in dep by default) and locks certain projects onto versions that are compatible with the Kubernetes version that is used. When dep doesn’t pick a compatible version, then checking Kubernetes’ Godeps.json helps to determine which revision might be the right one.

  Compiling and running the test suite

  go test ./test/e2e -args -help is the fastest way to test that the test suite compiles.

  Once it does compile and a cluster has been set up, the command go test -timeout=0 -v ./test/e2e -ginkgo.v runs all tests. In order to run tests in parallel, use the ginkgo -p ./test/e2e command instead.

  Getting involved

  The Kubernetes E2E framework is owned by the testing-commons sub-project in SIG-testing. See that page for contact information.

  There are various tasks that could be worked on, including but not limited to:

  • Moving test/e2e/framework into a staging repo and restructuring it so that it is more modular (#74352).
  • Simplifying e2e.go by moving more of its code into test/e2e/framework (#74353).
  • Removing provider-specific code from the Kubernetes E2E test suite (#70194).

  Special thanks to the reviewers of this article:

  • Olev Kartau (https://github.com/okartau)
  • Mary Camp (https://github.com/MCamp859)

  
  
  https://kubernetes.io/blog/2019/03/22/kubernetes-end-to-end-testing-for-everyone/
• 

  Blog: A Guide to Kubernetes Admission Controllers
  • sorangutan  

  1
  1
  Posts
  36
  Views

  

  Author: Malte Isberner (StackRox)

  Kubernetes has greatly improved the speed and manageability of backend clusters in production today. Kubernetes has emerged as the de facto standard in container orchestrators thanks to its flexibility, scalability, and ease of use. Kubernetes also provides a range of features that secure production workloads. A more recent introduction in security features is a set of plugins called “admission controllers.” Admission controllers must be enabled to use some of the more advanced security features of Kubernetes, such as pod security policies that enforce a security configuration baseline across an entire namespace. The following must-know tips and tricks will help you leverage admission controllers to make the most of these security capabilities in Kubernetes.

  What are Kubernetes admission controllers?

  In a nutshell, Kubernetes admission controllers are plugins that govern and enforce how the cluster is used. They can be thought of as a gatekeeper that intercept (authenticated) API requests and may change the request object or deny the request altogether. The admission control process has two phases: the mutating phase is executed first, followed by the validating phase. Consequently, admission controllers can act as mutating or validating controllers or as a combination of both. For example, the LimitRanger admission controller can augment pods with default resource requests and limits (mutating phase), as well as verify that pods with explicitly set resource requirements do not exceed the per-namespace limits specified in the LimitRange object (validating phase).

  

  Admission Controller Phases

  It is worth noting that some aspects of Kubernetes’ operation that many users would consider built-in are in fact governed by admission controllers. For example, when a namespace is deleted and subsequently enters the Terminating state, the NamespaceLifecycle admission controller is what prevents any new objects from being created in this namespace.

  Among the more than 30 admission controllers shipped with Kubernetes, two take a special role because of their nearly limitless flexibility - ValidatingAdmissionWebhooks and MutatingAdmissionWebhooks, both of which are in beta status as of Kubernetes 1.13. We will examine these two admission controllers closely, as they do not implement any policy decision logic themselves. Instead, the respective action is obtained from a REST endpoint (a webhook) of a service running inside the cluster. This approach decouples the admission controller logic from the Kubernetes API server, thus allowing users to implement custom logic to be executed whenever resources are created, updated, or deleted in a Kubernetes cluster.

  The difference between the two kinds of admission controller webhooks is pretty much self-explanatory: mutating admission webhooks may mutate the objects, while validating admission webhooks may not. However, even a mutating admission webhook can reject requests and thus act in a validating fashion. Validating admission webhooks have two main advantages over mutating ones: first, for security reasons it might be desirable to disable the MutatingAdmissionWebhook admission controller (or apply stricter RBAC restrictions as to who may create MutatingWebhookConfiguration objects) because of its potentially confusing or even dangerous side effects. Second, as shown in the previous diagram, validating admission controllers (and thus webhooks) are run after any mutating ones. As a result, whatever request object a validating webhook sees is the final version that would be persisted to etcd.

  The set of enabled admission controllers is configured by passing a flag to the Kubernetes API server. Note that the old –admission-control flag was deprecated in 1.10 and replaced with –enable-admission-plugins.

  --enable-admission-plugins=ValidatingAdmissionWebhook,MutatingAdmissionWebhook
  

  Kubernetes recommends the following admission controllers to be enabled by default.

  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,Priority,ResourceQuota,PodSecurityPolicy
  

  The complete list of admission controllers with their descriptions can be found in the official Kubernetes reference. This discussion will focus only on the webhook-based admission controllers.

  Why do I need admission controllers?

  • Security: Admission controllers can increase security by mandating a reasonable security baseline across an entire namespace or cluster. The built-in PodSecurityPolicy admission controller is perhaps the most prominent example; it can be used for disallowing containers from running as root or making sure the container’s root filesystem is always mounted read-only, for example. Further use cases that can be realized by custom, webhook-based admission controllers include:
    • Allow pulling images only from specific registries known to the enterprise, while denying unknown image registries.
    • Reject deployments that do not meet security standards. For example, containers using the privileged flag can circumvent a lot of security checks. This risk could be mitigated by a webhook-based admission controller that either rejects such deployments (validating) or overrides the privileged flag, setting it to false.
  • Governance: Admission controllers allow you to enforce the adherence to certain practices such as having good labels, annotations, resource limits, or other settings. Some of the common scenarios include:
    • Enforce label validation on different objects to ensure proper labels are being used for various objects, such as every object being assigned to a team or project, or every deployment specifying an app label.
    • Automatically add annotations to objects, such as attributing the correct cost center for a “dev” deployment resource.
  • Configuration management: Admission controllers allow you to validate the configuration of the objects running in the cluster and prevent any obvious misconfigurations from hitting your cluster. Admission controllers can be useful in detecting and fixing images deployed without semantic tags, such as by:
    • automatically adding resource limits or validating resource limits,
    • ensuring reasonable labels are added to pods, or
    • ensuring image references used in production deployments are not using the latest tags, or tags with a -dev suffix.

  In this way, admission controllers and policy management help make sure that applications stay in compliance within an ever-changing landscape of controls.

  Example: Writing and Deploying an Admission Controller Webhook

  To illustrate how admission controller webhooks can be leveraged to establish custom security policies, let’s consider an example that addresses one of the shortcomings of Kubernetes: a lot of its defaults are optimized for ease of use and reducing friction, sometimes at the expense of security. One of these settings is that containers are by default allowed to run as root (and, without further configuration and no USER directive in the Dockerfile, will also do so). Even though containers are isolated from the underlying host to a certain extent, running containers as root does increase the risk profile of your deployment— and should be avoided as one of many security best practices. The recently exposed runC vulnerability (CVE-2019-5736), for example, could be exploited only if the container ran as root.

  You can use a custom mutating admission controller webhook to apply more secure defaults: unless explicitly requested, our webhook will ensure that pods run as a non-root user (we assign the user ID 1234 if no explicit assignment has been made). Note that this setup does not prevent you from deploying any workloads in your cluster, including those that legitimately require running as root. It only requires you to explicitly enable this risker mode of operation in the deployment configuration, while defaulting to non-root mode for all other workloads.

  The full code along with deployment instructions can be found in our accompanying GitHub repository. Here, we will highlight a few of the more subtle aspects about how webhooks work.

  Mutating Webhook Configuration

  A mutating admission controller webhook is defined by creating a MutatingWebhookConfiguration object in Kubernetes. In our example, we use the following configuration:

  apiVersion: admissionregistration.k8s.io/v1beta1
  kind: MutatingWebhookConfiguration
  metadata:
  name: demo-webhook
  webhooks:
  - name: webhook-server.webhook-demo.svc
  clientConfig:
  service:
  name: webhook-server
  namespace: webhook-demo
  path: "/mutate"
  caBundle: ${CA_PEM_B64}
  rules:
  - operations: [ "CREATE" ]
  apiGroups: [""]
  apiVersions: ["v1"]
  resources: ["pods"]
  

  This configuration defines a webhook webhook-server.webhook-demo.svc, and instructs the Kubernetes API server to consult the service webhook-server in namespace webhook-demo whenever a pod is created by making a HTTP POST request to the /mutate URL. For this configuration to work, several prerequisites have to be met.

  Webhook REST API

  The Kubernetes API server makes an HTTPS POST request to the given service and URL path, with a JSON-encoded AdmissionReview (with the Request field set) in the request body. The response should in turn be a JSON-encoded AdmissionReview, this time with the Response field set.

  Our demo repository contains a function that takes care of the serialization/deserialization boilerplate code and allows you to focus on implementing the logic operating on Kubernetes API objects. In our example, the function implementing the admission controller logic is called applySecurityDefaults, and an HTTPS server serving this function under the /mutate URL can be set up as follows:

  mux := http.NewServeMux()
  mux.Handle("/mutate", admitFuncHandler(applySecurityDefaults))
  server := &http.Server{
  Addr: ":8443",
  Handler: mux,
  }
  log.Fatal(server.ListenAndServeTLS(certPath, keyPath))
  

  Note that for the server to run without elevated privileges, we have the HTTP server listen on port 8443. Kubernetes does not allow specifying a port in the webhook configuration; it always assumes the HTTPS port 443. However, since a service object is required anyway, we can easily map port 443 of the service to port 8443 on the container:

  apiVersion: v1
  kind: Service
  metadata:
  name: webhook-server
  namespace: webhook-demo
  spec:
  selector:
  app: webhook-server # specified by the deployment/pod
  ports:
  - port: 443
  targetPort: webhook-api # name of port 8443 of the container
  

  Object Modification Logic

  In a mutating admission controller webhook, mutations are performed via JSON patches. While the JSON patch standard includes a lot of intricacies that go well beyond the scope of this discussion, the Go data structure in our example as well as its usage should give the user a good initial overview of how JSON patches work:

  type patchOperation struct {
  Op string `json:"op"`
  Path string `json:"path"`
  Value interface{} `json:"value,omitempty"`
  }
  

  For setting the field .spec.securityContext.runAsNonRoot of a pod to true, we construct the following patchOperation object:

  patches = append(patches, patchOperation{
  Op: "add",
  Path: "/spec/securityContext/runAsNonRoot",
  Value: true,
  })
  

  TLS Certificates

  Since a webhook must be served via HTTPS, we need proper certificates for the server. These certificates can be self-signed (rather: signed by a self-signed CA), but we need Kubernetes to instruct the respective CA certificate when talking to the webhook server. In addition, the common name (CN) of the certificate must match the server name used by the Kubernetes API server, which for internal services is <service-name>.<namespace>.svc, i.e., webhook-server.webhook-demo.svc in our case. Since the generation of self-signed TLS certificates is well documented across the Internet, we simply refer to the respective shell script in our example.

  The webhook configuration shown previously contains a placeholder ${CA_PEM_B64}. Before we can create this configuration, we need to replace this portion with the Base64-encoded PEM certificate of the CA. The openssl base64 -A command can be used for this purpose.

  Testing the Webhook

  After deploying the webhook server and configuring it, which can be done by invoking the ./deploy.sh script from the repository, it is time to test and verify that the webhook indeed does its job. The repository contains three examples:

  • A pod that does not specify a security context (pod-with-defaults). We expect this pod to be run as non-root with user id 1234.
  • A pod that does specify a security context, explicitly allowing it to run as root (pod-with-override).
  • A pod with a conflicting configuration, specifying it must run as non-root but with a user id of 0 (pod-with-conflict). To showcase the rejection of object creation requests, we have augmented our admission controller logic to reject such obvious misconfigurations.

  Create one of these pods by running kubectl create -f examples/<name>.yaml. In the first two examples, you can verify the user id under which the pod ran by inspecting the logs, for example:

  $ kubectl create -f examples/pod-with-defaults.yaml
  $ kubectl logs pod-with-defaults
  I am running as user 1234
  

  In the third example, the object creation should be rejected with an appropriate error message:

  $ kubectl create -f examples/pod-with-conflict.yaml
  Error from server (InternalError): error when creating "examples/pod-with-conflict.yaml": Internal error occurred: admission webhook "webhook-server.webhook-demo.svc" denied the request: runAsNonRoot specified, but runAsUser set to 0 (the root user)
  

  Feel free to test this with your own workloads as well. Of course, you can also experiment a little bit further by changing the logic of the webhook and see how the changes affect object creation. More information on how to do experiment with such changes can be found in the repository’s readme.

  Summary

  Kubernetes admission controllers offer significant advantages for security. Digging into two powerful examples, with accompanying available code, will help you get started on leveraging these powerful capabilities.

  References:

  • https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/
  • https://docs.okd.io/latest/architecture/additional_concepts/dynamic_admission_controllers.html
  • https://kubernetes.io/blog/2018/01/extensible-admission-is-beta/
  • https://medium.com/ibm-cloud/diving-into-kubernetes-mutatingadmissionwebhook-6ef3c5695f74
  • https://github.com/kubernetes/kubernetes/blob/v1.10.0-beta.1/test/images/webhook/main.go
  • https://github.com/istio/istio
  • https://www.stackrox.com/post/2019/02/the-runc-vulnerability-a-deep-dive-on-protecting-yourself/

  
  
  https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/
• 

  Blog: A Look Back and What's in Store for Kubernetes Contributor Summits
  • sorangutan  

  1
  1
  Posts
  39
  Views

  

  Authors: Paris Pittman (Google), Jonas Rosland (VMware)

  tl;dr - click here for Barcelona Contributor Summit information.

  

  Seattle Contributor Summit

  As our contributing community grows in great numbers, with more than 16,000 contributors this year across 150+ GitHub repositories, it’s important to provide face to face connections for our large distributed teams to have opportunities for collaboration and learning. In Contributor Experience, our methodology with planning events is a lot like our documentation; we build from personas – interests, skills, and motivators to name a few. This way we ensure there is valuable content and learning for everyone.

  We build the contributor summits around you:

  • New Contributor
  • Current Contributor
    • docs
    • code
    • community management
  • Subproject OWNERs
  • Special Interest Group (SIG) / Working Group (WG) Chair or Tech Lead
  • Active Contributors
  • Casual Contributors

  

  New Contributor Workshop

  These personas combined with ample feedback from previous events, produce the altogether experience that welcomed over 600 contributors in Copenhagen (May), Shanghai(November), and Seattle(December) in 2018. Seattle’s event drew over 300+ contributors, equal to Shanghai and Copenhagen combined, for the 6th contributor event in Kubernetes history. In true Kubernetes fashion, we expect another record breaking year of attendance. We’ve pre-ordered 900+ contributor patches, a tradition, and we are looking forward to giving them to you!

  With that said…
  Save the Dates:
  Barcelona: May 19th (evening) and 20th (all day)
  Shanghai: June 24th (all day)
  San Diego: November 18th, 19th, and activities in KubeCon/CloudNativeCon week

  In an effort of continual improvement, here’s what to expect from us this year:

  • Large new contributor workshops and contributor socials at all three events expected to break previous attendance records
  • A multiple track event in San Diego for all contributor types including workshops, birds of a feather, lightning talks and more
  • Addition of a “201” / “Intermediate” edition of the new contributor workshop in San Diego
  • An event website!
  • Follow along with updates: [email protected] is our main communication hub as always; however, we will also blog here, our Thursday Kubernetes Community Meeting, twitter, SIG meetings, event site, discuss.kubernetes.io, and #contributor-summit on Slack.
  • Opportunities to get involved: We still have 2019 roles available! Reach out to Contributor Experience via [email protected], stop by a Wednesday SIG update meeting, or catch us on Slack (#sig-contribex).
    

  

  Unconference voting

  Thanks!

  Our 2018 crew 🥁
  Jorge Castro, Paris Pittman, Bob Killen, Jeff Sica, Megan Lehn, Guinevere Saenger, Josh Berkus, Noah Abrahams, Yang Li, Xiangpeng Zhao, Puja Abbassi, Lindsey Tulloch, Zach Corleissen, Tim Pepper, Ihor Dvoretskyi, Nancy Mohamed, Chris Short, Mario Loria, Jason DeTiberus, Sahdev Zala, Mithra Raja

  And an introduction to our 2019 crew (a thanks in advance ;) )…
  Jonas Rosland, Josh Berkus, Paris Pittman, Jorge Castro, Bob Killen, Deb Giles, Guinevere Saenger, Noah Abrahams, Yang Li, Xiangpeng Zhao, Puja Abbassi, Rui Chen, Tim Pepper, Ihor Dvoretskyi, Dawn Foster

  Relive Seattle Contributor Summit

  📈 80% growth rate since the Austin 2017 December event
  📜 Event waiting list: 103
  🎓 76 contributors were on-boarded through the New Contributor Workshop
  🎉 92% of the current contributors RSVPs attended and of those:
  👩🏻‍🚒 25% were Special Interest Group or Working Group Chairs or Tech Leads
  🗳 70% were eligible to vote in the last steering committee election
  📹 20+ Sessions
  👀 Most watched to date: Technical Vision, Security, API Code Base Tour
  🌟 Top 3 according to survey: Live API Code Review, Deflaking Unconference, Technical Vision
  🎱 🎳 160 attendees for the social at Garage on Sunday night where we sunk eight balls and recorded strikes (out in some cases)
  🏆 Special recognition: SIG Storage, @dims, and @jordan
  📸 Pictures (special thanks to rdodev) Garage Pic Reg Desk

  

  Some of the group in Seattle

  “I love Contrib Summit! The intros and deep dives during KubeCon were a great extension of Contrib Summit. Y’all did an excellent job in the morning to level set expectations and prime everyone.” – julianv
  “great work! really useful and fun!” - coffeepac

  
  
  https://kubernetes.io/blog/2019/03/20/a-look-back-and-whats-in-store-for-kubernetes-contributor-summits/
• 

  Blog: KubeEdge, a Kubernetes Native Edge Computing Framework
  • sorangutan  

  1
  1
  Posts
  31
  Views

  

  Author: Sanil Kumar D (Huawei), Jun Du(Huawei)

  KubeEdge becomes the first Kubernetes Native Edge Computing Platform with both Edge and Cloud components open sourced!

  Open source edge computing is going through its most dynamic phase of development in the industry. So many open source platforms, so many consolidations and so many initiatives for standardization! This shows the strong drive to build better platforms to bring cloud computing to the edges to meet ever increasing demand. KubeEdge, which was announced last year, now brings great news for cloud native computing! It provides a complete edge computing solution based on Kubernetes with separate cloud and edge core modules. Currently, both the cloud and edge modules are open sourced.

  Unlike certain light weight kubernetes platforms available around, KubeEdge is made to build edge computing solutions extending the cloud. The control plane resides in cloud, though scalable and extendable. At the same time, the edge can work in offline mode. Also it is lightweight and containerized, and can support heterogeneous hardware at the edge. With the optimization in edge resource utlization, KubeEdge positions to save significant setup and operation cost for edge solutions. This makes it the most compelling edge computing platform in the world currently, based on Kubernetes!

  Kube(rnetes)Edge! - Opening up a new Kubernetes-based ecosystem for Edge Computing

  The key goal for KubeEdge is extending Kubernetes ecosystem from cloud to edge. From the time it was announced to the public at KubeCon in Shanghai in November 2018, the architecture direction for KubeEdge was aligned to Kubernetes, as its name!

  It started with its v0.1 providing the basic edge computing features. Now, with its latest release v0.2, it brings the cloud components to connect and complete the loop. With consistent and scalable Kubernetes-based interfaces, KubeEdge enables the orchestration and management of edge clusters similar to how Kubernetes manages in the cloud. This opens up seamless possibilities of bringing cloud computing capabilities to the edge, quickly and efficiently.

  

  KubeEdge Links:

  • Website
  • SourceCode
  • Documentation

  Based on its roadmap and architecture, KubeEdge tries to support all edge nodes, applications, devices and even the cluster management consistent with the Kuberenetes interface. This will help the edge cloud act exactly like a cloud cluster. This can save a lot of time and cost on the edge cloud development deployment based on KubeEdge.

  KubeEdge provides a containerized edge computing platform, which is inherently scalable. As it’s modular and optimized, it is lightweight (66MB foot print and ~30MB running memory) and could be deployed on low resource devices. Similarly, the edge node can be of different hardware architecture and with different hardware configurations. For the device connectivity, it can support multiple protocols and it uses a standard MQTT-based communication. This helps in scaling the edge clusters with new nodes and devices efficiently.

  You heard it right!

  KubeEdge Cloud Core modules are open sourced!

  By open sourcing both the edge and cloud modules, KubeEdge brings a complete cloud vendor agnostic lightweight heterogeneous edge computing platform. It is now ready to support building a complete Kubernetes ecosystem for edge computing, exploiting most of the existing cloud native projects or software modules. This can enable a mini-cloud at the edge to support demanding use cases like data analytics, video analytics, machine learning and more.

  KubeEdge Architecture: Building Kuberenetes Native Edge computing!

  The core architecture tenet for KubeEdge is to build interfaces that are consistent with Kubernetes, be it on the cloud side or edge side.

  

  Edged: Manages containerized Applications at the Edge.

  EdgeHub: Communication interface module at the Edge. It is a web socket client responsible for interacting with Cloud Service for edge computing.

  CloudHub: Communication interface module at the Cloud. A web socket server responsible for watching changes on the cloud side, caching and sending messages to EdgeHub.

  EdgeController: Manages the Edge nodes. It is an extended Kubernetes controller which manages edge nodes and pods metadata so that the data can be targeted to a specific edge node.

  EventBus: Handles the internal edge communications using MQTT. It is an MQTT client to interact with MQTT servers (mosquitto), offering publish and subscribe capabilities to other components.

  DeviceTwin: It is software mirror for devices that handles the device metadata. This module helps in handling device status and syncing the same to cloud. It also provides query interfaces for applications, as it interfaces to a lightweight database (SQLite).

  MetaManager: It manages the metadata at the edge node. This is the message processor between edged and edgehub. It is also responsible for storing/retrieving metadata to/from a lightweight database (SQLite).

  Even if you want to add more control plane modules based on the architecture refinement and improvement (for example enhanced security), it is simple as it uses consistent registration and modular communication within these modules.

  KubeEdge provides scalable lightweight Kubernetes Native Edge Computing Platform which can work in offline mode.

  It helps simplify edge application development and deployment.

  Cloud vendor agnostic and can run the cloud core modules on any compute node.

  Release 0.1 to 0.2 – game changer!

  KubeEdge v0.1 was released at the end of December 2018 with very basic edge features to manage edge applications along with Kubernetes API primitives for node, pod, config etc. In ~2 months, KubeEdge v0.2 was release on March 5th, 2019. This release provides the cloud core modules and enables the end to end open source edge computing solution. The cloud core modules can be deployed to any compute node from any cloud vendors or on-prem.

  Now, the complete edge solution can be installed and tested very easily, also with a laptop.

  Run Anywhere - Simple and Light

  As described, the KubeEdge Edge and Cloud core components can be deployed easily and can run the user applications. The edge core has a foot print of 66MB and just needs 30MB memory to run. Similarly the cloud core can run on any cloud nodes. (User can experience by running it on a laptop as well)

  The installation is simple and can be done in few steps:

  1. Setup the pre-requisites Docker, Kubernetes, MQTT and openssl
  2. Clone and Build KubeEdge Cloud and Edge
  3. Run Cloud
     
  4. Run Edge

  The detailed steps for each are available at KubeEdge/kubeedge

  Future: Taking off with competent features and community collaboration

  KubeEdge has been developed by members from the community who are active contributors to Kubernetes/CNCF and doing research in edge computing. The KubeEdge team is also actively collaborating with Kubernetes IOT/EDGE WORKING GROUP. Within a few months of the KubeEdge announcement it has attracted members from different organizations including JingDong, Zhejiang University, SEL Lab, Eclipse, China Mobile, ARM, Intel to collaborate in building the platform and ecosystem.

  KubeEdge has a clear roadmap for its upcoming major releases in 2019. vc1.0 targets to provide a complete edge cluster and device management solution with standard edge to edge communication, while v2.0 targets to have advanced features like service mesh, function service , data analytics etc at edge. Also, for all the features, KubeEdge architecture would attempt to utilize the existing CNCF projects/software.

  The KubeEdge community needs varied organizations, their requirements, use cases and support to build it. Please join to make a kubernetes native edge computing platform which can extend the cloud native computing paradigm to edge cloud.

  How to Get Involved?

  We welcome more collaboration to build the Kubernetes native edge computing ecosystem. Please join us!

  • Twitter: https://twitter.com/kubeedge
  • Slack: kubeedge.slack.com
  • Website: https://kubeedge.io
  • GitHub: https://github.com/kubeedge/kubeedge
  • Email: [email protected]

  
  
  https://kubernetes.io/blog/2019/03/19/kubeedge-k8s-based-edge-intro/
• 

  Blog: Kubernetes Setup Using Ansible and Vagrant
  • sorangutan  

  1
  1
  Posts
  58
  Views

  

  Author: Naresh L J (Infosys)

  Objective

  This blog post describes the steps required to setup a multi node Kubernetes cluster for development purposes. This setup provides a production-like cluster that can be setup on your local machine.

  Why do we require multi node cluster setup?

  Multi node Kubernetes clusters offer a production-like environment which has various advantages. Even though Minikube provides an excellent platform for getting started, it doesn’t provide the opportunity to work with multi node clusters which can help solve problems or bugs that are related to application design and architecture. For instance, Ops can reproduce an issue in a multi node cluster environment, Testers can deploy multiple versions of an application for executing test cases and verifying changes. These benefits enable teams to resolve issues faster which make the more agile.

  Why use Vagrant and Ansible?

  Vagrant is a tool that will allow us to create a virtual environment easily and it eliminates pitfalls that cause the works-on-my-machine phenomenon. It can be used with multiple providers such as Oracle VirtualBox, VMware, Docker, and so on. It allows us to create a disposable environment by making use of configuration files.

  Ansible is an infrastructure automation engine that automates software configuration management. It is agentless and allows us to use SSH keys for connecting to remote machines. Ansible playbooks are written in yaml and offer inventory management in simple text files.

  Prerequisites

  • Vagrant should be installed on your machine. Installation binaries can be found here.
  • Oracle VirtualBox can be used as a Vagrant provider or make use of similar providers as described in Vagrant’s official documentation.
  • Ansible should be installed in your machine. Refer to the Ansible installation guide for platform specific installation.

  Setup overview

  We will be setting up a Kubernetes cluster that will consist of one master and two worker nodes. All the nodes will run Ubuntu Xenial 64-bit OS and Ansible playbooks will be used for provisioning.

  Step 1: Creating a Vagrantfile

  Use the text editor of your choice and create a file with named Vagrantfile, inserting the code below. The value of N denotes the number of nodes present in the cluster, it can be modified accordingly. In the below example, we are setting the value of N as 2.

  IMAGE_NAME = "bento/ubuntu-16.04"
  N = 2
  Vagrant.configure("2") do |config|
  config.ssh.insert_key = false
  config.vm.provider "virtualbox" do |v|
  v.memory = 1024
  v.cpus = 2
  end
  config.vm.define "k8s-master" do |master|
  master.vm.box = IMAGE_NAME
  master.vm.network "private_network", ip: "192.168.50.10"
  master.vm.hostname = "k8s-master"
  master.vm.provision "ansible" do |ansible|
  ansible.playbook = "kubernetes-setup/master-playbook.yml"
  end
  end
  (1..N).each do |i|
  config.vm.define "node-#{i}" do |node|
  node.vm.box = IMAGE_NAME
  node.vm.network "private_network", ip: "192.168.50.#{i + 10}"
  node.vm.hostname = "node-#{i}"
  node.vm.provision "ansible" do |ansible|
  ansible.playbook = "kubernetes-setup/node-playbook.yml"
  end
  end
  end

  Step 2: Create an Ansible playbook for Kubernetes master.

  Create a directory named kubernetes-setup in the same directory as the Vagrantfile. Create two files named master-playbook.yml and node-playbook.yml in the directory kubernetes-setup.

  In the file master-playbook.yml, add the code below.

  Step 2.1: Install Docker and its dependent components.

  We will be installing the following packages, and then adding a user named “vagrant” to the “docker” group. - docker-ce - docker-ce-cli - containerd.io

  ---
  - hosts: all
   become: true
   tasks:
   - name: Install packages that allow apt to be used over HTTPS
   apt:
   name: "{{ packages }}"
   state: present
   update_cache: yes
   vars:
   packages:
   - apt-transport-https
   - ca-certificates
   - curl
   - gnupg-agent
   - software-properties-common
  
   - name: Add an apt signing key for Docker
   apt_key:
   url: https://download.docker.com/linux/ubuntu/gpg
   state: present
  
   - name: Add apt repository for stable version
   apt_repository:
   repo: deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable
   state: present
  
   - name: Install docker and its dependecies
   apt:
   name: "{{ packages }}"
   state: present
   update_cache: yes
   vars:
   packages:
   - docker-ce
   - docker-ce-cli
   - containerd.io
   notify:
   - docker status
  
   - name: Add vagrant user to docker group
   user:
   name: vagrant
   group: docker

  Step 2.2: Kubelet will not start if the system has swap enabled, so we are disabling swap using the below code.

   - name: Remove swapfile from /etc/fstab
   mount:
   name: "{{ item }}"
   fstype: swap
   state: absent
   with_items:
   - swap
   - none
  
   - name: Disable swap
   command: swapoff -a
   when: ansible_swaptotal_mb > 0

  Step 2.3: Installing kubelet, kubeadm and kubectl using the below code.

   - name: Add an apt signing key for Kubernetes
   apt_key:
   url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
   state: present
  
   - name: Adding apt repository for Kubernetes
   apt_repository:
   repo: deb https://apt.kubernetes.io/ kubernetes-xenial main
   state: present
   filename: kubernetes.list
  
   - name: Install Kubernetes binaries
   apt:
   name: "{{ packages }}"
   state: present
   update_cache: yes
   vars:
   packages:
   - kubelet
   - kubeadm
   - kubectl

  Step 2.3: Initialize the Kubernetes cluster with kubeadm using the below code (applicable only on master node).

   - name: Initialize the Kubernetes cluster using kubeadm
   command: kubeadm init --apiserver-advertise-address="192.168.50.10" --apiserver-cert-extra-sans="192.168.50.10" --node-name k8s-master --pod-network-cidr=192.168.0.0/16

  Step 2.4: Setup the kube config file for the vagrant user to access the Kubernetes cluster using the below code.

   - name: Setup kubeconfig for vagrant user
   command: "{{ item }}"
   with_items:
   - mkdir -p /home/vagrant/.kube
   - cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config
   - chown vagrant:vagrant /home/vagrant/.kube/config

  Step 2.5: Setup the container networking provider and the network policy engine using the below code.

   - name: Install calico pod network
   become: false
   command: kubectl create -f https://docs.projectcalico.org/v3.4/getting-started/kubernetes/installation/hosted/calico.yaml

  Step 2.6: Generate kube join command for joining the node to the Kubernetes cluster and store the command in the file named join-command.

   - name: Generate join command
   command: kubeadm token create --print-join-command
   register: join_command
  
   - name: Copy join command to local file
   local_action: copy content="{{ join_command.stdout_lines[0] }}" dest="./join-command"

  Step 2.7: Setup a handler for checking Docker daemon using the below code.

   handlers:
   - name: docker status
   service: name=docker state=started

  Step 3: Create the Ansible playbook for Kubernetes node.

  Create a file named node-playbook.yml in the directory kubernetes-setup.

  Add the code below into node-playbook.yml

  Step 3.1: Start adding the code from Steps 2.1 till 2.3.

  Step 3.2: Join the nodes to the Kubernetes cluster using below code.

   - name: Copy the join command to server location
   copy: src=join-command dest=/tmp/join-command.sh mode=0777
  
   - name: Join the node to cluster
   command: sh /tmp/join-command.sh

  Step 3.3: Add the code from step 2.7 to finish this playbook.

  Step 4: Upon completing the Vagrantfile and playbooks follow the below steps.

  $ cd /path/to/Vagrantfile
  $ vagrant up

  Upon completion of all the above steps, the Kubernetes cluster should be up and running. We can login to the master or worker nodes using Vagrant as follows:

  $ ## Accessing master
  $ vagrant ssh k8s-master
  vagrant@k8s-master:~$ kubectl get nodes
  NAME STATUS ROLES AGE VERSION
  k8s-master Ready master 18m v1.13.3
  node-1 Ready <none> 12m v1.13.3
  node-2 Ready <none> 6m22s v1.13.3
  $ ## Accessing nodes
  $ vagrant ssh node-1
  $ vagrant ssh node-2

  
  
  https://kubernetes.io/blog/2019/03/15/kubernetes-setup-using-ansible-and-vagrant/
• 

  Blog: A Look Back and What's in Store for Kubernetes Contributor Summits
  • sorangutan  

  1
  1
  Posts
  45
  Views

  

  Authors: Paris Pittman (Google), Jonas Rosland (VMware)

  tl;dr - click here for Barcelona Contributor Summit information.

  

  Seattle Contributor Summit

  As our contributing community grows in great numbers, with more than 16,000 contributors this year across 150+ GitHub repositories, it’s important to provide face to face connections for our large distributed teams to have opportunities for collaboration and learning. In Contributor Experience, our methodology with planning events is a lot like our documentation; we build from personas – interests, skills, and motivators to name a few. This way we ensure there is valuable content and learning for everyone.

  We build the contributor summits around you: * New Contributor * Current Contributor * docs * code * community management * Subproject OWNERs * Special Interest Group (SIG) / Working Group (WG) Chair or Tech Lead * Active Contributors * Casual Contributors

  

  New Contributor Workshop

  These personas combined with ample feedback from previous events, produce the altogether experience that welcomed over 600 contributors in Copenhagen (May), Shanghai(November), and Seattle(December) in 2018. Seattle’s event drew over 300+ contributors, equal to Shanghai and Copenhagen combined, for the 6th contributor event in Kubernetes history. In true Kubernetes fashion, we expect another record breaking year of attendance. We’ve pre-ordered 900+ contributor patches, a tradition, and we are looking forward to giving them to you!

  With that said…
  Save the Dates:
  Barcelona: May 19th (evening) and 20th (all day)
  Shanghai: June 24th (all day)
  San Diego: November 18th, 19th, and activities in KubeCon/CloudNativeCon week

  In an effort of continual improvement, here’s what to expect from us this year:
  * Large new contributor workshops and contributor socials at all three events expected to break previous attendance records * A multiple track event in San Diego for all contributor types including workshops, birds of a feather, lightning talks and more * Addition of a “201” / “Intermediate” edition of the new contributor workshop in San Diego * An event website! * Follow along with updates: [email protected] is our main communication hub as always; however, we will also blog here, our Thursday Kubernetes Community Meeting, twitter, SIG meetings, event site, discuss.kubernetes.io, and #contributor-summit on Slack. * Opportunities to get involved: We still have 2019 roles available! Reach out to Contributor Experience via [email protected], stop by a Wednesday SIG update meeting, or catch us on Slack (#sig-contribex).

  

  Unconference voting

  Thanks!

  Our 2018 crew 🥁
  Jorge Castro, Paris Pittman, Bob Killen, Jeff Sica, Megan Lehn, Guinevere Saenger, Josh Berkus, Noah Abrahams, Yang Li, Xiangpeng Zhao, Puja Abbassi, Lindsey Tulloch, Zach Corleissen, Tim Pepper, Ihor Dvoretskyi, Nancy Mohamed, Chris Short, Mario Loria, Jason DeTiberus, Sahdev Zala, Mithra Raja

  And an introduction to our 2019 crew (a thanks in advance ;) )…
  Jonas Rosland, Josh Berkus, Paris Pittman, Jorge Castro, Bob Killen, Deb Giles, Guinevere Saenger, Noah Abrahams, Yang Li, Xiangpeng Zhao, Puja Abbassi, Rui Chen, Tim Pepper, Ihor Dvoretskyi, Dawn Foster

  Relive Seattle Contributor Summit

  📈 80% growth rate since the Austin 2017 December event
  📜 Event waiting list: 103
  🎓 76 contributors were on-boarded through the New Contributor Workshop
  🎉 92% of the current contributors RSVPs attended and of those:
  👩🏻‍🚒 25% were Special Interest Group or Working Group Chairs or Tech Leads
  🗳 70% were eligible to vote in the last steering committee election

  📹 20+ Sessions
  👀 Most watched to date: Technical Vision, Security, API Code Base Tour
  🌟 Top 3 according to survey: Live API Code Review, Deflaking Unconference, Technical Vision

  🎱 🎳 160 attendees for the social at Garage on Sunday night where we sunk eight balls and recorded strikes (out in some cases)
  🏆 Special recognition: SIG Storage, @dims, and @jordan
  📸 Pictures (special thanks to rdodev) Garage Pic Reg Desk

  

  Some of the group in Seattle

  “I love Contrib Summit! The intros and deep dives during KubeCon were a great extension of Contrib Summit. Y’all did an excellent job in the morning to level set expectations and prime everyone.” – julianv
  “great work! really useful and fun!” - coffeepac

  
  
  https://kubernetes.io/blog/2019/03/14/a-look-back-and-whats-in-store-for-kubernetes-contributor-summits/
• 

  Blog: Raw Block Volume support to Beta
  • sorangutan  

  1
  1
  Posts
  43
  Views

  

  Authors: Ben Swartzlander (NetApp), Saad Ali (Google)

  Kubernetes v1.13 moves raw block volume support to beta. This feature allows persistent volumes to be exposed inside containers as a block device instead of as a mounted file system.

  What are block devices?

  Block devices enable random access to data in fixed-size blocks. Hard drives, SSDs, and CD-ROMs drives are all examples of block devices.

  Typically persistent storage is implemented in a layered maner with a file system (like ext4) on top of a block device (like a spinning disk or SSD). Applications then read and write files instead of operating on blocks. The operating systems take care of reading and writing files, using the specified filesystem, to the underlying device as blocks.

  It’s worth noting that while whole disks are block devices, so are disk partitions, and so are LUNs from a storage area network (SAN) device.

  Why add raw block volumes to kubernetes?

  There are some specialized applications that require direct access to a block device because, for example, the file system layer introduces unneeded overhead. The most common case is databases, which prefer to organize their data directly on the underlying storage. Raw block devices are also commonly used by any software which itself implements some kind of storage service (software defined storage systems).

  From a programmer’s perspective, a block device is a very large array of bytes, usually with some minimum granularity for reads and writes, often 512 bytes, but frequently 4K or larger.

  As it becomes more common to run database software and storage infrastructure software inside of Kubernetes, the need for raw block device support in Kubernetes becomes more important.

  Which volume plugins support raw blocks?

  As of the publishing of this blog, the following in-tree volumes types support raw blocks:

  • AWS EBS
  • Azure Disk
  • Cinder
  • Fibre Channel
  • GCE PD
  • iSCSI
  • Local volumes
  • RBD (Ceph)
  • Vsphere

  Out-of-tree CSI volume drivers may also support raw block volumes. Kubernetes CSI support for raw block volumes is currently alpha. See documentation here.

  Kubernetes raw block volume API

  Raw block volumes share a lot in common with ordinary volumes. Both are requested by creating PersistentVolumeClaim objects which bind to PersistentVolume objects, and are attached to Pods in Kubernetes by including them in the volumes array of the PodSpec.

  There are 2 important differences however. First, to request a raw block PersistentVolumeClaim, you must set volumeMode = "Block" in the PersistentVolumeClaimSpec. Leaving volumeMode blank is the same as specifying volumeMode = "Filesystem" which results in the traditional behavior. PersistentVolumes also have a volumeMode field in their PersistentVolumeSpec, and "Block" type PVCs can only bind to "Block" type PVs and "Filesystem" PVCs can only bind to "Filesystem" PVs.

  Secondly, when using a raw block volume in your Pods, you must specify a VolumeDevice in the Container portion of the PodSpec rather than a VolumeMount. VolumeDevices have devicePaths instead of mountPaths, and inside the container, applications will see a device at that path instead of a mounted file system.

  Applications open, read, and write to the device node inside the container just like they would interact with any block device on a system in a non-containerized or virtualized context.

  Creating a new raw block PVC

  First, ensure that the provisioner associated with the storage class you choose is one that support raw blocks. Then create the PVC.

  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
  name: my-pvc
  spec:
  accessModes:
  - ReadWriteMany
  volumeMode: Block
  storageClassName: my-sc
  resources:
  requests:
  storage: 1Gi
  

  Using a raw block PVC

  When you use the PVC in a pod definition, you get to choose the device path for the block device rather than the mount path for the file system.

  apiVersion: v1
  kind: Pod
  metadata:
  name: my-pod
  spec:
  containers:
  - name: my-container
  image: busybox
  command:
  - sleep
  - “3600”
  volumeDevices:
  - devicePath: /dev/block
  name: my-volume
  imagePullPolicy: IfNotPresent
  volumes:
  - name: my-volume
  persistentVolumeClaim:
  claimName: my-pvc
  

  As a storage vendor, how do I add support for raw block devices to my CSI plugin?

  Raw block support for CSI plugins is still alpha, but support can be added today. The CSI specification details how to handle requests for volume that have the BlockVolume capability instead of the MountVolume capability. CSI plugins can support both kinds of volumes, or one or the other. For more details see documentation here.

  Issues/gotchas

  Because block devices are actually devices, it’s possible to do low-level actions on them from inside containers that wouldn’t be possible with file system volumes. For example, block devices that are actually SCSI disks support sending SCSI commands to the device using Linux ioctls.

  By default, Linux won’t allow containers to send SCSI commands to disks from inside containers though. In order to do so, you must grant the SYS_RAWIO capability to the container security context to allow this. See documentation here.

  Also, while Kubernetes is guaranteed to deliver a block device to the container, there’s no guarantee that it’s actually a SCSI disk or any other kind of disk for that matter. The user must either ensure that the desired disk type is used with his pods, or only deploy applications that can handle a variety of block device types.

  How can I learn more?

  Check out additional documentation on the snapshot feature here: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#raw-block-volume-support

  How do I get involved?

  Join the Kubernetes storage SIG and the CSI community and help us add more great features and improve existing ones like raw block storage!

  https://github.com/kubernetes/community/tree/master/sig-storage https://github.com/container-storage-interface/community/blob/master/README.md

  Special thanks to all the contributors who helped add block volume support to Kubernetes including:

  • Ben Swartzlander (https://github.com/bswartz)
  • Brad Childs (https://github.com/childsb)
  • Erin Boyd (https://github.com/erinboyd)
  • Masaki Kimura (https://github.com/mkimuram)
  • Matthew Wong (https://github.com/wongma7)
  • Michelle Au (https://github.com/msau42)
  • Mitsuhiro Tanino (https://github.com/mtanino)
  • Saad Ali (https://github.com/saad-ali)

  
  
  https://kubernetes.io/blog/2019/03/07/raw-block-volume-support-to-beta/
• 

  Blog: Automate Operations on your Cluster with OperatorHub.io
  • sorangutan  

  1
  1
  Posts
  36
  Views

  

  Author: Diane Mueller, Director of Community Development, Cloud Platforms, Red Hat

  One of the important challenges facing developers and Kubernetes administrators has been a lack of ability to quickly find common services that are operationally ready for Kubernetes. Typically, the presence of an Operator for a specific service - a pattern that was introduced in 2016 and has gained momentum - is a good signal for the operational readiness of the service on Kubernetes. However, there has to date not existed a registry of Operators to simplify the discovery of such services.

  To help address this challenge, today Red Hat is launching OperatorHub.io in collaboration with AWS, Google Cloud and Microsoft. OperatorHub.io enables developers and Kubernetes administrators to find and install curated Operator-backed services with a base level of documentation, active maintainership by communities or vendors, basic testing, and packaging for optimized life-cycle management on Kubernetes.

  The Operators currently in OperatorHub.io are just the start. We invite the Kubernetes community to join us in building a vibrant community for Operators by developing, packaging, and publishing Operators on OperatorHub.io.

  What does OperatorHub.io provide?

  OperatorHub.io is designed to address the needs of both Kubernetes developers and users. For the former it provides a common registry where they can publish their Operators alongside with descriptions, relevant details like version, image, code repository and have them be readily packaged for installation. They can also update already published Operators to new versions when they are released.

  Users get the ability to discover and download Operators at a central location, that has content which has been screened for the previously mentioned criteria and scanned for known vulnerabilities. In addition, developers can guide users of their Operators with prescriptive examples of the CustomResources that they introduce to interact with the application.

  What is an Operator?

  Operators were first introduced in 2016 by CoreOS and have been used by Red Hat and the Kubernetes community as a way to package, deploy and manage a Kubernetes-native application. A Kubernetes-native application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and well-known tooling, like kubectl.

  An Operator is implemented as a custom controller that watches for certain Kubernetes resources to appear, be modified or deleted. These are typically CustomResourceDefinitions that the Operator “owns.” In the spec properties of these objects the user declares the desired state of the application or the operation. The Operator’s reconciliation loop will pick these up and perform the required actions to achieve the desired state. For example, the intent to create a highly available etcd cluster could be expressed by creating an new resource of type EtcdCluster:

  apiVersion: "etcd.database.coreos.com/v1beta2"
  kind: "EtcdCluster"
  metadata:
  name: "my-etcd-cluster"
  spec:
  size: 3
  version: "3.3.12"
  

  The EtcdOperator would be responsible for creating a 3-node etcd cluster running version v3.3.12 as a result. Similarly, an object of type EtcdBackup could be defined to express the intent to create a consistent backup of the etcd database to an S3 bucket.

  How do I create and run an Operator?

  One way to get started is with the Operator Framework, an open source toolkit that provides an SDK, lifecycle management, metering and monitoring capabilities. It enables developers to build, test, and package Operators. Operators can be implemented in several programming and automation languages, including Go, Helm, and Ansible, all three of which are supported directly by the SDK.

  If you are interested in creating your own Operator, we recommend checking out the Operator Framework to get started.

  Operators vary in where they fall along the capability spectrum ranging from basic functionality to having specific operational logic for an application to automate advanced scenarios like backup, restore or tuning. Beyond basic installation, advanced Operators are designed to handle upgrades more seamlessly and react to failures automatically. Currently, Operators on OperatorHub.io span the maturity spectrum, but we anticipate their continuing maturation over time.

  While Operators on OperatorHub.io don’t need to be implemented using the SDK, they are packaged for deployment through the Operator Lifecycle Manager (OLM). The format mainly consists of a YAML manifest referred to as [ClusterServiceVersion](https://github.com/operator-framework/operator-lifecycle-manager/blob/master/Documentation/design/building-your-csv.md) which provides information about the CustomResourceDefinitions the Operator owns or requires, which RBAC definition it needs, where the image is stored, etc. This file is usually accompanied by additional YAML files which define the Operators’ own CRDs. This information is processed by OLM at the time a user requests to install an Operator to provide dependency resolution and automation.

  What does listing of an Operator on OperatorHub.io mean?

  To be listed, Operators must successfully show cluster lifecycle features, be packaged as a CSV to be maintained through OLM, and have acceptable documentation for its intended users.

  Some examples of Operators that are currently listed on OperatorHub.io include: Amazon Web Services Operator, Couchbase Autonomous Operator, CrunchyData’s PostgreSQL, etcd Operator, Jaeger Operator for Kubernetes, Kubernetes Federation Operator, MongoDB Enterprise Operator, Percona MySQL Operator, PlanetScale’s Vitess Operator, Prometheus Operator, and Redis Operator.

  Want to add your Operator to OperatorHub.io? Follow these steps

  If you have an existing Operator, follow the contribution guide using a fork of the community-operators repository. Each contribution contains the CSV, all of the CustomResourceDefinitions, access control rules and references to the container image needed to install and run your Operator, plus other info like a description of its features and supported Kubernetes versions. A complete example, including multiple versions of the Operator, can be found with the EtcdOperator.

  After testing out your Operator on your own cluster, submit a PR to the community repository with all of YAML files following this directory structure. Subsequent versions of the Operator can be published in the same way. At first this will be reviewed manually, but automation is on the way. After it’s merged by the maintainers, it will show up on OperatorHub.io along with its documentation and a convenient installation method.

  Want to learn more?

  • Attend one of the upcoming Kubernetes Operator Framework hands-on workshops at ScaleX in Pasadena on March 7 and at the OpenShift Commons Gathering on Operating at Scale in Santa Clara on March 11
  • Listen to this OpenShift Commons Briefing on “The State of Operators” with Daniel Messer and Diane Mueller
  • Join in on the online conversations in the community Kubernetes-Operator Slack Channel and the Operator Framework Google Group
  • Finally, read up on how to add your Operator to OperatorHub.io: https://operatorhub.io/contribute

  
  
  https://kubernetes.io/blog/2019/02/28/automate-operations-on-your-cluster-with-operatorhub.io/
• 

  Blog: Building a Kubernetes Edge (Ingress) Control Plane for Envoy v2
  • sorangutan  

  1
  1
  Posts
  57
  Views

  

  Author: Daniel Bryant, Product Architect, Datawire; Flynn, Ambassador Lead Developer, Datawire; Richard Li, CEO and Co-founder, Datawire

  Kubernetes has become the de facto runtime for container-based microservice applications, but this orchestration framework alone does not provide all of the infrastructure necessary for running a distributed system. Microservices typically communicate through Layer 7 protocols such as HTTP, gRPC, or WebSockets, and therefore having the ability to make routing decisions, manipulate protocol metadata, and observe at this layer is vital. However, traditional load balancers and edge proxies have predominantly focused on L3/4 traffic. This is where the Envoy Proxy comes into play.

  Envoy proxy was designed as a universal data plane from the ground-up by the Lyft Engineering team for today’s distributed, L7-centric world, with broad support for L7 protocols, a real-time API for managing its configuration, first-class observability, and high performance within a small memory footprint. However, Envoy’s vast feature set and flexibility of operation also makes its configuration highly complicated – this is evident from looking at its rich but verbose control plane syntax.

  With the open source Ambassador API Gateway, we wanted to tackle the challenge of creating a new control plane that focuses on the use case of deploying Envoy as an forward-facing edge proxy within a Kubernetes cluster, in a way that is idiomatic to Kubernetes operators. In this article, we’ll walk through two major iterations of the Ambassador design, and how we integrated Ambassador with Kubernetes.

  Ambassador pre-2019: Envoy v1 APIs, Jinja Template Files, and Hot Restarts

  Ambassador itself is deployed within a container as a Kubernetes service, and uses annotations added to Kubernetes Services as its core configuration model. This approach enables application developers to manage routing as part of the Kubernetes service definition. We explicitly decided to go down this route because of limitations in the current Ingress API spec, and we liked the simplicity of extending Kubernetes services, rather than introducing another custom resource type. An example of an Ambassador annotation can be seen here:

  kind: Service
  apiVersion: v1
  metadata:
  name: my-service
  annotations:
  getambassador.io/config: |
  ---
  apiVersion: ambassador/v0
  kind: Mapping
  name: my_service_mapping
  prefix: /my-service/
  service: my-service
  spec:
  selector:
  app: MyApp
  ports:
  - protocol: TCP
  port: 80
  targetPort: 9376
  

  Translating this simple Ambassador annotation config into valid Envoy v1 config was not a trivial task. By design, Ambassador’s configuration isn’t based on the same conceptual model as Envoy’s configuration – we deliberately wanted to aggregate and simplify operations and config. Therefore, translating between one set of concepts to the other involves a fair amount of logic within Ambassador.

  In this first iteration of Ambassador we created a Python-based service that watched the Kubernetes API for changes to Service objects. When new or updated Ambassador annotations were detected, these were translated from the Ambassador syntax into an intermediate representation (IR) which embodied our core configuration model and concepts. Next, Ambassador translated this IR into a representative Envoy configuration which was saved as a file within pods associated with the running Ambassador k8s Service. Ambassador then “hot-restarted” the Envoy process running within the Ambassador pods, which triggered the loading of the new configuration.

  There were many benefits with this initial implementation. The mechanics involved were fundamentally simple, the transformation of Ambassador config into Envoy config was reliable, and the file-based hot restart integration with Envoy was dependable.

  However, there were also notable challenges with this version of Ambassador. First, although the hot restart was effective for the majority of our customers’ use cases, it was not very fast, and some customers (particularly those with huge application deployments) found it was limiting the frequency with which they could change their configuration. Hot restart can also drop connections, especially long-lived connections like WebSockets or gRPC streams.

  More crucially, though, the first implementation of the IR allowed rapid prototyping but was primitive enough that it proved very difficult to make substantial changes. While this was a pain point from the beginning, it became a critical issue as Envoy shifted to the Envoy v2 API. It was clear that the v2 API would offer Ambassador many benefits – as Matt Klein outlined in his blog post, “The universal data plane API” – including access to new features and a solution to the connection-drop problem noted above, but it was also clear that the existing IR implementation was not capable of making the leap.

  Ambassador >= v0.50: Envoy v2 APIs (ADS), Testing with KAT, and Golang

  In consultation with the Ambassador community, the Datawire team undertook a redesign of the internals of Ambassador in 2018. This was driven by two key goals. First, we wanted to integrate Envoy’s v2 configuration format, which would enable the support of features such as SNI, rate limiting and gRPC authentication APIs. Second, we also wanted to do much more robust semantic validation of Envoy configuration due to its increasing complexity (particularly when operating with large-scale application deployments).

  Initial stages

  We started by restructuring the Ambassador internals more along the lines of a multipass compiler. The class hierarchy was made to more closely mirror the separation of concerns between the Ambassador configuration resources, the IR, and the Envoy configuration resources. Core parts of Ambassador were also redesigned to facilitate contributions from the community outside Datawire. We decided to take this approach for several reasons. First, Envoy Proxy is a very fast moving project, and we realized that we needed an approach where a seemingly minor Envoy configuration change didn’t result in days of reengineering within Ambassador. In addition, we wanted to be able to provide semantic verification of configuration.

  As we started working more closely with Envoy v2, a testing challenge was quickly identified. As more and more features were being supported in Ambassador, more and more bugs appeared in Ambassador’s handling of less common but completely valid combinations of features. This drove to creation of a new testing requirement that meant Ambassador’s test suite needed to be reworked to automatically manage many combinations of features, rather than relying on humans to write each test individually. Moreover, we wanted the test suite to be fast in order to maximize engineering productivity.

  Thus, as part of the Ambassador rearchitecture, we introduced the Kubernetes Acceptance Test (KAT) framework. KAT is an extensible test framework that:

  1. Deploys a bunch of services (along with Ambassador) to a Kubernetes cluster
  2. Run a series of verification queries against the spun up APIs
  3. Perform a bunch of assertions on those query results

  KAT is designed for performance – it batches test setup upfront, and then runs all the queries in step 3 asynchronously with a high performance client. The traffic driver in KAT runs locally using Telepresence, which makes it easier to debug issues.

  Introducing Golang to the Ambassador Stack

  With the KAT test framework in place, we quickly ran into some issues with Envoy v2 configuration and hot restart, which presented the opportunity to switch to use Envoy’s Aggregated Discovery Service (ADS) APIs instead of hot restart. This completely eliminated the requirement for restart on configuration changes, which we found could lead to dropped connection under high loads or long-lived connections.

  However, we faced an interesting question as we considered the move to the ADS. The ADS is not as simple as one might expect: there are explicit ordering dependencies when sending updates to Envoy. The Envoy project has reference implementations of the ordering logic, but only in Go and Java, where Ambassador was primarily in Python. We agonized a bit, and decided that the simplest way forward was to accept the polyglot nature of our world, and do our ADS implementation in Go.

  We also found, with KAT, that our testing had reached the point where Python’s performance with many network connections was a limitation, so we took advantage of Go here, as well, writing KAT’s querying and backend services primarily in Go. After all, what’s another Golang dependency when you’ve already taken the plunge?

  With a new test framework, new IR generating valid Envoy v2 configuration, and the ADS, we thought we were done with the major architectural changes in Ambassador 0.50. Alas, we hit one more issue. On the Azure Kubernetes Service, Ambassador annotation changes were no longer being detected.

  Working with the highly-responsive AKS engineering team, we were able to identify the issue – namely, the Kubernetes API server in AKS is exposed through a chain of proxies, requiring clients to be updating to understand how to connect using the FQDN of the API server, which is provided through a mutating webhook in AKS. Unfortunately, support for this feature was not available in the official Kubernetes Python client, so this was the third spot where we chose to switch to Go instead of Python.

  This raises the interesting question of, “why not ditch all the Python code, and just rewrite Ambassador entirely in Go?” It’s a valid question. The main concern with a rewrite is that Ambassador and Envoy operate at different conceptual levels rather than simply expressing the same concepts with different syntax. Being certain that we’ve expressed the conceptual bridges in a new language is not a trivial challenge, and not something to undertake without already having really excellent test coverage in place

  At this point, we use Go to coverage very specific, well-contained functions that can be verified for correctness much more easily that we could verify a complete Golang rewrite. In the future, who knows? But for 0.50.0, this functional split let us both take advantage of Golang’s strengths, while letting us retain more confidence about all the changes already in 0.50.

  Lessons Learned

  We’ve learned a lot in the process of building Ambassador 0.50. Some of our key takeaways:

  • Kubernetes and Envoy are very powerful frameworks, but they are also extremely fast moving targets – there is sometimes no substitute for reading the source code and talking to the maintainers (who are fortunately all quite accessible!)
  • The best supported libraries in the Kubernetes / Envoy ecosystem are written in Go. While we love Python, we have had to adopt Go so that we’re not forced to maintain too many components ourselves.
  • Redesigning a test harness is sometimes necessary to move your software forward.
  • The real cost in redesigning a test harness is often in porting your old tests to the new harness implementation.
  • Designing (and implementing) an effective control plane for the edge proxy use case has been challenging, and the feedback from the open source community around Kubernetes, Envoy and Ambassador has been extremely useful.

  Migrating Ambassador to the Envoy v2 configuration and ADS APIs was a long and difficult journey that required lots of architecture and design discussions and plenty of coding, but early feedback from results have been positive. Ambassador 0.50 is available now, so you can take it for a test run and share your feedback with the community on our Slack channel or on Twitter.

  
  
  https://kubernetes.io/blog/2019/02/12/building-a-kubernetes-edge-control-plane-for-envoy-v2/
• 

  Blog: Runc and CVE-2019-5736
  • sorangutan  

  1
  1
  Posts
  52
  Views

  

  This morning a container escape vulnerability in runc was announced. We wanted to provide some guidance to Kubernetes users to ensure everyone is safe and secure.

  What Is Runc?

  Very briefly, runc is the low-level tool which does the heavy lifting of spawning a Linux container. Other tools like Docker, Containerd, and CRI-O sit on top of runc to deal with things like data formatting and serialization, but runc is at the heart of all of these systems.

  Kubernetes in turn sits on top of those tools, and so while no part of Kubernetes itself is vulnerable, most Kubernetes installations are using runc under the hood.

  What Is The Vulnerability?

  While full details are still embargoed to give people time to patch, the rough version is that when running a process as root (UID 0) inside a container, that process can exploit a bug in runc to gain root privileges on the host running the container. This then allows them unlimited access to the server as well as any other containers on that server.

  If the process inside the container is either trusted (something you know is not hostile) or is not running as UID 0, then the vulnerability does not apply. It can also be prevented by SELinux, if an appropriate policy has been applied. RedHat Enterprise Linux, CentOS, and Fedora all include appropriate SELinux permissions with their packages and so are believed to be unaffected.

  The most common source of risk is attacker-controller container images, such as unvetted images from public repositories.

  What Should I Do?

  As with all security issues, the two main options are to mitigate the vulnerability or upgrade your version of runc to one that includes the fix.

  As the exploit requires UID 0 within the container, a direct mitigation is to ensure all your containers are running as a non-0 user. This can be set within the container image, or via your pod specification:

  apiVersion: v1
  kind: Pod
  metadata:
   name: run-as-uid-1000
  spec:
   securityContext:
   runAsUser: 1000
   # ...

  This can also be enforced globally using a PodSecurityPolicy:

  apiVersion: policy/v1beta1
  kind: PodSecurityPolicy
  metadata:
   name: non-root
  spec:
   privileged: false
   allowPrivilegeEscalation: false
   runAsUser:
   # Require the container to run without root privileges.
   rule: 'MustRunAsNonRoot'

  Setting a policy like this is highly encouraged given the overall risks of running as UID 0 inside a container.

  Another potential mitigation is to ensure all your container images are vetted and trusted. This can be accomplished by building all your images yourself, or by vetting the contents of an image and then pinning to the image version hash (image: external/someimage@sha256:7832659873hacdef).

  Upgrading runc can generally be accomplished by upgrading the package runc for your distribution or by upgrading your OS image if using immutable images. This is a list of known safe versions for various distributions and platforms:

  • Ubuntu - runc 1.0.0~rc4+dfsg1-6ubuntu0.18.10.1
  • Debian - runc 0.1.1+dfsg1-2
  • RedHat Enterprise Linux - docker 1.13.1-91.git07f3374.el7 (if SELinux is disabled)
  • Amazon Linux - docker 18.06.1ce-7.25.amzn1.x86_64
  • CoreOS - 2051.0.0
  • Kops Debian - in progress
  • Docker - 18.09.2

  Some platforms have also posted more specific instructions:

  Google Container Engine (GKE)

  Google has issued a security bulletin with more detailed information but in short, if you are using the default GKE node image then you are safe. If you are using an Ubuntu or CoreOS node image then you will need to mitigate or upgrade to an image with a fixed version of runc.

  Amazon Elastic Container Service for Kubernetes (EKS)

  Amazon has also issued a security bulletin with more detailed information. All EKS users should mitigate the issue or upgrade to a new node image.

  Docker

  We don’t have specific confirmation that Docker for Mac and Docker for Windows are vulnerable, however it seems likely. Docker has released a fix in version 18.09.2 and it is recommended you upgrade to it. This also applies to other deploy systems using Docker under the hood.

  If you are unable to upgrade Docker, the Rancher team has provided backports of the fix for many older versions at github.com/rancher/runc-cve.

  Getting More Information

  If you have any further questions about how this vulnerability impacts Kubernetes, please join us at discuss.kubernetes.io.

  If you would like to get in contact with the runc team, you can reach them on Google Groups or #opencontainers on Freenode IRC.

  
  
  https://kubernetes.io/blog/2019/02/11/runc-and-cve-2019-5736/
• 

  Blog: Poseidon-Firmament Scheduler – Flow Network Graph Based Scheduler
  • sorangutan  

  1
  1
  Posts
  65
  Views

  

  Authors: Deepak Vij (Huawei), Shivram Shrivastava (Huawei)

  Introduction

  Cluster Management systems such as Mesos, Google Borg, Kubernetes etc. in a cloud scale datacenter environment (also termed as Datacenter-as-a-Computer or Warehouse-Scale Computing - WSC) typically manage application workloads by performing tasks such as tracking machine live-ness, starting, monitoring, terminating workloads and more importantly using a Cluster Scheduler to decide on workload placements.

  A Cluster Scheduler essentially performs the scheduling of workloads to compute resources – combining the global placement of work across the WSC environment makes the “warehouse-scale computer” more efficient, increases utilization, and saves energy. Cluster Scheduler examples are Google Borg, Kubernetes, Firmament, Mesos, Tarcil, Quasar, Quincy, Swarm, YARN, Nomad, Sparrow, Apollo etc.

  In this blog post, we briefly describe the novel Firmament flow network graph based scheduling approach (OSDI paper) in Kubernetes. We specifically describe the Firmament Scheduler and how it integrates with the Kubernetes cluster manager using Poseidon as the integration glue. We have seen extremely impressive scheduling throughput performance benchmarking numbers with this novel scheduling approach. Originally, Firmament Scheduler was conceptualized, designed and implemented by University of Cambridge researchers, Malte Schwarzkopf & Ionel Gog.

  Poseidon-Firmament Scheduler – How It Works

  At a very high level, Poseidon-Firmament scheduler augments the current Kubernetes scheduling capabilities by incorporating novel flow network graph based scheduling capabilities alongside the default Kubernetes Scheduler. It models the scheduling problem as a constraint-based optimization over a flow network graph – by reducing scheduling to a min-cost max-flow optimization problem. Due to the inherent rescheduling capabilities, the new scheduler enables a globally optimal scheduling environment that constantly keeps refining the workloads placements dynamically.

  Key Advantages

  Flow graph scheduling based Poseidon-Firmament scheduler provides the following key advantages:

  • Workloads (pods) are bulk scheduled to enable scheduling decisions at massive scale.

  • Based on the extensive performance test results, Poseidon-Firmament scales much better than Kubernetes default scheduler as the number of nodes increase in a cluster. This is due to the fact that Poseidon-Firmament is able to amortize more and more work across workloads.

  • Poseidon-Firmament Scheduler outperforms the Kubernetes default scheduler by a wide margin when it comes to throughput performance numbers for scenarios where compute resource requirements are somewhat uniform across jobs (Replicasets/Deployments/Jobs). Poseidon-Firmament scheduler end-to-end throughput performance numbers, including bind time, consistently get better as the number of nodes in a cluster increase. For example, for a 2,700 node cluster (shown in the graphs here), Poseidon-Firmament scheduler achieves a 7X or greater end-to-end throughput than the Kubernetes default scheduler, which includes bind time.

  • Availability of complex rule constraints.

  • Scheduling in Poseidon-Firmament is very dynamic; it keeps cluster resources in a global optimal state during every scheduling run.

  • Highly efficient resource utilizations.

  Firmament Flow Network Graph – An Overview

  Firmament scheduler runs a min-cost flow algorithm over the flow network to find an optimal flow, from which it extracts the implied workload (pod placements). A flow network is a directed graph whose arcs carry flow from source nodes (i.e. pod nodes) to a sink node. A cost and capacity associated with each arc constrain the flow, and specify preferential routes for it.

  Figure 1 below shows an example of a flow network for a cluster with two tasks (workloads or pods) and four machines (nodes) – each workload on the left hand side, is a source of one unit of flow. All such flow must be drained into the sink node (S) for a feasible solution to the optimization problem.

  

  Figure 1. Example of a Flow Network

  Poseidon Mediation Layer – An Overview

  Poseidon is a service that acts as the integration glue for the Firmament scheduler with Kubernetes. It augments the current Kubernetes scheduling capabilities by incorporating new flow network graph based Firmament scheduling capabilities alongside the default Kubernetes Scheduler; multiple schedulers running simultaneously. Figure 2 below describes the high level overall design as far as how Poseidon integration glue works in conjunction with the underlying Firmament flow network graph based scheduler.

  

  Figure 2. Firmament Kubernetes Integration Overview

  As part of the Kubernetes multiple schedulers support, each new pod is typically scheduled by the default scheduler, but Kubernetes can be instructed to use another scheduler by specifying the name of another custom scheduler (in our case, Poseidon-Firmament) at the time of pod deployment. In this case, the default scheduler will ignore that Pod and allow Poseidon scheduler to schedule the Pod to a relevant node.

  Note: For details about the design of this project see the design document.

  Possible Use Case Scenarios – When To Use It

  Poseidon-Firmament scheduler enables extremely high throughput scheduling environment at scale due to its bulk scheduling approach superiority versus K8s pod-at-a-time approach. In our extensive tests, we have observed substantial throughput benefits as long as resource requirements (CPU/Memory) for incoming Pods is uniform across jobs (Replicasets/Deployments/Jobs), mainly due to efficient amortization of work across jobs.

  Although, Poseidon-Firmament scheduler is capable of scheduling various types of workloads (service, batch, etc.), following are the few use cases where it excels the most:

  1. For “Big Data/AI” jobs consisting of a large number of tasks, throughput benefits are tremendous.

  2. Substantial throughput benefits also for service or batch job scenarios where workload resource requirements are uniform across jobs (Replicasets/Deplyments/Jobs).

  Current Project Stage

  Currently Poseidon-Firmament project is an incubation project. Alpha Release is available at https://github.com/kubernetes-sigs/poseidon.

  
  
  https://kubernetes.io/blog/2019/02/06/poseidon-firmament-scheduler-flow-network-graph-based-scheduler/
• 

  Blog: Update on Volume Snapshot Alpha for Kubernetes
  • sorangutan  

  1
  1
  Posts
  57
  Views

  

  Authors: Jing Xu (Google), Xing Yang (Huawei), Saad Ali (Google)

  Volume snapshotting support was introduced in Kubernetes v1.12 as an alpha feature. In Kubernetes v1.13, it remains an alpha feature, but a few enhancements were added and some breaking changes were made. This post summarizes the changes.

  Breaking Changes

  CSI spec v1.0 introduced a few breaking changes to the volume snapshot feature. CSI driver maintainers should be aware of these changes as they upgrade their drivers to support v1.0.

  SnapshotStatus replaced with Boolean ReadyToUse

  CSI v0.3.0, defined a SnapshotStatus enum in CreateSnapshotResponse which indicates whether the snapshot is READY, UPLOADING, or ERROR_UPLOADING. In CSI v1.0, SnapshotStatus has been removed from CreateSnapshotResponse and replaced with a boolean ReadyToUse. A ReadyToUse value of true indicates that post snapshot processing (such as uploading) is complete and the snapshot is ready to be used as a source to create a volume.

  Storage systems that need to do post snapshot processing (such as uploading after the snapshot is cut) should return a successful CreateSnapshotResponse with the ReadyToUse field set to false as soon as the snapshot has been taken. This indicates that the Container Orchestration System (CO) can resume any workload that was quiesced for the snapshot to be taken. The CO can then repeatedly call CreateSnapshot until the ReadyToUse field is set to true or the call returns an error indicating a problem in processing. The CSI ListSnapshot call could be used along with snapshot_id filtering to determine if the snapshot is ready to use, but is not recommended because it provides no way to detect errors during processing (the ReadyToUse field simply remains false indefinitely).

  The v1.x.x releases of the CSI external-snapshotter sidecar container already handle this change by calling CreateSnapshot instead of ListSnapshots to check if a snapshot is ready to use. When upgrading their drivers to CSI 1.0, driver maintainers should use the appropriate 1.0 compatible sidecar container.

  To be consistent with the change in the CSI spec, the Ready field in the VolumeSnapshot API object has been renamed to ReadyToUse. This change is visible to the user when running kubectl describe volumesnapshot to view the details of a snapshot.

  Timestamp Data Type

  The creation time of a snapshot is available to Kubernetes admins as part of the VolumeSnapshotContent API object. This field is populated using the creation_time field in the CSI CreateSnapshotResponse. In CSI v1.0, this creation_time field type was changed to .google.protobuf.Timestamp instead of int64. When upgrading drivers to CSI 1.0, driver maintainers must make changes accordingly. The v1.x.x releases of the CSI external-snapshotter sidecar container has been updated to handle this change.

  Deprecations

  The following VolumeSnapshotClass parameters are deprecated and will be removed in a future release. They will be replaced with parameters listed in the Replacement section below.

  Deprecated Replacement csiSnapshotterSecretName csi.storage.k8s.io/snapshotter-secret-name csiSnapshotterSecretNameSpace csi.storage.k8s.io/snapshotter-secret-namespace

  New Features

  SnapshotContent Deletion/Retain Policy

  As described in the initial blog post announcing the snapshot alpha, the Kubernetes snapshot APIs are similar to the PV/PVC APIs: just like a volume is represented by a bound PVC and PV pair, a snapshot is represented by a bound VolumeSnapshot and VolumeSnapshotContent pair.

  With PV/PVC pairs, when a user is done with a volume, they can delete the PVC. And the reclaim policy on the PV determines what happens to the PV (whether it is also deleted or retained).

  In the initial alpha release, snapshots did not support the ability to specify a reclaim policy. Instead when a snapshot object was deleted it always resulted in the snapshot being deleted. In Kubernetes v1.13, a snapshot content DeletionPolicy was added. It enables an admin to configure what what happens to a VolumeSnapshotContent after the VolumeSnapshot object it is bound to is deleted. The DeletionPolicy of a volume snapshot can either be Retain or Delete. If the value is not specified, the default depends on whether the SnapshotContent object was created via static binding or dynamic provisioning.

  Retain

  The Retain policy allows for manual reclamation of the resource. If a VolumeSnapshotContent is statically created and bound, the default DeletionPolicy is Retain. When the VolumeSnapshot is deleted, the VolumeSnapshotContent continues to exist and the VolumeSnapshotContent is considered “released”. But it is not available for binding to other VolumeSnapshot objects because it contains data. It is up to an administrator to decide how to handle the remaining API object and resource cleanup.

  Delete

  A Delete policy enables automatic deletion of the bound VolumeSnapshotContent object from Kubernetes and the associated storage asset in the external infrastructure (such as an AWS EBS snapshot or GCE PD snapshot, etc.). Snapshots that are dynamically provisioned inherit the deletion policy of their VolumeSnapshotClass, which defaults to Delete. The administrator should configure the VolumeSnapshotClass with the desired retention policy. The policy may be changed for individual VolumeSnapshotContent after it is created by patching the object.

  The following example demonstrates how to check the deletion policy of a dynamically provisioned VolumeSnapshotContent.

  $ kubectl create -f ./examples/kubernetes/demo-defaultsnapshotclass.yaml
  $ kubectl create -f ./examples/kubernetes/demo-snapshot.yaml
  $ kubectl get volumesnapshots demo-snapshot-podpvc -o yaml
  apiVersion: snapshot.storage.k8s.io/v1alpha1
  kind: VolumeSnapshot
  metadata:
  creationTimestamp: "2018-11-27T23:57:09Z"
  ...
  spec:
  snapshotClassName: default-snapshot-class
  snapshotContentName: snapcontent-26cd0db3-f2a0-11e8-8be6-42010a800002
  source:
  apiGroup: null
  kind: PersistentVolumeClaim
  name: podpvc
  status:
  …
  $ kubectl get volumesnapshotcontent snapcontent-26cd0db3-f2a0-11e8-8be6-42010a800002 -o yaml
  apiVersion: snapshot.storage.k8s.io/v1alpha1
  kind: VolumeSnapshotContent
  …
  spec:
  csiVolumeSnapshotSource:
  creationTime: 1546469777852000000
  driver: pd.csi.storage.gke.io
  restoreSize: 6442450944
  snapshotHandle: projects/jing-k8s-dev/global/snapshots/snapshot-26cd0db3-f2a0-11e8-8be6-42010a800002
  deletionPolicy: Delete
  persistentVolumeRef:
  apiVersion: v1
  kind: PersistentVolume
  name: pvc-853622a4-f28b-11e8-8be6-42010a800002
  resourceVersion: "21117"
  uid: ae400e9f-f28b-11e8-8be6-42010a800002
  snapshotClassName: default-snapshot-class
  volumeSnapshotRef:
  apiVersion: snapshot.storage.k8s.io/v1alpha1
  kind: VolumeSnapshot
  name: demo-snapshot-podpvc
  namespace: default
  resourceVersion: "6948065"
  uid: 26cd0db3-f2a0-11e8-8be6-42010a800002
  

  User can change the deletion policy by using patch:

  $ kubectl patch volumesnapshotcontent snapcontent-26cd0db3-f2a0-11e8-8be6-42010a800002 -p '{"spec":{"deletionPolicy":"Retain"}}' --type=merge
  $ kubectl get volumesnapshotcontent snapcontent-26cd0db3-f2a0-11e8-8be6-42010a800002 -o yaml
  apiVersion: snapshot.storage.k8s.io/v1alpha1
  kind: VolumeSnapshotContent
  ...
  spec:
  csiVolumeSnapshotSource:
  ...
  deletionPolicy: Retain
  persistentVolumeRef:
  apiVersion: v1
  kind: PersistentVolume
  name: pvc-853622a4-f28b-11e8-8be6-42010a800002
  ...
  

  Snapshot Object in Use Protection

  The purpose of the Snapshot Object in Use Protection feature is to ensure that in-use snapshot API objects are not removed from the system (as this may result in data loss). There are two cases that require “in-use” protection:

  1. If a volume snapshot is in active use by a persistent volume claim as a source to create a volume.
  2. If a VolumeSnapshotContent API object is bound to a VolumeSnapshot API object, the content object is considered in use.

  If a user deletes a VolumeSnapshot API object in active use by a PVC, the VolumeSnapshot object is not removed immediately. Instead, removal of the VolumeSnapshot object is postponed until the VolumeSnapshot is no longer actively used by any PVCs. Similarly, if an admin deletes a VolumeSnapshotContent that is bound to a VolumeSnapshot, the VolumeSnapshotContent is not removed immediately. Instead, the VolumeSnapshotContent removal is postponed until the VolumeSnapshotContent is not bound to the VolumeSnapshot object.

  Which volume plugins support Kubernetes Snapshots?

  Snapshots are only supported for CSI drivers (not for in-tree or Flexvolume). To use the Kubernetes snapshots feature, ensure that a CSI Driver that implements snapshots is deployed on your cluster.

  As of the publishing of this blog post, the following CSI drivers support snapshots:

  • GCE Persistent Disk CSI Driver
  • OpenSDS CSI Driver
  • Ceph RBD CSI Driver
  • Portworx CSI Driver
  • GlusterFS CSI Driver
  • Digital Ocean CSI Driver
  • Ember CSI Driver
  • Cinder CSI Driver
  • Datera CSI Driver
  • NexentaStor CSI Driver

  Snapshot support for other drivers is pending, and should be available soon. Read the “Container Storage Interface (CSI) for Kubernetes GA” blog post to learn more about CSI and how to deploy CSI drivers.

  What’s next?

  Depending on feedback and adoption, the Kubernetes team plans to push the CSI Snapshot implementation to beta in either 1.15 or 1.16. Some of the features we are interested in supporting include consistency groups, application consistent snapshots, workload quiescing, in-place restores, and more.

  How can I learn more?

  The code repository for snapshot APIs and controller is here: https://github.com/kubernetes-csi/external-snapshotter

  Check out additional documentation on the snapshot feature here: http://k8s.io/docs/concepts/storage/volume-snapshots and https://kubernetes-csi.github.io/docs/

  How do I get involved?

  This project, like all of Kubernetes, is the result of hard work by many contributors from diverse backgrounds working together.

  Special thanks to all the contributors that helped add CSI v1.0 support and improve the snapshot feature in this release, including Saad Ali (saadali), Michelle Au (msau42), Deep Debroy (ddebroy), James DeFelice (jdef), John Griffith (j-griffith), Julian Hjortshoj (julian-hj), Tim Hockin (thockin), Patrick Ohly (pohly), Luis Pabon (lpabon), Cheng Xing (verult), Jing Xu (jingxu97), Shiwei Xu (wackxu), Xing Yang (xing-yang), Jie Yu (jieyu), David Zhu (davidz627).

  Those interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the Kubernetes Storage Special Interest Group (SIG). We’re rapidly growing and always welcome new contributors.

  We also hold regular SIG-Storage Snapshot Working Group meetings. New attendees are welcome to join for design and development discussions.

  
  
  https://kubernetes.io/blog/2019/01/17/update-on-volume-snapshot-alpha-for-kubernetes/
• 

  Blog: Container Storage Interface (CSI) for Kubernetes GA
  • sorangutan  

  1
  1
  Posts
  58
  Views

  

  

  Author: Saad Ali, Senior Software Engineer, Google

  The Kubernetes implementation of the Container Storage Interface (CSI) has been promoted to GA in the Kubernetes v1.13 release. Support for CSI was introduced as alpha in Kubernetes v1.9 release, and promoted to beta in the Kubernetes v1.10 release.

  The GA milestone indicates that Kubernetes users may depend on the feature and its API without fear of backwards incompatible changes in future causing regressions. GA features are protected by the Kubernetes deprecation policy.

  Why CSI?

  Although prior to CSI Kubernetes provided a powerful volume plugin system, it was challenging to add support for new volume plugins to Kubernetes: volume plugins were “in-tree” meaning their code was part of the core Kubernetes code and shipped with the core Kubernetes binaries—vendors wanting to add support for their storage system to Kubernetes (or even fix a bug in an existing volume plugin) were forced to align with the Kubernetes release process. In addition, third-party storage code caused reliability and security issues in core Kubernetes binaries and the code was often difficult (and in some cases impossible) for Kubernetes maintainers to test and maintain.

  CSI was developed as a standard for exposing arbitrary block and file storage storage systems to containerized workloads on Container Orchestration Systems (COs) like Kubernetes. With the adoption of the Container Storage Interface, the Kubernetes volume layer becomes truly extensible. Using CSI, third-party storage providers can write and deploy plugins exposing new storage systems in Kubernetes without ever having to touch the core Kubernetes code. This gives Kubernetes users more options for storage and makes the system more secure and reliable.

  What’s new?

  With the promotion to GA, the Kubernetes implementation of CSI introduces the following changes:

  • Kubernetes is now compatible with CSI spec v1.0 and v0.3 (instead of CSI spec v0.2).
    • There were breaking changes between CSI spec v0.3.0 and v1.0.0, but Kubernetes v1.13 supports both versions so either version will work with Kubernetes v1.13.
    • Please note that with the release of the CSI 1.0 API, support for CSI drivers using 0.3 and older releases of the CSI API is deprecated, and is planned to be removed in Kubernetes v1.15.
    • There were no breaking changes between CSI spec v0.2 and v0.3, so v0.2 drivers should also work with Kubernetes v1.10.0+.
    • There were breaking changes between the CSI spec v0.1 and v0.2, so very old drivers implementing CSI 0.1 must be updated to be at least 0.2 compatible before use with Kubernetes v1.10.0+.
  • The Kubernetes VolumeAttachment object (introduced in v1.9 in the storage v1alpha1 group, and added to the v1beta1 group in v1.10) has been added to the storage v1 group in v1.13.
  • The Kubernetes CSIPersistentVolumeSource volume type has been promoted to GA.
  • The Kubelet device plugin registration mechanism, which is the means by which kubelet discovers new CSI drivers, has been promoted to GA in Kubernetes v1.13.

  How to deploy a CSI driver?

  Kubernetes users interested in how to deploy or manage an existing CSI driver on Kubernetes should look at the documentation provided by the author of the CSI driver.

  How to use a CSI volume?

  Assuming a CSI storage plugin is already deployed on a Kubernetes cluster, users can use CSI volumes through the familiar Kubernetes storage API objects: PersistentVolumeClaims, PersistentVolumes, and StorageClasses. Documented here.

  Although the Kubernetes implementation of CSI is a GA feature in Kubernetes v1.13, it may require the following flag:

  • API server binary and kubelet binaries:
    • --allow-privileged=true
    • Most CSI plugins will require bidirectional mount propagation, which can only be enabled for privileged pods. Privileged pods are only permitted on clusters where this flag has been set to true (this is the default in some environments like GCE, GKE, and kubeadm).

  Dynamic Provisioning

  You can enable automatic creation/deletion of volumes for CSI Storage plugins that support dynamic provisioning by creating a StorageClass pointing to the CSI plugin.

  The following StorageClass, for example, enables dynamic creation of “fast-storage” volumes by a CSI volume plugin called “csi-driver.example.com”.

  kind: StorageClass
  apiVersion: storage.k8s.io/v1
  metadata:
  name: fast-storage
  provisioner: csi-driver.example.com
  parameters:
  type: pd-ssd
  csi.storage.k8s.io/provisioner-secret-name: mysecret
  csi.storage.k8s.io/provisioner-secret-namespace: mynamespace
  

  New for GA, the CSI external-provisioner (v1.0.1+) reserves the parameter keys prefixed with csi.storage.k8s.io/. If the keys do not correspond to a set of known keys the values are simply ignored (and not passed to the CSI driver). The older secret parameter keys (csiProvisionerSecretName, csiProvisionerSecretNamespace, etc.) are also supported by CSI external-provisioner v1.0.1 but are deprecated and may be removed in future releases of the CSI external-provisioner.

  Dynamic provisioning is triggered by the creation of a PersistentVolumeClaim object. The following PersistentVolumeClaim, for example, triggers dynamic provisioning using the StorageClass above.

  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
  name: my-request-for-storage
  spec:
  accessModes:
  - ReadWriteOnce
  resources:
  requests:
  storage: 5Gi
  storageClassName: fast-storage
  

  When volume provisioning is invoked, the parameter type: pd-ssd and the secret any referenced secret(s) are passed to the CSI plugin csi-driver.example.com via a CreateVolume call. In response, the external volume plugin provisions a new volume and then automatically create a PersistentVolume object to represent the new volume. Kubernetes then binds the new PersistentVolume object to the PersistentVolumeClaim, making it ready to use.

  If the fast-storage StorageClass is marked as “default”, there is no need to include the storageClassName in the PersistentVolumeClaim, it will be used by default.

  Pre-Provisioned Volumes

  You can always expose a pre-existing volume in Kubernetes by manually creating a PersistentVolume object to represent the existing volume. The following PersistentVolume, for example, exposes a volume with the name “existingVolumeName” belonging to a CSI storage plugin called “csi-driver.example.com”.

  apiVersion: v1
  kind: PersistentVolume
  metadata:
  name: my-manually-created-pv
  spec:
  capacity:
  storage: 5Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  csi:
  driver: csi-driver.example.com
  volumeHandle: existingVolumeName
  readOnly: false
  fsType: ext4
  volumeAttributes:
  foo: bar
  controllerPublishSecretRef:
  name: mysecret1
  namespace: mynamespace
  nodeStageSecretRef:
  name: mysecret2
  namespace: mynamespace
  nodePublishSecretRef
  name: mysecret3
  namespace: mynamespace
  

  Attaching and Mounting

  You can reference a PersistentVolumeClaim that is bound to a CSI volume in any pod or pod template.

  kind: Pod
  apiVersion: v1
  metadata:
  name: my-pod
  spec:
  containers:
  - name: my-frontend
  image: nginx
  volumeMounts:
  - mountPath: "/var/www/html"
  name: my-csi-volume
  volumes:
  - name: my-csi-volume
  persistentVolumeClaim:
  claimName: my-request-for-storage
  

  When the pod referencing a CSI volume is scheduled, Kubernetes will trigger the appropriate operations against the external CSI plugin (ControllerPublishVolume, NodeStageVolume, NodePublishVolume, etc.) to ensure the specified volume is attached, mounted, and ready to use by the containers in the pod.

  For more details please see the CSI implementation design doc and documentation.

  How to write a CSI Driver?

  The kubernetes-csi site details how to develop, deploy, and test a CSI driver on Kubernetes. In general, CSI Drivers should be deployed on Kubernetes along with the following sidecar (helper) containers:

  • external-attacher
    • Watches Kubernetes VolumeAttachment objects and triggers ControllerPublish and ControllerUnpublish operations against a CSI endpoint.
  • external-provisioner
    • Watches Kubernetes PersistentVolumeClaim objects and triggers CreateVolume and DeleteVolume operations against a CSI endpoint.
  • node-driver-registrar
    • Registers the CSI driver with kubelet using the Kubelet device plugin mechanism.
  • cluster-driver-registrar (Alpha)
    • Registers a CSI Driver with the Kubernetes cluster by creating a CSIDriver object which enables the driver to customize how Kubernetes interacts with it.
  • external-snapshotter (Alpha)
    • Watches Kubernetes VolumeSnapshot CRD objects and triggers CreateSnapshot and DeleteSnapshot operations against a CSI endpoint.
  • livenessprobe
    • May be included in a CSI plugin pod to enable the Kubernetes Liveness Probe mechanism.

  Storage vendors can build Kubernetes deployments for their plugins using these components, while leaving their CSI driver completely unaware of Kubernetes.

  List of CSI Drivers

  CSI drivers are developed and maintained by third parties. You can find a non-definitive list of CSI drivers here.

  What about in-tree volume plugins?

  There is a plan to migrate most of the persistent, remote in-tree volume plugins to CSI. For more details see design doc.

  Limitations of GA

  The GA implementation of CSI has the following limitations:

  • Ephemeral local volumes must create a PVC (pod inline referencing of CSI volumes is not supported).

  What’s next?

  • Work on moving Kubernetes CSI features that are still alpha to beta:
    • Raw block volumes
    • Topology awareness (the ability for Kubernetes to understand and influence where a CSI volume is provisioned (zone, regions, etc.).
    • Features depending on CSI CRDs (e.g. “Skip attach” and “Pod info on mount”).
    • Volume Snapshots
  • Work on completing support for local ephemeral volumes.
  • Work on migrating remote persistent in-tree volume plugins to CSI.

  How to get involved?

  The Kubernetes Slack channel wg-csi and the Google group kubernetes-sig-storage-wg-csi along with any of the standard SIG storage communication channels are all great mediums to reach out to the SIG Storage team.

  This project, like all of Kubernetes, is the result of hard work by many contributors from diverse backgrounds working together. We offer a huge thank you to the new contributors who stepped up this quarter to help the project reach GA:

  • Saad Ali (saad-ali)
  • Michelle Au (msau42)
  • Serguei Bezverkhi (sbezverk)
  • Masaki Kimura (mkimuram)
  • Patrick Ohly (pohly)
  • Luis Pabón (lpabon)
  • Jan Šafránek (jsafrane)
  • Vladimir Vivien (vladimirvivien)
  • Cheng Xing (verult)
  • Xing Yang (xing-yang)
  • David Zhu (davidz627)

  If you’re interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the Kubernetes Storage Special Interest Group (SIG). We’re rapidly growing and always welcome new contributors.

  
  
  https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/
• 

  Blog: APIServer dry-run and kubectl diff
  • sorangutan  

  1
  1
  Posts
  66
  Views

  

  Author: Antoine Pelisse (Google Cloud, @apelisse)

  Declarative configuration management, also known as configuration-as-code, is one of the key strengths of Kubernetes. It allows users to commit the desired state of the cluster, and to keep track of the different versions, improve auditing and automation through CI/CD pipelines. The Apply working-group is working on fixing some of the gaps, and is happy to announce that Kubernetes 1.13 promoted server-side dry-run and kubectl diff to beta. These two features are big improvements for the Kubernetes declarative model.

  Challenges

  A few pieces are still missing in order to have a seamless declarative experience with Kubernetes, and we tried to address some of these:

  • While compilers and linters do a good job to detect errors in pull-requests for code, a good validation is missing for Kubernetes configuration files. The existing solution is to run kubectl apply --dry-run, but this runs a local dry-run that doesn’t talk to the server: it doesn’t have server validation and doesn’t go through validating admission controllers. As an example, Custom resource names are only validated on the server so a local dry-run won’t help.
  • It can be difficult to know how your object is going to be applied by the server for multiple reasons:
    • Defaulting will set some fields to potentially unexpected values,
    • Mutating webhooks might set fields or clobber/change some values.
    • Patch and merges can have surprising effects and result in unexpected objects. For example, it can be hard to know how lists are going to be ordered once merged.

  The working group has tried to address these problems.

  APIServer dry-run

  APIServer dry-run was implemented to address these two problems:

  • it allows individual requests to the apiserver to be marked as “dry-run”,
  • the apiserver guarantees that dry-run requests won’t be persisted to storage,
  • the request is still processed as typical request: the fields are defaulted, the object is validated, it goes through the validation admission chain, and through the mutating admission chain, and then the final object is returned to the user as it normally would, without being persisted.

  While dynamic admission controllers are not supposed to have side-effects on each request, dry-run requests are only processed if all admission controllers explicitly announce that they don’t have any dry-run side-effects.

  How to enable it

  Server-side dry-run is enabled through a feature-gate. Now that the feature is Beta in 1.13, it should be enabled by default, but still can be enabled/disabled using kube-apiserver --feature-gates DryRun=true.

  If you have dynamic admission controllers, you might have to fix them to:

  • Remove any side-effects when the dry-run parameter is specified on the webhook request,
  • Specify in the sideEffects field of the admissionregistration.k8s.io/v1beta1.Webhook object to indicate that the object doesn’t have side-effects on dry-run (or at all).

  How to use it

  You can trigger the feature from kubectl by using kubectl apply --server-dry-run, which will decorate the request with the dryRun flag and return the object as it would have been applied, or an error if it would have failed.

  Kubectl diff

  APIServer dry-run is convenient because it lets you see how the object would be processed, but it can be hard to identify exactly what changed if the object is big. kubectl diff does exactly what you want by showing the differences between the current “live” object and the new “dry-run” object. It makes it very convenient to focus on only the changes that are made to the object, how the server has merged these and how the mutating webhooks affects the output.

  How to use it

  kubectl diff is meant to be as similar as possible to kubectl apply: kubectl diff -f some-resources.yaml will show a diff for the resources in the yaml file. One can even use the diff program of their choice by using the KUBECTL_EXTERNAL_DIFF environment variable, for example:

  KUBECTL_EXTERNAL_DIFF=meld kubectl diff -f some-resources.yaml
  

  What’s next

  The working group is still busy trying to improve some of these things:

  • Server-side apply is trying to improve the apply scenario, by adding owner semantics to fields! It’s also going to improve support for CRDs and unions!
  • Some kubectl apply features are missing from diff and could be useful, like the ability to filter by label, or to display pruned resources.
  • Eventually, kubectl diff will use server-side apply!

  
  
  https://kubernetes.io/blog/2019/01/14/apiserver-dry-run-and-kubectl-diff/
• 

  Blog: Kubernetes Federation Evolution
  • sorangutan  

  1
  1
  Posts
  111
  Views

  

  Authors: Irfan Ur Rehman (Huawei), Paul Morie (RedHat) and Shashidhara T D (Huawei)

  Deploying applications to a kubernetes cluster is well defined and can in some cases be as simple as kubectl create -f app.yaml. The user’s story to deploy apps across multiple clusters has not been that simple. How should an app workload be distributed? Should the app resources be replicated into all clusters, or replicated into selected clusters or partitioned into clusters? How is the access to clusters managed? What happens if some of the resources, which user wants to distribute pre-exist in all or fewer clusters in some form.

  In SIG multicluster, our journey has revealed that there are multiple possible models to solve these problems and there probably is no single best fit all scenario solution. Federation however is the single biggest kubernetes open source sub project which has seen maximum interest and contribution from the community in this problem space. The project initially reused the k8s API to do away with any added usage complexity for an existing k8s user. This became non-viable because of problems best discussed in this community update.

  What has evolved further is a federation specific API architecture and a community effort which now continues as Federation V2.

  Conceptual Overview

  Because federation attempts to address a complex set of problems, it pays to break the different parts of those problems down. Let’s take a look at the different high-level areas involved:

  Kubernetes Federation V2 Concepts

  Federating arbitrary resources

  One of the main goals of Federation is to be able to define the APIs and API groups which encompass basic tenets needed to federate any given k8s resource. This is crucial due to the popularity of Custom Resource Definitions as a way to extend Kubernetes with new APIs.

  The workgroup did arrive at a common definition of the federation API and API groups as ‘a mechanism that distributes “normal” Kubernetes API resources into different clusters’. The distribution in its most simple form could be imagined as simple propagation of this ‘normal Kubernetes API resource’ across the federated clusters. A thoughtful reader can certainly discern more complicated mechanisms, other than this simple propagation of the Kubernetes resources.

  During the journey of defining building blocks of the federation APIs, one of the near term goals also evolved as ‘to be able to create a simple federation aka simple propagation of any Kubernetes resource or a CRD, writing almost zero code’. What ensued further was a core API group defining the building blocks as a Template resource, a Placement resource and an Override resource per given Kubernetes resource, a TypeConfig to specify sync or no sync for the given resource and associated controller(s) to carry out the sync. More details follow in the next section Federating resources: the details. Further sections will also talks about being able to follow a layered behaviour with higher level Federation APIs consuming the behaviour of these core building blocks, and users being able to consume whole or part of the API and associated controllers. Lastly this architecture also allows the users to write additional controllers or replace the available reference controllers with their own to carry out desired behaviour.

  The ability to ‘easily federate arbitrary Kubernetes resources’, and a decoupled API, divided into building blocks APIs, higher level APIs and possible user intended types, presented such that different users can consume parts and write controllers composing solutions specific to them, makes a compelling case for Federation V2.

  Federating resources: the details

  Fundamentally, federation must be configured with two types of information: Which API types federation should handle Which clusters federation should target for distributing those resources. For each API type that federation handles, different parts of the declared state live in different API resources: A template type holds the base specification of the resource - for example, a type called FederatedReplicaSet holds the base specification of a ReplicaSet that should be distributed to the targeted clusters A placement type holds the specification of the clusters the resource should be distributed to - for example, a type called FederatedReplicaSetPlacement holds information about which clusters FederatedReplicaSets should be distributed to An optional overrides type holds the specification of how the template resource should be varied in some clusters - for example, a type called FederatedReplicaSetOverrides holds information about how a FederatedReplicaSet should be varied in certain clusters. These types are all associated by name - meaning that for a particular template resource with name foo, the placement and override information for that resource are contained by the override and placement resources with the same name and namespace as that of the template.

  Higher level behaviour

  The architecture of federation v2 API allows higher level APIs to be constructed using the mechanics provided by the core API types (template, placement and override) and associated controllers for a given resource. In the community we could uncover few use cases and did implement the higher level APIs and associated controllers useful for those cases. Some of these types described in further sections also provide an useful reference to anybody interested in solving more complex use cases, building on top of the mechanics already available with federation v2 API.

  ReplicaSchedulingPreference

  ReplicaSchedulingPreference provides an automated mechanism of distributing and maintaining total number of replicas for deployment or replicaset based federated workloads into federated clusters. This is based on high level user preferences given by the user. These preferences include the semantics of weighted distribution and limits (min and max) for distributing the replicas. These also include semantics to allow redistribution of replicas dynamically in case some replica pods remain unscheduled in some clusters, for example due to insufficient resources in that cluster. More details can be found at the user guide for ReplicaSchedulingPreferences.

  Federated Services & Cross-cluster service discovery

  kubernetes services are very useful construct in micro-service architecture. There is a clear desire to deploy these services across cluster, zone, region and cloud boundaries. Services that span clusters provide geographic distribution, enable hybrid and multi-cloud scenarios and improve the level of high availability beyond single cluster deployments. Customers who want their services to span one or more (possibly remote) clusters, need them to be reachable in a consistent manner from both within and outside their clusters.

  Federated Service at its core contains a template (definition of a kubernetes service), a placement(which clusters to be deployed into), an override (optional variation in particular clusters) and a ServiceDNSRecord (specifying details on how to discover it).

  Note: The Federated Service has to be of type LoadBalancer in order for it to be discoverable across clusters.

  Discovering a Federated Service from pods inside your Federated Clusters

  By default, Kubernetes clusters come preconfigured with a cluster-local DNS server, as well as an intelligently constructed DNS search path which together ensure that DNS queries like myservice, myservice.mynamespace, some-other-service.other-namespace, etc issued by your software running inside Pods are automatically expanded and resolved correctly to the appropriate service IP of services running in the local cluster.

  With the introduction of Federated Services and Cross-Cluster Service Discovery, this concept is extended to cover Kubernetes services running in any other cluster across your Cluster Federation, globally. To take advantage of this extended range, you use a slightly different DNS name (e.g. myservice.mynamespace.myfederation) to resolve federated services. Using a different DNS name also avoids having your existing applications accidentally traversing cross-zone or cross-region networks and you incurring perhaps unwanted network charges or latency, without you explicitly opting in to this behavior.

  Lets consider an example: (The example uses a service named nginx and the query name for described above)

  A Pod in a cluster in the us-central1-a availability zone needs to contact our nginx service. Rather than use the service’s traditional cluster-local DNS name (nginx.mynamespace, which is automatically expanded to nginx.mynamespace.svc.cluster.local) it can now use the service’s Federated DNS name, which is nginx.mynamespace.myfederation. This will be automatically expanded and resolved to the closest healthy shard of my nginx service, wherever in the world that may be. If a healthy shard exists in the local cluster, that service’s cluster-local IP address will be returned (by the cluster-local DNS). This is exactly equivalent to non-federated service resolution.

  If the service does not exist in the local cluster (or it exists but has no healthy backend pods), the DNS query is automatically expanded to nginx.mynamespace.myfederation.svc.us-central1-a.us-central1.example.com. Behind the scenes, this is finding the external IP of one of the shards closest to my availability zone. This expansion is performed automatically by cluster-local DNS server, which returns the associated CNAME record. This results in a traversal of the hierarchy of DNS records, and ends up at one of the external IP’s of the Federated Service near by.

  It is also possible to target service shards in availability zones and regions other than the ones local to a Pod by specifying the appropriate DNS names explicitly, and not relying on automatic DNS expansion. For example, nginx.mynamespace.myfederation.svc.europe-west1.example.comwill resolve to all of the currently healthy service shards in Europe, even if the Pod issuing the lookup is located in the U.S., and irrespective of whether or not there are healthy shards of the service in the U.S. This is useful for remote monitoring and other similar applications.

  Discovering a Federated Service from Other Clients Outside your Federated Clusters

  For external clients, automatic DNS expansion described is currently not possible. External clients need to specify one of the fully qualified DNS names of the federated service, be that a zonal, regional or global name. For convenience reasons, it is often a good idea to manually configure additional static CNAME records in your service, for example:

  SHORT NAME	CNAME
  eu.nginx.acme.com	nginx.mynamespace.myfederation.svc.europe-west1.example.com
  us.nginx.acme.com	nginx.mynamespace.myfederation.svc.us-central1.example.com
  nginx.acme.com	nginx.mynamespace.myfederation.svc.example.com

  That way your clients can always use the short form on the left, and always be automatically routed to the closest healthy shard on their home continent. All of the required failover is handled for you automatically by Kubernetes Cluster Federation.

  As a further reading a more elaborate guide for users is available at Multi-Cluster Service DNS with ExternalDNS Guide

  Try it yourself

  To get started with Federation V2, please refer to the user guide hosted on github.
  Deployment can be accomplished with a helm chart, and once the control plane is available, the user guide’s example can be used to get some hands-on experience with using Federation V2.

  Federation V2 can be deployed in both cluster-scoped and namespace-scoped configurations. A cluster-scoped deployment will require cluster-admin privileges to both host and member clusters, and may be a good fit for evaluating federation on clusters that are not running critical workloads. Namespace-scoped deployment requires access to only a single namespace on host and member clusters, and is a better fit for evaluating federation on clusters running workloads. Most of the user guide refers to cluster-scoped deployment, with the Namespaced Federation section documenting how use of a namespaced deployment differs. Infact same cluster can host multiple federations and/or same clusters can be part of multiple federations in case of Namespaced Federation.

  Next Steps

  As we noted in the beginning of this post, the multicluster problem space is extremely broad. It can be difficult to know exactly how to handle broad problem spaces without concrete pieces of software to frame those conversations around. Our hope in the Federation Working Group is that federation-v2 can be such a concrete artifact to frame discussions around. We would love to know experiences that folks have had in this problem space, how they feel about federation-v2, and what use-cases they’re interested in exploring in the future. Please feel welcome to join us at the sig-multicluster slack channel or at Federation WG meeting @7:30 PT in the future!

  
  
  https://kubernetes.io/blog/2018/12/12/kubernetes-federation-evolution/
• 

  Blog: etcd: Current status and future roadmap
  • sorangutan  

  1
  1
  Posts
  82
  Views

  

  Author: Gyuho Lee (Amazon Container OSS Team, @gyuho), Joe Betz (Google Cloud, @jpbetz)

  etcd is a distributed key value store that provides a reliable way to manage the coordination state of distributed systems. etcd was first announced in June 2013 by CoreOS (part of Red Hat as of 2018). Since its adoption in Kubernetes in 2014, etcd has become a fundamental part of the Kubernetes cluster management software design, and the etcd community has grown exponentially. etcd is now being used in production environments of multiple companies, including large cloud provider environments such as AWS, Google Cloud Platform, Azure, and other on-premises Kubernetes implementations. CNCF currently has 32 conformant Kubernetes platforms and distributions, all of which use etcd as the datastore.

  In this blog post, we’ll review some of the milestones achieved in latest etcd releases, and go over the future roadmap for etcd. Share your thoughts and feedback on features you consider important on the mailing list: [email protected].

  etcd, 2013

  In June 2014, Kubernetes was released with etcd as a backing storage for all master states. Kubernetes v0.4 used etcd v0.2 API, which was in an alpha stage at the time. As Kubernetes reached the v1.0 milestone in 2015, etcd stabilized its v2.0 API. The widespread adoption of Kubernetes led to a dramatic increase in the scalability requirements for etcd. To handle large number of workloads and the growing requirements on scale, etcd released v3.0 API in June 2016. Kubernetes v1.13 finally dropped support for etcd v2.0 API and adopted the etcd v3.0 API. The table below gives a visual snapshot of the release cycles of etcd and Kubernetes.

  	etcd	Kubernetes
  Initial Commit	June 2, 2013	June 1, 2014
  First Stable Release	January 28, 2015 (v2.0.0)	July 13, 2016 (v1.0.0)
  Latest Release	October 10, 2018 (v3.3.10)	December 3, 2018 (v1.13.0)

  etcd v3.1, early 2017

  etcd v3.1 features provide better read performance and better availability during version upgrades. Given the high use of etcd in production even to this day, these features were very useful for users. It implements Raft read index, which bypasses Raft WAL disk writes for linearizable reads. The follower requests read index from the leader. Responses from the leader indicate whether a follower has advanced as much as the leader. When the follower’s logs are up-to-date, quorum read is served locally without going through the full Raft protocol. Thus, no disk write is required for read requests. etcd v3.1 introduces automatic leadership transfer. When etcd leader receives an interrupt signal, it automatically transfers its leadership to a follower. This provides higher availability when the cluster adds or loses a member.

  etcd v3.2 (summer 2017)

  etcd v3.2 focuses on stability. Its client was shipped in Kubernetes v1.10, v1.11, and v1.12. The etcd team still actively maintains the branch by backporting all the bug fixes. This release introduces gRPC proxy to support, watch, and coalesce all watch event broadcasts into one gRPC stream. These event broadcasts can go up to one million events per second.

  etcd v3.2 also introduces changes such as “snapshot-count” default value from 10,000 to 100,000. With higher snapshot count, etcd server holds Raft entries in-memory for longer periods before compacting the old ones. etcd v3.2 default configuration shows higher memory usage, while giving more time for slow followers to catch up. It is a trade-off between less frequent snapshot sends and higher memory usage. Users can employ lower etcd --snapshot-count value to reduce the memory usage or higher “snapshot-count” value to increase the availability of slow followers.

  Another new feature backported to etcd v3.2.19 was etcd --initial-election-tick-advance flag. By default, a rejoining follower fast-forwards election ticks to speed up its initial cluster bootstrap. For example, the starting follower node only waits 200ms instead of full election timeout 1-second before starting an election. Ideally, within the 200ms, it receives a leader heartbeat and immediately joins the cluster as a follower. However, if network partition happens, heartbeat may drop and thus leadership election will be triggered. A vote request from a partitioned node is quite disruptive. If it contains a higher Raft term, current leader is forced to step down. With “initial-election-tick-advance” set to false, a rejoining node has more chance to receive leader heartbeats before disrupting the cluster.

  etcd v3.3 (early 2018)

  etcd v3.3 continues the theme of stability. Its client is included in Kubernetes v1.13. Previously, etcd client carelessly retried on network disconnects without any backoff or failover logic. The client was often stuck with a partitioned node, affecting several production users. v3.3 client balancer now maintains a list of unhealthy endpoints using gRPC health checking protocol, making more efficient retries and failover in the face of transient disconnects and network partitions. This was backported to etcd v3.2 and also included in Kubernetes v1.10 API server. etcd v3.3 also provides more predictable database size. etcd used to maintain a separate freelist DB to track pages that were no longer in use and freed after transactions, so that following transactions can reuse them. However, it turns out persisting freelist demands high disk space and introduces high latency for Kubernetes workloads. Especially when there were frequent snapshots with lots of read transactions, etcd database size quickly grew from 16 MB to 4 GB. etcd v3.3 disables freelist sync and rebuilds the freelist on restart. The overhead is so small that it is unnoticeable to most users. See “database space exceeded” issue for more information on this.

  etcd v3.4 and beyond

  etcd v3.4 focuses on improving the operational experience. It adds Raft pre-vote feature to improve the robustness of leadership election. When a node becomes isolated (e.g. network partition), this member will start an election requesting votes with increased Raft terms. When a leader receives a vote request with a higher term, it steps down to a follower. With pre-vote, Raft runs an additional election phase to check if the candidate can get enough votes to win an election. The isolated follower’s vote request is rejected because it does not contain the latest log entries.

  etcd v3.4 adds a Raft learner that joins the cluster as a non-voting member that still receives all the updates from leader. Adding a learner node does not increase the size of quorum and hence improves the cluster availability during membership reconfiguration. It only serves as a standby node until it gets promoted to a voting member. Moreover, to handle unexpected upgrade failures, v3.4 introduces etcd downgrade feature.

  etcd v3 storage uses multi-version concurrency control model to preserve key updates as event history. Kubernetes runs compaction to discard the event history that is no longer needed, and reclaims the storage space. etcd v3.4 will improve this storage compact operation, boost backend concurrency for large read transactions, and optimize storage commit interval for Kubernetes use-case.

  To further improve etcd client load balancer, the v3.4 balancer was rewritten to leverage the newly introduced gRPC load balancing API. By leveraging gPRC, the etcd client load balancer codebase was substantially simplified while retaining feature parity with the v3.3 implementation and improving overall load balancing by round-robining requests across healthy endpoints. See Client Architecture for more details.

  Additionally, etcd maintainers will continue to make improvements to Kubernetes test frameworks: kubemark integration for scalability tests, Kubernetes API server conformance tests with etcd to provide release recommends and version skew policy, specifying conformance testing requirements for each cloud provider, etc.

  etcd Joins CNCF

  etcd now has a new home at etcd-io and joined CNCF as an incubating project.

  The synergistic efforts with Kubernetes have driven the evolution of etcd. Without community feedback and contribution, etcd could not have achieved its maturity and reliability. We’re looking forward to continuing the growth of etcd as an open source project and are excited to work with the Kubernetes and the wider CNCF community.

  Finally, we’d like to thank all contributors with special thanks to Xiang Li for his leadership in etcd and Kubernetes.

  
  
  https://kubernetes.io/blog/2018/12/11/etcd-current-status-and-future-roadmap/
• 

  Blog: New Contributor Workshop Shanghai
  • sorangutan  

  1
  1
  Posts
  81
  Views

  

  Author: Josh Berkus (Red Hat), Yang Li (The Plant), Puja Abbassi (Giant Swarm), Xiang Peng Zhao (ZTE)

  

  We recently completed our first New Contributor Summit in China, at the first KubeCon in China. It was very exciting to see all of the Chinese and Asian developers (plus a few folks from around the world) interested in becoming contributors. Over the course of a long day, they learned how, why, and where to contribute to Kubernetes, created a pull requests, attended a panel of current contributors, and got their CLAs signed.

  This was our second New Contributor Workshop (NCW), building on the one created and led by SIG Contributor Experience members in Copenhagen. Because of the audience, it was held in both Chinese and English, taking advantage of the superb simultranslation services the CNCF sponsored. Likewise, the NCW team included both English and Chinese-speaking members of the community: Yang Li, Xiang Peng (Peter) Zhao, Puja Abbassi, Noah Abrahams, Tim Pepper, Zach Corleissen, Sen Lu, and Josh Berkus. In addition to presenting and helping students, the bilingual members of the team translated all of the slides into Chinese. 51 students attended.

  

  The NCW takes participants through the stages of contributing to Kubernetes, starting from deciding where to contribute, followed by an introduction to the SIG system and our repository structure. We also have “guest speakers” from Docs and Test Infrastructure who cover contributing in those areas. We finally wind up with some hands-on exercises in filing issues and creating and approving PRs.

  Those hands-on exercises use a repository known as the contributor playground, created by SIG Contributor Experience as a place for new contributors to try out performing various actions on a Kubernetes repo. It has modified Prow automation and Tide merges, and uses Owners files like in the real repositories. This lets students learn how the mechanics of contributing to our repositories work without disrupting normal development.

  

  Both the “Great Firewall” and the language barrier prevent contributing Kubernetes from China from being straightforward. What’s more, because open source business models are not mature in China, the time for employees work on open source projects is limited.

  Chinese engineers are eager to participate in the development of Kubernetes but many of them don’t know where to start since Kubernetes is such a large project. With this workshop, we hope to help those who want to contribute, whether they want to fix some bugs they encountered, improve or localize documentation, or they need to work with Kubernetes at their work. We are glad to see more and more Chinese contributors joining the community in the past few years, and we hope to see more of them in the future.

  “I have been participating in the Kubernetes community for about three years,” said Xiang Peng Zhao. “In the community, I notice that more and more Chinese developers showing their interest in contributing to Kubernetes. However, it’s not easy to start contributing to such a project. I tried my best to help those who I met in the community, but I think there might still be some new contributors leaving the community due to not knowing where to get help when in trouble. Fortunately, the community initiated NCW at KubeCon Copenhagen and held a second one at KubeCon Shanghai. I was so excited to be invited by Josh Berkus to help organize this workshop. During the workshop, I met community friends in person, mentored attendees in the exercises, and so on. All of this was memorable experience to me. I also learned a lot as a contributor who already has years of contributing experience. I wish I had attended such a workshop when I started contributing to Kubernetes years ago.”

  

  The workshop ended with a panel of current contributors, featuring Lucas Käldström, Janet Kuo, Da Ma, Pengfei Ni, Zefeng Wang, and Chao Xu. The panel aimed to give both new and current contributors a look behind the scenes on the day-to-day of some of the most active contributors and maintainers, both from China and around the world. Panelists talked about where to begin your contributor’s journey, but also how to interact with reviewers and maintainers. They further touched upon the main issues of contributing from China and gave attendees an outlook into exciting features they can look forward to in upcoming releases of Kubernetes.

  After the workshop, Xiang Peng Zhao chatted with some attendees on WeChat and Twitter about their experiences. They were very glad to have attended the NCW and had some suggestions on improving the workshop. One attendee, Mohammad, said, “I had a great time at workshop and learned a lot about the entire process of k8s for a contributor.” Another attendee, Jie Jia, said, “The workshop was wonderful. It systematically explained how to contribute to Kubernetes. The attendee could understand the process even if s/he knew nothing about that before. For those who were already contributors, they could also learn something new. Furthermore, I could make new friends from inside or outside of China in the workshop. It was awesome!”

  SIG Contributor Experience will continute to run New Contributor Workshops at each upcoming Kubecon, including Seattle, Barcelona, and the return to Shanghai in June 2019. If you failed to get into one this year, register for one at a future Kubecon. And, when you meet an NCW attendee, make sure to welcome them to the community.

  Links:

  • English versions of the slides: PDF format or GoogleDocs with speaker notes
  • Chinese version of the slides(PDF)
  • Contributor playground

  
  
  https://kubernetes.io/blog/2018/12/05/new-contributor-workshop-shanghai/
• 

  Blog: Production-Ready Kubernetes Cluster Creation with kubeadm
  • sorangutan  

  1
  1
  Posts
  96
  Views

  

  Authors: Lucas Käldström (CNCF Ambassador) and Luc Perkins (CNCF Developer Advocate)

  kubeadm is a tool that enables Kubernetes administrators to quickly and easily bootstrap minimum viable clusters that are fully compliant with Certified Kubernetes guidelines. It’s been under active development by SIG Cluster Lifecycle since 2016 and we’re excited to announce that it has now graduated from beta to stable and generally available (GA)!

  This GA release of kubeadm is an important event in the progression of the Kubernetes ecosystem, bringing stability to an area where stability is paramount.

  The goal of kubeadm is to provide a foundational implementation for Kubernetes cluster setup and administration. kubeadm ships with best-practice defaults but can also be customized to support other ecosystem requirements or vendor-specific approaches. kubeadm is designed to be easy to integrate into larger deployment systems and tools.

  The scope of kubeadm

  kubeadm is focused on bootstrapping Kubernetes clusters on existing infrastructure and performing an essential set of maintenance tasks. The core of the kubeadm interface is quite simple: new control plane nodes are created by running kubeadm init and worker nodes are joined to the control plane by running kubeadm join. Also included are utilities for managing already bootstrapped clusters, such as control plane upgrades and token and certificate renewal.

  To keep kubeadm lean, focused, and vendor/infrastructure agnostic, the following tasks are out of its scope:

  • Infrastructure provisioning
  • Third-party networking
  • Non-critical add-ons, e.g. for monitoring, logging, and visualization
  • Specific cloud provider integrations

  Infrastructure provisioning, for example, is left to other SIG Cluster Lifecycle projects, such as the Cluster API. Instead, kubeadm covers only the common denominator in every Kubernetes cluster: the control plane. The user may install their preferred networking solution and other add-ons on top of Kubernetes after cluster creation.

  What kubeadm’s GA release means

  General Availability means different things for different projects. For kubeadm, going GA means not only that the process of creating a conformant Kubernetes cluster is now stable, but also that kubeadm is flexible enough to support a wide variety of deployment options.

  We now consider kubeadm to have achieved GA-level maturity in each of these important domains:

  • Stable command-line UX — The kubeadm CLI conforms to #5a GA rule of the Kubernetes Deprecation Policy, which states that a command or flag that exists in a GA version must be kept for at least 12 months after deprecation.
  • Stable underlying implementation — kubeadm now creates a new Kubernetes cluster using methods that shouldn’t change any time soon. The control plane, for example, is run as a set of static Pods, bootstrap tokens are used for the kubeadm join flow, and ComponentConfig is used for configuring the kubelet.
  • Configuration file schema — With the new v1beta1 API version, you can now tune almost every part of the cluster declaratively and thus build a “GitOps” flow around kubeadm-built clusters. In future versions, we plan to graduate the API to version v1 with minimal changes (and perhaps none).
  • The “toolbox” interface of kubeadm — Also known as phases. If you don’t want to perform all kubeadm init tasks, you can instead apply more fine-grained actions using the kubeadm init phase command (for example generating certificates or control plane Static Pod manifests).
  • Upgrades between minor versions — The kubeadm upgrade command is now fully GA. It handles control plane upgrades for you, which includes upgrades to etcd, the API Server, the Controller Manager, and the Scheduler. You can seamlessly upgrade your cluster between minor or patch versions (e.g. v1.12.2 -> v1.13.1 or v1.13.1 -> v1.13.3).
  • etcd setup — etcd is now set up in a way that is secure by default, with TLS communication everywhere, and allows for expanding to a highly available cluster when needed.

  Who will benefit from a stable kubeadm

  SIG Cluster Lifecycle has identified a handful of likely kubeadm user profiles, although we expect that kubeadm at GA can satisfy many other scenarios as well.

  Here’s our list:

  • You’re a new user who wants to take Kubernetes for a spin. kubeadm is the fastest way to get up and running on Linux machines. If you’re using Minikube on a Mac or Windows workstation, you’re actually already running kubeadm inside the Minikube VM!
  • You’re a system administrator responsible for setting up Kubernetes on bare metal machines and you want to quickly create Kubernetes clusters that are secure and in conformance with best practices but also highly configurable.
  • You’re a cloud provider who wants to add a Kubernetes offering to your suite of cloud services. kubeadm is the go-to tool for creating clusters at a low level.
  • You’re an organization that requires highly customized Kubernetes clusters. Existing public cloud offerings like Amazon EKS and Google Kubernetes Engine won’t cut it for you; you need customized Kubernetes clusters tailored to your hardware, security, policy, and other needs.
  • You’re creating a higher-level cluster creation tool than kubeadm, building the cluster experience from the ground up, but you don’t want to reinvent the wheel. You can “rebase” on top of kubeadm and utilize the common bootstrapping tools kubeadm provides for you. Several community tools have adopted kubeadm, and it’s a perfect match for Cluster API implementations.

  All these users can benefit from kubeadm graduating to a stable GA state.

  kubeadm survey

  Although kubeadm is GA, the SIG Cluster Lifecycle will continue to be committed to improving the user experience in managing Kubernetes clusters. We’re launching a survey to collect community feedback about kubeadm for the sake of future improvement.

  The survey is available at https://bit.ly/2FPfRiZ. Your participation would be highly valued!

  Thanks to the community!

  This release wouldn’t have been possible without the help of the great people that have been contributing to the SIG. SIG Cluster Lifecycle would like to thank a few key kubeadm contributors:

  Name	Organization	Role
  Tim St. Clair	Heptio	SIG co-chair
  Robert Bailey	Google	SIG co-chair
  Fabrizio Pandini	Independent	Approver
  Lubomir Ivanov	VMware	Approver
  Mike Danese	Google	Emeritus approver
  Ilya Dmitrichenko	Weaveworks	Emeritus approver
  Peter Zhao	ZTE	Reviewer
  Di Xu	Ant Financial	Reviewer
  Chuck Ha	Heptio	Reviewer
  Liz Frost	Heptio	Reviewer
  Jason DeTiberus	Heptio	Reviewer
  Alexander Kanievsky	Intel	Reviewer
  Ross Georgiev	VMware	Reviewer
  Yago Nobre	Nubank	Reviewer

  We also want to thank all the companies making it possible for their developers to work on Kubernetes, and all the other people that have contributed in various ways towards making kubeadm as stable as it is today!

  About the authors

  Lucas Käldström

  • kubeadm subproject owner and SIG Cluster Lifecycle co-chair
  • Kubernetes upstream contractor, last two years contracting for Weaveworks
  • CNCF Ambassador
  • GitHub: luxas

  Luc Perkins

  • CNCF Developer Advocate
  • Kubernetes SIG Docs contributor and SIG Docs tooling WG chair
  • GitHub: lucperkins

  
  
  https://kubernetes.io/blog/2018/12/04/production-ready-kubernetes-cluster-creation-with-kubeadm/
• 

  Blog: Kubernetes Docs Updates, International Edition
  • sorangutan  

  1
  1
  Posts
  101
  Views

  

  Author: Zach Corleissen (Linux Foundation)

  As a co-chair of SIG Docs, I’m excited to share that Kubernetes docs have a fully mature workflow for localization (l10n).

  Abbreviations galore

  L10n is an abbreviation for localization.

  I18n is an abbreviation for internationalization.

  I18n is what you do to make l10n easier. L10n is a fuller, more comprehensive process than translation (t9n).

  Why localization matters

  The goal of SIG Docs is to make Kubernetes easier to use for as many people as possible.

  One year ago, we looked at whether it was possible to host the output of a Chinese team working independently to translate the Kubernetes docs. After many conversations (including experts on OpenStack l10n), much transformation, and renewed commitment to easier localization, we realized that open source documentation is, like open source software, an ongoing exercise at the edges of what’s possible.

  Consolidating workflows, language labels, and team-level ownership may seem like simple improvements, but these features make l10n scalable for increasing numbers of l10n teams. While SIG Docs continues to iterate improvements, we’ve paid off a significant amount of technical debt and streamlined l10n in a single workflow. That’s great for the future as well as the present.

  Consolidated workflow

  Localization is now consolidated in the kubernetes/website repository. We’ve configured the Kubernetes CI/CD system, Prow, to handle automatic language label assignment as well as team-level PR review and approval.

  Language labels

  Prow automatically applies language labels based on file path. Thanks to SIG Docs contributor June Yi, folks can also manually assign language labels in pull request (PR) comments. For example, when left as a comment on an issue or PR, this command assigns the label language/ko (Korean).

  /language ko
  

  These repo labels let reviewers filter for PRs and issues by language. For example, you can now filter the k/website dashboard for PRs with Chinese content.

  Team review

  L10n teams can now review and approve their own PRs. For example, review and approval permissions for English are assigned in an OWNERS file in the top subfolder for English content.

  Adding OWNERS files to subdirectories lets localization teams review and approve changes without requiring a rubber stamp approval from reviewers who may lack fluency.

  What’s next

  We’re looking forward to the doc sprint in Shanghai to serve as a resource for the Chinese l10n team.

  We’re excited to continue supporting the Japanese and Korean l10n teams, who are making excellent progress.

  If you’re interested in localizing Kubernetes for your own language or region, check out our guide to localizing Kubernetes docs and reach out to a SIG Docs chair for support.

  Get involved with SIG Docs

  If you’re interested in Kubernetes documentation, come to a SIG Docs weekly meeting, or join #sig-docs in Kubernetes Slack.

  
  
  https://kubernetes.io/blog/2018/11/08/kubernetes-docs-updates-international-edition/