Blog: Use KPNG to Write Specialized kube-proxiers
-
Author: Lars Ekman (Ericsson)
The post will show you how to create a specialized service kube-proxy style network proxier using Kubernetes Proxy NG kpng without interfering with the existing kube-proxy. The kpng project aims at renewing the the default Kubernetes Service implementation, the "kube-proxy". An important feature of kpng is that it can be used as a library to create proxiers outside K8s. While this is useful for CNI-plugins that replaces the kube-proxy it also opens the possibility for anyone to create a proxier for a special purpose.
Define a service that uses a specialized proxier
apiVersion: v1 kind: Service metadata: name: kpng-example labels: service.kubernetes.io/service-proxy-name: kpng-example spec: clusterIP: None ipFamilyPolicy: RequireDualStack externalIPs: - 10.0.0.55 - 1000::55 selector: app: kpng-alpine ports: - port: 6000
If the
service.kubernetes.io/service-proxy-name
label is defined thekube-proxy
will ignore the service. A custom controller can watch services with the label set to it's own name, "kpng-example" in this example, and setup specialized load-balancing.The
service.kubernetes.io/service-proxy-name
label is not new, but so far is has been quite hard to write a specialized proxier.The common use for a specialized proxier is assumed to be handling external traffic for some use-case not supported by K8s. In that case
ClusterIP
is not needed, so we use a "headless" service in this example.Specialized proxier using kpng
A kpng based proxier consists of the
kpng
controller handling all the K8s api related functions, and a "backend" implementing the load-balancing. The backend can be linked with thekpng
controller binary or be a separate program communicating with the controller using gRPC.kpng kube --service-proxy-name=kpng-example to-api
This starts the
kpng
controller and tell it to watch only services with the "kpng-example" service proxy name. The "to-api" parameter will open a gRPC server for backends.You can test this yourself outside your cluster. Please see the example below.
Now we start a backend that simply prints the updates from the controller.
$ kubectl apply -f kpng-example.yaml $ kpng-json | jq # (this is the backend) { "Service": { "Namespace": "default", "Name": "kpng-example", "Type": "ClusterIP", "IPs": { "ClusterIPs": {}, "ExternalIPs": { "V4": [ "10.0.0.55" ], "V6": [ "1000::55" ] }, "Headless": true }, "Ports": [ { "Protocol": 1, "Port": 6000, "TargetPort": 6000 } ] }, "Endpoints": [ { "IPs": { "V6": [ "1100::202" ] }, "Local": true }, { "IPs": { "V4": [ "11.0.2.2" ] }, "Local": true }, { "IPs": { "V4": [ "11.0.1.2" ] } }, { "IPs": { "V6": [ "1100::102" ] } } ] }
A real backend would use some mechanism to load-balance traffic from the external IPs to the endpoints.
Writing a backend
The
kpng-json
backend looks like this:package main import ( "os" "encoding/json" "sigs.k8s.io/kpng/client" ) func main() { client.Run(jsonPrint) } func jsonPrint(items []*client.ServiceEndpoints) { enc := json.NewEncoder(os.Stdout) for _, item := range items { _ = enc.Encode(item) } }
(yes, that is the entire program)
A real backend would of course be much more complex, but this illustrates how
kpng
let you focus on load-balancing.You can have several backends connected to a
kpng
controller, so during development or debug it can be useful to let something like thekpng-json
backend run in parallel with your real backend.Example
The complete example can be found here.
As an example we implement an "all-ip" backend. It direct all traffic for the externalIPs to a local endpoint, regardless of ports and upper layer protocols. There is a KEP for this function and this example is a much simplified version.
To direct all traffic from an external address to a local POD only one iptables rule is needed, for instance;
ip6tables -t nat -A PREROUTING -d 1000::55/128 -j DNAT --to-destination 1100::202
As you can see the addresses are in the call to the backend and all it have to do is:
- Extract the addresses with
Local: true
- Setup iptables rules for the
ExternalIPs
A script doing that may look like:
xip=$(cat /tmp/out | jq -r .Service.IPs.ExternalIPs.V6[0]) podip=$(cat /tmp/out | jq -r '.Endpoints[]|select(.Local == true)|select(.IPs.V6 != null)|.IPs.V6[0]') ip6tables -t nat -A PREROUTING -d $xip/128 -j DNAT --to-destination $podip
Assuming the JSON output above is stored in
/tmp/out
(jq is an awesome program!).As this is an example we make it really simple for ourselves by using a minor variation of the
kpng-json
backend above. Instead of just printing, a program is called and the JSON output is passed asstdin
to that program. The backend can be tested stand-alone:CALLOUT=jq kpng-callout
Where
jq
can be replaced with your own program or script. A script may look like the example above. For more info and the complete example please see https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec.Summary
While kpng is in early stage of development this post wants to show how you may build your own specialized K8s proxiers in the future. The only thing your applications need to do is to add the
service.kubernetes.io/service-proxy-name
label in the Service manifest.It is a tedious process to get new features into the
kube-proxy
and it is not unlikely that they will be rejected, so to write a specialized proxier may be the only option.
https://kubernetes.io/blog/2021/10/18/use-kpng-to-write-specialized-kube-proxiers/
- Extract the addresses with
© Lightnetics 2024