DigitalOcean Kubernetes with Static IPv4

Table of Contents

Thanks to the new VPC functionality in DigitalOcean can be used to provide Kubernetes with a static external IPv4. This can be handy in cases where you need to deal with IP whitelists, for example, if you use your Kubernetes cluster as a CI building tool. However, this requires some config setup and a privileged pod running on each node to automatically update the routes. This article will help guide you through the setup.

This will work for other types of Kubernetes instances as long as you have an internal range and one VM in it which you can setup as router. Disclaimer: I am not a Kubernetes expert and I only confirmed it to work in our environment. If you want to use it, please test it yourself first.

Setting up a VM for routing #

To begin, you need a separate small VM for routing with an external static IPv4. DigitalOcean will already have created a default VPC per region you have servers/Kubernetes in. We will use this Private IP range for routing internally.

Create a basic Debian server within the DigitalOcean control panel. The external IP of this VM will be the external IP of your Kubernetes pods soon. Run the following commands to start:

apt update
apt install iptables iptables-persistent
sysctl -w net.ipv4.ip_forward=1
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf

Now you need to find which interface your public IP network is connected to and your private IP range with subnet. Use the following to gain this info:

ip address

After this, use this info replace the vpc_network_prefix (eg. 10.100.0.0/20) and public_interface_name (eg. ens3).

iptables -t nat -A POSTROUTING -s <vpc_network_prefix> -o <public_interface_name> -j MASQUERADE
iptables-save >/etc/iptables.rules.v4

Note down the VPC internal IP address (eg. 10.100.10.5), this server is complete.

Kubernetes #

Disclaimer! This should no longer be used. When a Node’s default route is changed, it is then unable to contact the Control plane and it’s associated components. Instead, take a look at DigitalOcean’s own static route operator and only set routes for IP ranges you want to access like this.

For Kubernetes, I created a container that can change the route on the node hosts with an environment variable. Deploy it like this using kubectl create -f techwolf12.yaml, if you save the contents of this file to techwolf12.yaml. Don’t forget to change the INETROUTE env value in this config before applying it to your cluster!

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-static-route
  namespace: default
automountServiceAccountToken: true
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-static-route
rules:
- apiGroups: [""]
  resources:
    - "nodes"
    - "nodes/metrics"
    - "nodes/stats"
    - "nodes/proxy"
    - "pods"
    - "secrets"
    - "services"
  verbs: ["get", "list"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-static-route
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-static-route
subjects:
- kind: ServiceAccount
  name: kube-static-route
  namespace: default
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-static-route
  namespace: default
  labels:
    app: kube-static-route
spec:
  selector:
    matchLabels:
      name: kube-static-route
  updateStrategy:
      type: RollingUpdate # Only supported in Kubernetes version 1.6 or later.
  template:
    metadata:
      labels:
        name: kube-static-route
      annotations:
        # Needed for Kubernetes versions prior to 1.6.0, where tolerations were set via annotations.
        scheduler.alpha.kubernetes.io/tolerations: |
          [{"operator": "Exists", "effect": "NoSchedule"},{"operator": "Exists", "effect": "NoExecute"}]          
    spec:
      serviceAccountName: kube-static-route
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: kube-static-route
          image: ghcr.io/techwolf12/kube-static-route:master
          securityContext:
            privileged: true
          resources:
            limits:
              memory: 150Mi
            requests:
              cpu: 100m
              memory: 150Mi
          env:
            - name: "INETROUTE"
              value: "<CHANGE THIS VALUE TO YOUR VPC INTERNAL ROUTE IP>"
      tolerations:
        - operator: "Exists"
          effect: "NoSchedule"
        - operator: "Exists"
          effect: "NoExecute"

The routing should now automatically update on all nodes, including the ones being added or upgraded. Pods should use this routing by default as well. This works because the container techwolf12/kubernetes-static-route will change the route on the host node and make sure it stays on the correct route.