Network is one of the trickiest parts of a Kubernetes (K8s)’ cluster. That’s because there are multiple layers of abstraction that has to be implemented in different ways (Host, Container Engine, Kubernetes, and so on) to make the communication between objects within the cluster feasible.

Today’s post is not about explaining or getting into the details of how networking is handled by Kubernetes. Actually, K8s documentation does a great job on laying this down. If you would like to dig deeper on that topic, I strongly recommend taking a look in there. I’m going to briefly touch some of the concepts related to that though, for contextualization purposes, but it doesn’t intend to serve as guide towards K8s networking. I’m also considering we are targeting a Linux-based environment for Kubernetes.

Rather, what I would like to cover with this article is a small piece of universe that might happen to occur within your environment: the communication of a host level element (Node, VM, whatever under the same network) with k8s level elements (Pods). For instance, how do I make a Pod communicate with a legacy service living on a Windows Server-based VM? I’m going to answer that and give insights on the best-practices to make that kind of scenario doable.

A word about network in Kubernetes

Lets briefly talk about networking in Kubernetes, then.

Kubernetes’ clusters runs on top of what we call “Host” machines. In the context of the cluster, these hosts are known as “Nodes”. In the context of a cloud platform, a Node is a Virtual Machine (VM). In Azure, as you might know, to have a VM up and running, you have to have it attached to a virtual network previously spun up. It means that, at the host level, we have a first network required. The Figure 1 below visually illustrates that process.

Figure 1. IP allocation for Virtual Machines in a VNet in Azure

At this point, it has nothing to do with Kubernetes yet. It is just a basic allocation of IPs in a given subnet in Azure. Then, we bring Kubernetes into play and this will change the scenario a little bit, as you can see through Figure 2.

Figure 2. Kubernetes network abstraction on top of the nodes

As you might know, Kubernetes has a central piece of operation – Pods. As you might also know, a Pod can bear one or multiple containers within it. All of those elements need to communicate some point. In fact, Kubernetes does apply the following premises when it comes to communication among cluster’s elements:

  • All Pods can communicate with all other Pods without using network address translation (NAT).
  • All Nodes can communicate with all Pods without NAT.
  • The IP that a Pod sees itself as is the same IP that others see it as.

Given these constraints, we are left off with four distinct networking problems to solve:

  1. Container-to-Container networking
  2. Pod-to-Pod networking
  3. Pod-to-Service networking
  4. Internet-to-Service networking

Important to mention that at this point to kind of “set the ground up” towards the addition of the red piece at Figure 2. Still, some additional explanation about what is happening there might be helpful:

  1. K8s needs its own CIDR to properly operate. That’s why we are setting up the cluster to use 10.240.0.0/24. It means that every node under that given CIDR will be able to receive its own sub-range (in the example, 10.240.0.0/24, 10.240.1.0/24, 10.240.2.0/24) of IPs to properly allocate to underlying Pods. Worth to mention that these IPs are all virtual. This information will be important later on.
  2. Inside the node, as you can see, every Pod does receive its own IP. Containers within the same Pod will be able to communicate to each other through localhost:<port>, as they are local in that context. You are in charge of making sure that containers in the same pod don’t duplicate ports.
  3. The acronym cnio0 does relate to the Linux ethernet bridge, which is one of the key elements under-the-hood to make the communication pod-to-pod throughout cluster’s nodes possible.
  4. As the name suggests, kube-dns (which in practical terms is a Pod living in the worker nodes) is the element that does DNS work for us, translating IPs into service names and vice-versa. It has to be able to communicate at the VNet level as well to properly NAT stuff when needed. That’s why it has 10.32.0.10 address tied to it.

Going up a level, Kubernetes has this concept of “Services”, which is how we make Pods in a cluster, reachable. Again, K8s documentation does a great job laying that concept down, so I’m not getting into the details here. If this is new to you, please, go visit the documentation.

It is really important to understand that concept here once it will be critical for the solution we’re going to build from now on.

The scenario

Imagine now that we have a service running in a given Linux-based Virtual Machine. After going through a process of refactoring some internal applications, an important piece of information that the service living on the Virtual Machine depends on was moved to a microservice that leaves in a Azure Kubernetes Service (AKS) placed at the same Azure VNet. With that move, the communication got broken (Pod inaccessible) and we’ve got to fix it.

Figure 3 brings a high-level view of the scenario that needs to be solved.

Figure 3. Depicted scenario to be solved

Assuming now that we know by a fact that a service (svc) was already created in the cluster to expose internally the Pods (through clusterIP) that returns the requested data to the caller via GET, and we continue to witness the broken communication. Why is the communication broken? What’s wrong with that set up?

Well, pretty straightforward if you remember how network is handled by Kubernetes. As we saw early on, K8s has its own “Service CIDR”, and it is a virtual IP range, that is valid on the context of the cluster only. How does either kube-dns or coreDNS know how to resolve a call coming from a “external” IP?

That’s where the solution we’re proposing here comes into play.

The solution: Internal load-balancer through service

As I mentioned early on, the way we expose a Pod to an external call (whatever it is coming from) is through Services. In our case, the call is not going to succeed as the CIDR from the caller is different of Pod’s CIDR. So, we need “something” in the middle that can understand VNet’s IP (caller’s IP) and cluster’s IP (Pods). That element for us is going to be an internal load-balancer.

So, this is what we’re going to do. I mean, we are going to create a Kubernetes service that leverages an Azure plugin to deploy an internal load-balancer. Because the load-balancer has a VNet IP attached to it and also has the ability to “oversee” cluster’s IPs, that service will then become reachable from whatever resource calling from the same VNet and will be able to properly balance incoming requests to the Pods underneath.

The new architecture will then look like the one being shown by Figure 4.

Figure 4. Internal load-balancer being shown

But first, in case you would like to follow along with the entire process, let’s go through the process of creating all the resources (resource group, network, subnet, cluster, sample app and service).

Deploying the solution

Step 1. Spinning up AKS cluster + underlying resources

To deploy the solution, I’m going to use Azure CLI. From now on, I’m assuming you already have it set up at your end. If that is not your case and you want to follow along, you can go through the steps described on the document available through this link. Also, you’ll need to have an Azure subscription with proper deployment allowance for the resources set up.

Let’s start by defining some global variables for us to save some time and script lines. Please notice that I’m using Azure CLI over PowerShell to get it done.

$RG="Demo-AKSInternalCommunication"
$LOCATION="eastus"
$SUB="{your-subscriptionid-here}"
$VNET="vnet-vmtopod"
$SUBNET="cluster"
$AKS="aks-vmtopod"

Time to get ourselves authenticated and properly authorized in Azure.

az login

Next, let’s set up the Azure subscription we’re going to use for our deployments.

az account set --subscription $SUB

Up next, we’re going to create a new Resource Group to logically group the resources of our environment.

az group create `
    -n $RG `
    -l $LOCATION

Next, I’m going to create a new virtual network and subnet to support the solution. If you already have one and would rather leverage it, not a problem. Go for it. Just add a new subnet to the cluster later on.

az network vnet create `
    -n $VNET `
    -g $RG `
    -l $LOCATION `
    --address-prefix 10.0.0.0/16 `
    --subnet-name $SUBNET `
    --subnet-prefix 10.0.0.0/24

Now, before creating the AKS cluster, I’ll capture the subnet Id. We will need to pass that information along to the cluster by the time we create it. Bellow’s command does just that.

$SUBNET_ID=$(az network vnet subnet show --resource-group $RG --vnet-name $VNET --name $SUBNET --query id -o tsv)

Now that we finally have everything we need to spin up the AKS cluster, let’s execute the command that does that. As you may have noticed, I’m defining couple of configurations for the cluster: version, how the SSH keys will be managed, number of nodes, network plugin, plugin for network policy, service CIDR (remember when we talked about it?), DNS service IP, and more. Also, I have hooked up the subnet ID, as mentioned on the previous step. This is how Azure knows to which VNet + Subnet to attach the AKS cluster.

az aks create `
    -n $AKS `
    -g $RG `
    -l $LOCATION `
    --kubernetes-version 1.20.5 `
    --generate-ssh-keys `
    --node-count 2 `
    --network-plugin kubenet `
    --node-vm-size Standard_DS2_v2 `
    --network-policy calico `
    --vnet-subnet-id $SUBNET_ID `
    --service-cidr 10.200.0.0/16 `
    --dns-service-ip 10.200.0.10 `
    --pod-cidr 10.244.0.0/16 `
    --docker-bridge-address 172.17.0.1/16

If you receive a confirmation prompt related to User assigned identity versus System assigned identity, you can safely go ahead and confirm it. Alternatively, you could create a user assigned identity and leverage it at the cluster level as authority.

Before to move on, let’s make sure everything was deployed correctly. By navigating through the Azure portal, you should be able to see the new AKS cluster’s panel indicating that everything is working properly (Figure 5).

Figure 5. Azure portal showing the success of cluster’s deployment

Now, from the Kubernetes side, to make sure everything is up and running, we’re going to run kubectl get nodes command. But, before of doing so we need to authenticate ourselves with the cluster. That’s why I’m requesting to Azure the cluster’s credential so that, it can be merged into my local ~/.kube/config file. Bellow’s command does that for me.

az aks get-credentials `
    --resource-group $RG `
    --name $AKS

Now we’re ready to communicate with the cluster from whatever we are. Assuming that kubectl command line tool is already set up in your source machine (if not, here’s how you get it done), just ran the following to receive back the information about the newly deployed cluster. As result, you should be seeing something quite similar to what has been shown by Figure 6.

kubectl get nodes -o wide
Figure 6. Information about AKS’ nodes newly deployed

Step 2. Creating the external Virtual Machine

Next, we need to create the external virtual machine. This will be the one to “simulate” the external service which will depend on Pod’s return (upon a HTTP Get request).

But before doing so, we need to add a new subnet to existing VNet where our AKS cluster is attached to. The command below does exactly that.

az network vnet subnet create `
    -n servers-farm `
    -g $RG `
    --vnet-name $VNET `
    --address-prefixes 10.0.1.0/24

Now, lets capture the newly create subnet’s Resource ID. As we did a bit go with AKS, we need to inform Azure which VNet and subnet we would like to have that VM attached to. Once again, we do this by supplying subnet’s ID. To do so, I have defined two new global variables in my script: $SUBNET_ID_VM and $SUBNET_VM.

Command below does that for us.

$SUBNET_ID_VM=$(az network vnet subnet show --resource-group $RG --vnet-name $VNET --name $SUBNET_VM --query id -o tsv)

Next up, the Virtual Machine. The command below creates a Linux-based server (Ubuntu LTS) and attach it to its respective subnet. If, upon completion of the script below you’ve been presented to a JSON object like the one depicted by Figure 7, everything went well and the VM was successfully deployed.

az vm create `
    -n service-vmtopod `
    -g $RG `
    -l $LOCATION `
    --subnet $SUBNET_ID_VM `
    --generate-ssh-keys `
    --image UbuntuLTS `
    --admin-username {you-admin-username-here}
Figure 7. VM successfully deployed.

Step 3. Deploying the services and testing

Our infrastructure is ready. Now, it is time for us to deploy a couple of services both on the virtual machine and in AKS to make sure the approach we are proposing here does work and solve the connectivity’s issue.

Lets start by AKS. First, we’re going to create an application in AKS that will simulate the API that is going to be called later on by our external service living in within the VM. Bellow’s YAML present that deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: basicpod
spec:
  selector:
    matchLabels:
      type: webserver
  replicas: 3
  template:
    metadata:
      labels:
        type: webserver
    spec:
      containers:
      - name: webcont
        image: nginx
        ports:
        - containerPort: 80
  • We are creating a K8s object called Deployment. It will make sure to deploy our Pods properly, taking care of aspects like replicability, resiliency, so on so forth.
  • We are defining a label (webserver) to enable a service later on to match up its configurations to the proper Pods.
  • We are defining three stateless replicas to deal with requests targeting our “API”.
  • We are deploying a NGINX service. It will listen requests at the port 80 of the container.

To deploy the Pods, assuming the command is being executed from the same directory where the YAML file is currently sitting on, just execute the command below.

kubectl apply -f {your-file-name}.yaml

Upon completion, a message “deployment.apps/{your-pod-name} created” should be issued, indicating the pods were successfully deployed in AKS. To make sure they’re up and running, run the following command line. The return should be something quite like Figure 8.

kubectl get pod -o wide
Figure 8. Pods successfully created in AKS

We have now three instances of our API running but still, there is no way to access it from the outside of the node. Even worse, there is no way to call it from outside of the cluster, which is our ultimate goal. Then, what I’m going to do is create a service (as discussed earlier in this post). The following YAML snippet code describes the service configuration.

Characteristics of this service will be:

  • Type: LoadBalancer
  • Selector: webserver (it will match up the label previously defined for the Pods)
  • Plugin: Microsoft Azure Internal Load Balancer
  • Port: 80
apiVersion: v1
kind: Service
metadata:
 name: basicinternalservice
 annotations: 
  service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
 type: LoadBalancer
 ports:
 - port: 80
 selector:
  type: webserver

In the same way, run the following command line to apply the service creating in Kubernetes.

kubectl apply -f {your-service-file-name}.yaml

Again, if everything went well, a message “service/{your-service-name} created” has been issued. The service is now created. Lets make sure we have it running in AKS. We do so by running the following command. The result should be something like what is presented by Figure 9.

kubectl get svc -o wide
Figure 9. Service created, up and running

As you might have noticed, the service object has two different IPs. CLUSTER-IP and EXTERNAL-IP. While CLUSTER-IP‘s address fits within Kubernetes’s CIDR, EXTERNAL-IP does fit on the VNet’s IP. That’s the reason by which the service (which is internal, meaning, there is no public IP exposed) is able to handle calls both from and to the Pods underneath. Also, it is mapping port 80 to 31347.

Theoretically, we should now be able to call the Pods running our fake API from our VM, where our origin service is running on. So, let’s make sure we can make that communication happen.

From the inside of your Virtual Machine (and I’m assuming here that you know how to access a Linux-VM on Azure, but if you don’t there is a good documentation explaining how to do so in here), run this simple curl command.

curl http://10.0.0.6

If everything is well configured, you should be able to see NGINX welcome page, coming out, as Figure 10 depics.

Figure 10. NGINX welcome page being shown

Done! Goal achieved. Before closing though, I wanted to make sure that you understand what is happening.

When we curl the IP 10.0.0.6 from a Virtual Machine from the same network, we are actually hitting a load balancer that was deployed attached to the same VNet. Kubernetes’ service uses that Azure load balancer to balance the loads among the Pods managed by it by keeping the track of both IPs (we briefly discussed it earlier).

In the other hand, if another Pod sitting into a different node needs access to the Pods under the service we just created, it could hit it by calling 10.200.215.66, which is a internal IP to the cluster and fits into the K8s’ condition described early on “Pod-to-Pod networking”.

That’s it. Hope it helps!


1 Comment

Nikhil Utane · October 12, 2021 at 11:03 am

Nice write-up. One question though, shouldn’t the cluster CIDR be 10.240.0.0/16 instead of 10.240.0.0/24?

Leave a Reply

Your email address will not be published. Required fields are marked *