By Srinivas Gumdelli & Nathan Typanski

Migrations to Kubernetes are inherently messy. While progress has been made in recent years with StatefulSet and persistent volumes, these are relatively young features in Kubernetes, which is built upon the concept of ephemeral pods. As such, compromises must be made for legacy migrations – a common one is to migrate only stateless applications to Kubernetes. But this leaves the question of network security: how do we restrict access to external services to only some containers on a single host virtual machine (VM)?

While there are a couple of ways to go about this, the most common solution is to use a service mesh.

Today, Istio is one of the most popular mature service meshes available. It was initially developed by Google and IBM using the Envoy proxy from Lyft. The basic idea of service meshes is to solve cross-cutting concerns for applications like:

• Telemetry (Prometheus and Grafana),
• Authorization and authentication (custom RBAC framework),
• Canary and red/black deployments (built-in tools),
• Traffic routing and restriction (L7 routing of network traffic).

Istio addresses these concerns by running a sidecar attached to each pod in Kubernetes:

These sidecars are set up to intercept all traffic from the service container, apply security and routing policies, and collect telemetry data. By injecting these sidecars automatically, we can control egress traffic from a Kubernetes cluster in a systematic way.

Istio also provides a feature called mesh expansion that allows the services running outside the kubernetes cluster (on the VMs) to also join the service mesh and utilize its benefits as if it were a first class citizen. 

During our service migration to Kubernetes, we found a compelling use-case for the mesh expansion feature.

Mesh Expansion

Using mesh expansion, we can integrate services on VMs into a Kubernetes-native service mesh, providing all of Istio’s features to bare-metal virtual machines:

There’s just one catch: The official documentation for mesh expansion focuses mainly on Google Kubernetes Engine, IBM Cloud, and bare-metal virtual machines, but there is nothing written about mesh expansion on AWS (so we assume nobody has accomplished it!). We asked on the forums if anyone has done this before, and community members requested that we share the steps if we were to translate mesh expansion to AWS. 

In this article, we’ll share some of our learnings about getting mesh expansion to work in a Kubernetes cluster setup in AWS with Kops and how we automated the process.

Mesh Expansion with AWS

For keeping the post to the point, we’re going to skip the detailed steps for mesh expansion and list only the steps that are specific to AWS. Take a look at the official documentation for setting up mesh expansion.

Setup the VM to be able to talk to Citadel and Pilot

1. Expose the Citadel and Pilot services with either an internal network load balancer or an Istio ingress gateway (source can be found here)
2. Setup DNS resolver for Citadel and Pilot services to be able to resolve through the DNS names istio-citadel, istio-pilot and istio-pilot.istio-system
   1. The docs for mesh expansion suggest using the IP address of the load balancer for Citadel and Pilot, hard coded as an alias for the above hostnames in /etc/hosts. Since AWS by default uses DNS records for the load balancers as opposed to IPs used by GCP and IBM Cloud, we can work around it by using a DNS resolver like resolvconf and the primary DNS (Route53 in our case) provider. Resolvconf completes a query name to a fully qualified domain name when no domain suffix is supplied. 
   2. Add the domain suffix to /etc/resolv.conf and the sidecar running on the VM should now be able to resolve both Pilot and Citadel CNAME the DNS records in Route53 (or on the primary DNS server) for both the load balancers to match the DNS names that the sidecar looks for. If you use external-dns in Kubernetes, the network load balancer yaml can automatically create the DNS records for you. If not, add those CNAMEs manually to your primary DNS server (Route53 in our case).
3. ISTIO_SERVICE_CIDR: The docs suggest using gcloud cli to get the CIDR block for the Kubernetes cluster. You can instead use the following command to get the same result:

   ISTIO_SERVICE_CIDR=$(kubectl cluster-info dump | grep -m 1 service-cluster-ip-range | sed -n 's/.*service-cluster-ip-range=//p' | awk '{print $1}')

These are the main differences from the official documentation to get sidecar running in the VM to join the mesh.

Verifying the setup

Once the sidecar is up and running, we can verify the setup for iptables rules on the VM using the following command: 

$ iptables -t nat nVL 

This should display all the nat rules that divert the traffic to the sidecar and also display the packet count for each port/rule.

Automating the setup

At Paxos, we are big on automating everything. We’re currently automating as many processes as possible across the organization. We use Terraform/Terragrunt for setting up our infrastructure, Chef for managing the configuration, Jenkins for CI, Helm for managing and deploying services to Kubernetes and Vault for managing our secrets. We ensure that any new instances (for services that need to be run on VMs) are set up with Terraform and are automatically bootstrapped to our Chef server along with the run-list based on the service/application.  

In order to automate the setup, we wrote a chef LWRP (will be open-sourced soon) that fetches the certs from Pilot, generates the config files and sets up everything required to join the mesh. For this to be possible, we generated a limited access Kubernetes service token that is stored in Vault and the instances that need access to the token are assigned an IAM role that has the ability to read the secret from Vault.

Final Thoughts 

Overall, setting up the mesh expansion took us at least two weeks to figure out and then automate it from end-to-end. We know we would have appreciated more documentation, so we wanted to share our progress with the community.