Multi-cluster service mesh and federation — connectivity across clusters

A multi-cluster deployment needs explicit tooling for service connectivity. Pods in cluster A cannot reach Pods in cluster B by default. This lesson walks Submariner, Skupper, service mesh federation, and the operational discipline.

The cross-cluster connectivity problem

flowchart LR
    A["Cluster A: pod 10.0.1.5"] -->|default: no route| X["Cluster B: pod 10.0.2.5"]

By default, Kubernetes Pod CIDRs do not overlap across clusters but there is no route between them. A Pod in cluster A cannot reach a Pod in cluster B unless the cluster-to-cluster network is configured.

The connectivity options:

flowchart TD
    A[Cross-cluster connectivity] --> B["L3: Submariner"]
    A --> C["L7: Skupper"]
    A --> D["Service mesh: Istio, Linkerd"]
    B --> B1["IPsec tunnels, Pod-to-Pod IP reachable"]
    C --> C1["AMQP bridges, app-level proxy"]
    D --> D1["Unified service identity, traffic management"]

Submariner for L3 cross-cluster

flowchart LR
    A["Cluster A: pod 10.0.1.5"] -->|IPsec tunnel| B[Gateway A]
    B -->|public network| C[Gateway B]
    C --> D["Cluster B: pod 10.0.2.5"]

Submariner creates IPsec tunnels between cluster gateways. Once the tunnels are up, Pod CIDRs from cluster A are reachable from cluster B and vice versa. Service ClusterIPs are exported across clusters.

The Submariner components:

• Gateway engine — runs on dedicated nodes in each cluster; terminates the IPsec tunnel.
• Route agent — runs on every worker node; programs routes for the remote Pod CIDRs.
• Service export — declares which Services are exported to remote clusters.

apiVersion: submariner.io/v1alpha1
kind: ServiceExport
metadata:
  name: billing
  namespace: prod-app

The ServiceExport makes the billing Service in cluster A reachable from cluster B via its ServiceExport name.

Skupper for L7 bridging

flowchart LR
    A["Cluster A: app"] -->|Skupper proxy| B[AMQP router]
    B -->|AMQP over TLS| C["Cluster B: Skupper proxy"]
    C --> D["Cluster B: app"]

Skupper provides application-level bridging via AMQP over TLS. The application connects to a local Skupper proxy; the proxy forwards the connection to the remote cluster’s Skupper proxy; the remote proxy delivers to the application.

Skupper does not require Pod-to-Pod connectivity — only AMQP over TLS between the routers. The application sees a local connection; the network sees an AMQP stream.

Service mesh federation

flowchart LR
    A[Istio mesh in cluster A] -->|mTLS| B[Istio mesh in cluster B]
    C[Linkerd mesh in cluster A] -->|mTLS| D[Linkerd mesh in cluster B]

Istio multi-primary and Linkerd multi-cluster provide service mesh federation:

• Istio multi-primary. Each cluster has its own Istio control plane; the control planes exchange service identities; mTLS between clusters is automatic.
• Linkerd multi-cluster. Each cluster has its own Linkerd control plane; service mirroring exports Services; mTLS is automatic.

Service mesh federation provides:

• Unified service identity. A Service in cluster A is identifiable from cluster B via its mesh identity.
• mTLS across clusters. Traffic between clusters is encrypted and authenticated.
• Traffic management. Cross-cluster traffic can be routed by the mesh (failover, load balancing).

KubeFed (deprecated)

KubeFed (Kubernetes Federation) was the original multi-cluster API. It is deprecated; most workloads have migrated to Submariner, service mesh federation, or vendor-specific solutions. New deployments should not use KubeFed.

The choice framework

The choice depends on the use case. Most production multi-cluster deployments use Submariner for network connectivity plus a service mesh for traffic management and identity.

The operational failure modes

Cross-cluster connectivity fails for predictable reasons:

• Gateway nodes unreachable. The Submariner gateway nodes cannot reach each other across the network. VPC peering is misconfigured.
• ServiceExport not declared. A Service is not reachable from remote clusters because it has no ServiceExport.
• Mesh certificates expired. Istio or Linkerd certificates expire; cross-cluster mTLS fails.
• MTU mismatch. Submariner’s IPsec tunnel MTU does not match the underlying network; large packets are dropped.
• Service mesh version skew. Different Istio or Linkerd versions across clusters produce API incompatibilities.

Quiz

Production discipline

Cross-cluster connectivity in production rests on five non-negotiable elements:

• Verify the underlying network first. VPC peering, security groups, route tables. Submariner requires network connectivity between gateways.
• Declare ServiceExports explicitly. A Service is not reachable from remote clusters without a ServiceExport.
• Test cross-cluster connectivity quarterly. A tunnel that has never been tested may fail at the worst moment.
• Monitor mesh certificate expiry. Istio and Linkerd certificates expire; alert on the remaining validity.
• Document the topology. The architecture diagram must show the cross-cluster connectivity, the gateways, the ServiceExports, and the mesh configuration.

Cross-cluster connectivity is production networking. Treat it as such: HA, monitored, tested, documented.