- Egress Gateways
- Use case
- Before you begin
- Deploy Istio egress gateway
- Egress gateway for HTTP traffic
- Cleanup HTTP gateway
- Egress gateway for HTTPS traffic
- Cleanup HTTPS gateway
- Additional security considerations
- Apply Kubernetes network policies
- Cleanup network policies
- Troubleshooting
- Cleanup
- See also
Egress Gateways
This example does not work in Minikube.
The Control Egress Traffic task shows how to configureIstio to allow access to external HTTP and HTTPS services from applications inside the mesh.There, the external services are called directly from the client sidecar.This example also shows how to configure Istio to call external services, although this timeindirectly via a dedicated egress gateway service.
Istio uses ingress and egress gatewaysto configure load balancers executing at the edge of a service mesh.An ingress gateway allows you to define entry points into the mesh that all incoming traffic flows through.Egress gateway is a symmetrical concept; it defines exit points from the mesh. Egress gateways allowyou to apply Istio features, for example, monitoring and route rules, to traffic exiting the mesh.
Use case
Consider an organization that has a strict security requirement that all traffic leavingthe service mesh must flow through a set of dedicated nodes. These nodes will run on dedicated machines,separated from the rest of the nodes running applications in the cluster. These special nodes will servefor policy enforcement on the egress traffic and will be monitored more thoroughly than other nodes.
Another use case is a cluster where the application nodes don’t have public IPs, so the in-mesh services that runon them cannot access the Internet. Defining an egress gateway, directing all the egress traffic through it, andallocating public IPs to the egress gateway nodes allows the application nodes to access external services in acontrolled way.
Before you begin
Setup Istio by following the instructions in the Installation guide.
Deploy the sleep sample app to use as a test source for sending requests.If you haveautomatic sidecar injectionenabled, run the following command to deploy the sample app:
Zip
$ kubectl apply -f @samples/sleep/sleep.yaml@
Otherwise, manually inject the sidecar before deploying the sleep application with the following command:
Zip
$ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@)
You can use any pod with curl installed as a test source.
- Set the
SOURCE_PODenvironment variable to the name of your source pod:
$ export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
- Enable Envoy’s access logging
Deploy Istio egress gateway
- Check if the Istio egress gateway is deployed:
$ kubectl get pod -l istio=egressgateway -n istio-system
If no pods are returned, deploy the Istio egress gateway by performing the next step.
- Run the following command:
$ istioctl manifest apply --set values.global.istioNamespace=istio-system \--set values.gateways.istio-ingressgateway.enabled=false \--set values.gateways.istio-egressgateway.enabled=true
The following instructions create a destination rule for the egress gateway in the default namespaceand assume that the client, SOURCE_POD, is also running in the default namespace.If not, the destination rule will not be found on thedestination rule lookup pathand the client requests will fail.
Egress gateway for HTTP traffic
First create a ServiceEntry to allow direct traffic to an external service.
- Define a
ServiceEntryforedition.cnn.com:
$ kubectl apply -f - <<EOFapiVersion: networking.istio.io/v1alpha3kind: ServiceEntrymetadata:name: cnnspec:hosts:- edition.cnn.comports:- number: 80name: http-portprotocol: HTTP- number: 443name: httpsprotocol: HTTPSresolution: DNSEOF
- Verify that your
ServiceEntrywas applied correctly by sending an HTTP request to http://edition.cnn.com/politics.
$ kubectl exec -it $SOURCE_POD -c sleep -- curl -sL -o /dev/null -D - http://edition.cnn.com/politicsHTTP/1.1 301 Moved Permanently...location: https://edition.cnn.com/politics...HTTP/1.1 200 OKContent-Type: text/html; charset=utf-8...Content-Length: 151654...
The output should be the same as in theTLS Origination for Egress Traffic example,without TLS origination.
- Create an egress
Gatewayfor edition.cnn.com, port 80, and a destination rule fortraffic directed to the egress gateway.
Choose the instructions corresponding to whether or not you havemutual TLS Authentication enabled in Istio.
$ kubectl apply -f - <<EOFapiVersion: networking.istio.io/v1alpha3kind: Gatewaymetadata:name: istio-egressgatewayspec:selector:istio: egressgatewayservers:- port:number: 80name: httpsprotocol: HTTPShosts:- edition.cnn.comtls:mode: MUTUALserverCertificate: /etc/certs/cert-chain.pemprivateKey: /etc/certs/key.pemcaCertificates: /etc/certs/root-cert.pem---apiVersion: networking.istio.io/v1alpha3kind: DestinationRulemetadata:name: egressgateway-for-cnnspec:host: istio-egressgateway.istio-system.svc.cluster.localsubsets:- name: cnntrafficPolicy:loadBalancer:simple: ROUND_ROBINportLevelSettings:- port:number: 80tls:mode: ISTIO_MUTUALsni: edition.cnn.comEOF
$ kubectl apply -f - <<EOFapiVersion: networking.istio.io/v1alpha3kind: Gatewaymetadata:name: istio-egressgatewayspec:selector:istio: egressgatewayservers:- port:number: 80name: httpprotocol: HTTPhosts:- edition.cnn.com---apiVersion: networking.istio.io/v1alpha3kind: DestinationRulemetadata:name: egressgateway-for-cnnspec:host: istio-egressgateway.istio-system.svc.cluster.localsubsets:- name: cnnEOF
- Define a
VirtualServiceto direct traffic from the sidecars to the egress gateway and from the egress gatewayto the external service:
$ kubectl apply -f - <<EOFapiVersion: networking.istio.io/v1alpha3kind: VirtualServicemetadata:name: direct-cnn-through-egress-gatewayspec:hosts:- edition.cnn.comgateways:- istio-egressgateway- meshhttp:- match:- gateways:- meshport: 80route:- destination:host: istio-egressgateway.istio-system.svc.cluster.localsubset: cnnport:number: 80weight: 100- match:- gateways:- istio-egressgatewayport: 80route:- destination:host: edition.cnn.comport:number: 80weight: 100EOF
- Resend the HTTP request to http://edition.cnn.com/politics.
$ kubectl exec -it $SOURCE_POD -c sleep -- curl -sL -o /dev/null -D - http://edition.cnn.com/politicsHTTP/1.1 301 Moved Permanently...location: https://edition.cnn.com/politics...HTTP/1.1 200 OKContent-Type: text/html; charset=utf-8...Content-Length: 151654...
The output should be the same as in the step 2.
- Check the log of the
istio-egressgatewaypod for a line corresponding to our request.If Istio is deployed in theistio-systemnamespace, the command to print the log is:
$ kubectl logs -l istio=egressgateway -c istio-proxy -n istio-system | tail
You should see a line similar to the following:
[2019-09-03T20:57:49.103Z] "GET /politics HTTP/2" 301 - "-" "-" 0 0 90 89 "10.244.2.10" "curl/7.64.0" "ea379962-9b5c-4431-ab66-f01994f5a5a5" "edition.cnn.com" "151.101.65.67:80" outbound|80||edition.cnn.com - 10.244.1.5:80 10.244.2.10:50482 edition.cnn.com -
Note that you only redirected the traffic from port 80 to the egress gateway. The HTTPS traffic to port 443went directly to edition.cnn.com.
Cleanup HTTP gateway
Remove the previous definitions before proceeding to the next step:
$ kubectl delete gateway istio-egressgateway$ kubectl delete serviceentry cnn$ kubectl delete virtualservice direct-cnn-through-egress-gateway$ kubectl delete destinationrule egressgateway-for-cnn
Egress gateway for HTTPS traffic
In this section you direct HTTPS traffic (TLS originated by the application) through an egress gateway.You need to specify port 443 with protocol TLS in a corresponding ServiceEntry, an egress Gateway and a VirtualService.
- Define a
ServiceEntryforedition.cnn.com:
$ kubectl apply -f - <<EOFapiVersion: networking.istio.io/v1alpha3kind: ServiceEntrymetadata:name: cnnspec:hosts:- edition.cnn.comports:- number: 443name: tlsprotocol: TLSresolution: DNSEOF
- Verify that your
ServiceEntrywas applied correctly by sending an HTTPS request to https://edition.cnn.com/politics.
$ kubectl exec -it $SOURCE_POD -c sleep -- curl -sL -o /dev/null -D - https://edition.cnn.com/politicsHTTP/1.1 200 OKContent-Type: text/html; charset=utf-8...Content-Length: 151654...
- Create an egress
Gatewayfor edition.cnn.com, a destination rule and a virtual serviceto direct the traffic through the egress gateway and from the egress gateway to the external service.
Choose the instructions corresponding to whether or not you havemutual TLS Authentication enabled in Istio.
$ kubectl apply -f - <<EOFapiVersion: networking.istio.io/v1alpha3kind: Gatewaymetadata:name: istio-egressgatewayspec:selector:istio: egressgatewayservers:- port:number: 443name: tls-cnnprotocol: TLShosts:- edition.cnn.comtls:mode: MUTUALserverCertificate: /etc/certs/cert-chain.pemprivateKey: /etc/certs/key.pemcaCertificates: /etc/certs/root-cert.pem---apiVersion: networking.istio.io/v1alpha3kind: DestinationRulemetadata:name: egressgateway-for-cnnspec:host: istio-egressgateway.istio-system.svc.cluster.localsubsets:- name: cnntrafficPolicy:loadBalancer:simple: ROUND_ROBINportLevelSettings:- port:number: 443tls:mode: ISTIO_MUTUALsni: edition.cnn.com---apiVersion: networking.istio.io/v1alpha3kind: VirtualServicemetadata:name: direct-cnn-through-egress-gatewayspec:hosts:- edition.cnn.comgateways:- mesh- istio-egressgatewaytls:- match:- gateways:- meshport: 443sni_hosts:- edition.cnn.comroute:- destination:host: istio-egressgateway.istio-system.svc.cluster.localsubset: cnnport:number: 443tcp:- match:- gateways:- istio-egressgatewayport: 443route:- destination:host: edition.cnn.comport:number: 443weight: 100EOF
$ kubectl apply -f - <<EOFapiVersion: networking.istio.io/v1alpha3kind: Gatewaymetadata:name: istio-egressgatewayspec:selector:istio: egressgatewayservers:- port:number: 443name: tlsprotocol: TLShosts:- edition.cnn.comtls:mode: PASSTHROUGH---apiVersion: networking.istio.io/v1alpha3kind: DestinationRulemetadata:name: egressgateway-for-cnnspec:host: istio-egressgateway.istio-system.svc.cluster.localsubsets:- name: cnn---apiVersion: networking.istio.io/v1alpha3kind: VirtualServicemetadata:name: direct-cnn-through-egress-gatewayspec:hosts:- edition.cnn.comgateways:- mesh- istio-egressgatewaytls:- match:- gateways:- meshport: 443sni_hosts:- edition.cnn.comroute:- destination:host: istio-egressgateway.istio-system.svc.cluster.localsubset: cnnport:number: 443- match:- gateways:- istio-egressgatewayport: 443sni_hosts:- edition.cnn.comroute:- destination:host: edition.cnn.comport:number: 443weight: 100EOF
- Send an HTTPS request to https://edition.cnn.com/politics.The output should be the same as before.
$ kubectl exec -it $SOURCE_POD -c sleep -- curl -sL -o /dev/null -D - https://edition.cnn.com/politicsHTTP/1.1 200 OKContent-Type: text/html; charset=utf-8...Content-Length: 151654...
- Check the log of the egress gateway’s proxy. If Istio is deployed in the
istio-systemnamespace, the command toprint the log is:
$ kubectl logs -l istio=egressgateway -n istio-system
You should see a line similar to the following:
[2019-01-02T11:46:46.981Z] "- - -" 0 - 627 1879689 44 - "-" "-" "-" "-" "151.101.129.67:443" outbound|443||edition.cnn.com 172.30.109.80:41122 172.30.109.80:443 172.30.109.112:59970 edition.cnn.com
Cleanup HTTPS gateway
$ kubectl delete serviceentry cnn$ kubectl delete gateway istio-egressgateway$ kubectl delete virtualservice direct-cnn-through-egress-gateway$ kubectl delete destinationrule egressgateway-for-cnn
Additional security considerations
Note that defining an egress Gateway in Istio does not in itself provides any special treatment for the nodeson which the egress gateway service runs. It is up to the cluster administrator or the cloud provider to deploythe egress gateways on dedicated nodes and to introduce additional security measures to make these nodes moresecure than the rest of the mesh.
Istio cannot securely enforce that all egress traffic actually flows through the egress gateways. Istio onlyenables such flow through its sidecar proxies. If attackers bypass the sidecar proxy, they could directly accessexternal services without traversing the egress gateway. Thus, the attackers escape Istio’s control and monitoring.The cluster administrator or the cloud provider must ensure that no traffic leaves the mesh bypassing the egressgateway. Mechanisms external to Istio must enforce this requirement. For example, the cluster administratorcan configure a firewall to deny all traffic not coming from the egress gateway.The Kubernetes network policies canalso forbid all the egress traffic not originating from the egress gateway (seethe next section for an example).Additionally, the cluster administrator or the cloud provider can configure the network to ensure application nodes canonly access the Internet via a gateway. To do this, the cluster administrator or the cloud provider can prevent theallocation of public IPs to pods other than gateways and can configure NAT devices to drop packets not originating atthe egress gateways.
Apply Kubernetes network policies
This section shows you how to create aKubernetes network policy to preventbypassing of the egress gateway. To test the network policy, you create a namespace, test-egress, deploythe sleep sample to it, and then attempt to send requests to a gateway-securedexternal service.
Follow the steps in theEgress gateway for HTTPS traffic section.
Create the
test-egressnamespace:
$ kubectl create namespace test-egress
- Deploy the sleep sample to the
test-egressnamespace.
Zip
$ kubectl apply -n test-egress -f @samples/sleep/sleep.yaml@
- Check that the deployed pod has a single container with no Istio sidecar attached:
$ kubectl get pod $(kubectl get pod -n test-egress -l app=sleep -o jsonpath={.items..metadata.name}) -n test-egressNAME READY STATUS RESTARTS AGEsleep-776b7bcdcd-z7mc4 1/1 Running 0 18m
- Send an HTTPS request to https://edition.cnn.com/politics from the
sleeppod inthetest-egressnamespace. The request will succeed since you did not define any restrictive policies yet.
$ kubectl exec -it $(kubectl get pod -n test-egress -l app=sleep -o jsonpath={.items..metadata.name}) -n test-egress -c sleep -- curl -s -o /dev/null -w "%{http_code}\n" https://edition.cnn.com/politics200
- Label the namespaces where the Istio components (the control plane and the gateways) run.If you deployed the Istio components to
istio-system, the command is:
$ kubectl label namespace istio-system istio=system
- Label the
kube-systemnamespace.
$ kubectl label ns kube-system kube-system=true
- Define a
NetworkPolicyto limit the egress traffic from thetest-egressnamespace to traffic destined toistio-system, and to thekube-systemDNS service (port 53):
$ cat <<EOF | kubectl apply -n test-egress -f -apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:name: allow-egress-to-istio-system-and-kube-dnsspec:podSelector: {}policyTypes:- Egressegress:- to:- namespaceSelector:matchLabels:kube-system: "true"ports:- protocol: UDPport: 53- to:- namespaceSelector:matchLabels:istio: systemEOF
- Resend the previous HTTPS request to https://edition.cnn.com/politics. Now itshould fail since the traffic is blocked by the network policy. Note that the
sleeppod cannot bypassistio-egressgateway. The only way it can accessedition.cnn.comis by using an Istio sidecar proxy and bydirecting the traffic toistio-egressgateway. This setting demonstrates that even if some malicious pod manages tobypass its sidecar proxy, it will not be able to access external sites and will be blocked by the network policy.
$ kubectl exec -it $(kubectl get pod -n test-egress -l app=sleep -o jsonpath={.items..metadata.name}) -n test-egress -c sleep -- curl -v https://edition.cnn.com/politicsHostname was NOT found in DNS cacheTrying 151.101.65.67...Trying 2a04:4e42:200::323...Immediate connect fail for 2a04:4e42:200::323: Cannot assign requested addressTrying 2a04:4e42:400::323...Immediate connect fail for 2a04:4e42:400::323: Cannot assign requested addressTrying 2a04:4e42:600::323...Immediate connect fail for 2a04:4e42:600::323: Cannot assign requested addressTrying 2a04:4e42::323...Immediate connect fail for 2a04:4e42::323: Cannot assign requested addressconnect to 151.101.65.67 port 443 failed: Connection timed out
- Now inject an Istio sidecar proxy into the
sleeppod in thetest-egressnamespace by first enablingautomatic sidecar proxy injection in thetest-egressnamespace:
$ kubectl label namespace test-egress istio-injection=enabled
- Then redeploy the
sleepdeployment:
Zip
$ kubectl delete deployment sleep -n test-egress$ kubectl apply -f @samples/sleep/sleep.yaml@ -n test-egress
- Check that the deployed pod has two containers, including the Istio sidecar proxy (
istio-proxy):
$ kubectl get pod $(kubectl get pod -n test-egress -l app=sleep -o jsonpath={.items..metadata.name}) -n test-egress -o jsonpath='{.spec.containers[*].name}'sleep istio-proxy
- Create the same destination rule as for the
sleeppod in thedefaultnamespace to direct the traffic through the egress gateway:
Choose the instructions corresponding to whether or not you havemutual TLS Authentication enabled in Istio.
$ kubectl apply -n test-egress -f - <<EOFapiVersion: networking.istio.io/v1alpha3kind: DestinationRulemetadata:name: egressgateway-for-cnnspec:host: istio-egressgateway.istio-system.svc.cluster.localsubsets:- name: cnntrafficPolicy:loadBalancer:simple: ROUND_ROBINportLevelSettings:- port:number: 443tls:mode: ISTIO_MUTUALsni: edition.cnn.comEOF
$ kubectl apply -n test-egress -f - <<EOFapiVersion: networking.istio.io/v1alpha3kind: DestinationRulemetadata:name: egressgateway-for-cnnspec:host: istio-egressgateway.istio-system.svc.cluster.localsubsets:- name: cnnEOF
- Send an HTTPS request to https://edition.cnn.com/politics. Now it should succeedsince the traffic flows to
istio-egressgatewayin theistio-systemnamespace, which is allowed by theNetwork Policy you defined.istio-egressgatewayforwards the traffic toedition.cnn.com.
$ kubectl exec -it $(kubectl get pod -n test-egress -l app=sleep -o jsonpath={.items..metadata.name}) -n test-egress -c sleep -- curl -s -o /dev/null -w "%{http_code}\n" https://edition.cnn.com/politics200
- Check the statistics of the egress gateway’s proxy and see a counter that corresponds to ourrequests to edition.cnn.com. If Istio is deployed in the
istio-systemnamespace, the command to print thecounter is:
$ kubectl exec $(kubectl get pod -l istio=egressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -c istio-proxy -n istio-system -- pilot-agent request GET stats | grep edition.cnn.com.upstream_cx_totalcluster.outbound|443||edition.cnn.com.upstream_cx_total: 2
Cleanup network policies
- Delete the resources created in this section:
Zip
$ kubectl delete -f @samples/sleep/sleep.yaml@ -n test-egress$ kubectl delete destinationrule egressgateway-for-cnn -n test-egress$ kubectl delete networkpolicy allow-egress-to-istio-system-and-kube-dns -n test-egress$ kubectl label namespace kube-system kube-system-$ kubectl label namespace istio-system istio-$ kubectl delete namespace test-egress
- Follow the steps in the Cleanup HTTPS gateway section.
Troubleshooting
Check if you have mutual TLS Authentication enabled in Istio, following thesteps in Verify mutual TLS configuration.If mutual TLS is enabled, make sure you create the configurationitems accordingly (note the remarks If you have mutual TLS Authentication enabled in Istio, you must create…).
If mutual TLS Authentication is enabled, verify the correct certificate of theegress gateway:
$ kubectl exec -i -n istio-system $(kubectl get pod -l istio=egressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -- cat /etc/certs/cert-chain.pem | openssl x509 -text -noout | grep 'Subject Alternative Name' -A 1X509v3 Subject Alternative Name:URI:spiffe://cluster.local/ns/istio-system/sa/istio-egressgateway-service-account
- For HTTPS traffic (TLS originated by the application), test the traffic flow by using the openssl command.openssl has an explicit option for setting the SNI, namely
-servername.
$ kubectl exec -it $SOURCE_POD -c sleep -- openssl s_client -connect edition.cnn.com:443 -servername edition.cnn.comCONNECTED(00000003)...Certificate chain0 s:/C=US/ST=California/L=San Francisco/O=Fastly, Inc./CN=turner-tls.map.fastly.neti:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign CloudSSL CA - SHA256 - G31 s:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign CloudSSL CA - SHA256 - G3i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA---Server certificate-----BEGIN CERTIFICATE-----...
If you get the certificate as in the output above, your traffic is routed correctly. Check the statistics of the egress gateway’s proxy and see a counter that corresponds to your requests (sent by openssl and curl) to edition.cnn.com.
$ kubectl exec $(kubectl get pod -l istio=egressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -c istio-proxy -n istio-system -- pilot-agent request GET stats | grep edition.cnn.com.upstream_cx_totalcluster.outbound|443||edition.cnn.com.upstream_cx_total: 2
Cleanup
Shutdown the sleep service:
Zip
$ kubectl delete -f @samples/sleep/sleep.yaml@
See also
Secure Control of Egress Traffic in Istio, part 3
Comparison of alternative solutions to control egress traffic including performance considerations.
Secure Control of Egress Traffic in Istio, part 2
Use Istio Egress Traffic Control to prevent attacks involving egress traffic.
Secure Control of Egress Traffic in Istio, part 1
Attacks involving egress traffic and requirements for egress traffic control.
Egress Gateway Performance Investigation
Verifies the performance impact of adding an egress gateway.
Consuming External MongoDB Services
Describes a simple scenario based on Istio's Bookinfo example.
Monitoring and Access Policies for HTTP Egress Traffic
Describes how to configure Istio for monitoring and access policies of HTTP egress traffic.
