• Secure Gateways (SDS)
    • Before you begin
    • Generate client and server certificates and keys
    • Configure a TLS ingress gateway using SDS
      • Configure a TLS ingress gateway for a single host
      • Configure a TLS ingress gateway for multiple hosts
      • Configure a mutual TLS ingress gateway
    • Troubleshooting
    • Cleanup
    • See also

    Secure Gateways (SDS)

    The Control Ingress Traffic taskdescribes how to configure an ingress gateway to expose an HTTPservice to external traffic. This task shows how to expose a secure HTTPSservice using either simple or mutual TLS.

    The TLS required private key, server certificate, and root certificate, are configuredusing the Secret Discovery Service (SDS).

    Before you begin

    • Perform the steps in the Before you beginand Determining the ingress IP and portssections of the Control Ingress Traffic task. After performingthose steps you should have Istio and the httpbin service deployed,and the environment variables INGRESS_HOST and SECURE_INGRESS_PORT set.

    • For macOS users, verify that you use curl compiled with the LibreSSL library:

    1. $ curl --version | grep LibreSSL
    2. curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0

    If the previous command outputs a version of LibreSSL as shown, your curl commandshould work correctly with the instructions in this task. Otherwise, trya different implementation of curl, for example on a Linux machine.

    If you configured an ingress gateway using the file mount-based approach,and you want to migrate your ingress gateway to use the SDS approach, there are noextra steps required.

    Generate client and server certificates and keys

    For this task you can use your favorite tool to generate certificates and keys.This example uses a scriptfrom the https://github.com/nicholasjackson/mtls-go-example repository.

    • Clone the example’s repository:
    1. $ git clone https://github.com/nicholasjackson/mtls-go-example
    • Go to the cloned repository:
    1. $ pushd mtls-go-example
    • Generate the certificates for httpbin.example.com. Replace <password> withany value in the following command:
    1. $ ./generate.sh httpbin.example.com <password>

    When prompted, answer y to all the questions. The command generatesfour directories: 1_root, 2_intermediate, 3_application, and4_client containing the client and server certificates to use in theprocedures below.

    • Move the certificates into a directory named httpbin.example.com:
    1. $ mkdir ../httpbin.example.com && mv 1_root 2_intermediate 3_application 4_client ../httpbin.example.com
    • Go back to your previous directory:
    1. $ popd

    Configure a TLS ingress gateway using SDS

    You can configure a TLS ingress gateway to fetch credentialsfrom the ingress gateway agent via secret discovery service (SDS). The ingressgateway agent runs in the same pod as the ingress gateway and watches thecredentials created in the same namespace as the ingress gateway. Enabling SDSat ingress gateway brings the following benefits.

    • The ingress gateway can dynamically add, delete, or update itskey/certificate pairs and its root certificate. You do not have to restart the ingressgateway.

    • No secret volume mount is needed. Once you create a kubernetessecret, that secret is captured by the gateway agent and sent to ingress gatewayas key/certificate or root certificate.

    • The gateway agent can watch multiple key/certificate pairs. You onlyneed to create secrets for multiple hosts and update the gateway definitions.

    • Enable SDS at ingress gateway and deploy the ingress gateway agent.Since this feature is disabled by default, you need to enable theistio-ingressgateway.sds.enabled installation option and generate the istio-ingressgateway.yaml file:
    1. $ istioctl manifest generate \
    2. --set values.gateways.istio-egressgateway.enabled=false \
    3. --set values.gateways.istio-ingressgateway.sds.enabled=true > \
    4. $HOME/istio-ingressgateway.yaml
    5. $ kubectl apply -f $HOME/istio-ingressgateway.yaml
    • Set the environment variables INGRESS_HOST and SECURE_INGRESS_PORT:
    1. $ export SECURE_INGRESS_PORT=$(kubectl -n istio-system \
    2. get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="https")].port}')
    3. $ export INGRESS_HOST=$(kubectl -n istio-system \
    4. get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

    Configure a TLS ingress gateway for a single host

    • Start the httpbin sample:
    1. $ cat <<EOF | kubectl apply -f -
    2. apiVersion: v1
    3. kind: Service
    4. metadata:
    5.   name: httpbin
    6.   labels:
    7.     app: httpbin
    8. spec:
    9.   ports:
    10.   - name: http
    11.     port: 8000
    12.   selector:
    13.     app: httpbin
    14. ---
    15. apiVersion: apps/v1
    16. kind: Deployment
    17. metadata:
    18.   name: httpbin
    19. spec:
    20.   replicas: 1
    21.   selector:
    22.     matchLabels:
    23.       app: httpbin
    24.       version: v1
    25.   template:
    26.     metadata:
    27.       labels:
    28.         app: httpbin
    29.         version: v1
    30.     spec:
    31.       containers:
    32.       - image: docker.io/citizenstig/httpbin
    33.         imagePullPolicy: IfNotPresent
    34.         name: httpbin
    35.         ports:
    36.         - containerPort: 8000
    37. EOF
    • Create a secret for the ingress gateway:
    1. $ kubectl create -n istio-system secret generic httpbin-credential \
    2. --from-file=key=httpbin.example.com/3_application/private/httpbin.example.com.key.pem \
    3. --from-file=cert=httpbin.example.com/3_application/certs/httpbin.example.com.cert.pem

    The secret name should not begin with istio or prometheus, andthe secret should not contain a token field.

    • Define a gateway with a servers: section for port 443, and specify values forcredentialName to be httpbin-credential. The values are the same as thesecret’s name. The TLS mode should have the value of SIMPLE.
    1. $ cat <<EOF | kubectl apply -f -
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: Gateway
    4. metadata:
    5.   name: mygateway
    6. spec:
    7.   selector:
    8.     istio: ingressgateway # use istio default ingress gateway
    9.   servers:
    10.   - port:
    11.       number: 443
    12.       name: https
    13.       protocol: HTTPS
    14.     tls:
    15.       mode: SIMPLE
    16.       credentialName: "httpbin-credential" # must be the same as secret
    17.     hosts:
    18.     - "httpbin.example.com"
    19. EOF
    • Configure the gateway’s ingress traffic routes. Define the correspondingvirtual service.
    1. $ cat <<EOF | kubectl apply -f -
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: VirtualService
    4. metadata:
    5.   name: httpbin
    6. spec:
    7.   hosts:
    8.   - "httpbin.example.com"
    9.   gateways:
    10.   - mygateway
    11.   http:
    12.   - match:
    13.     - uri:
    14.         prefix: /status
    15.     - uri:
    16.         prefix: /delay
    17.     route:
    18.     - destination:
    19.         port:
    20.           number: 8000
    21.         host: httpbin
    22. EOF
    • Send an HTTPS request to access the httpbin service through HTTPS:
    1. $ curl -v -HHost:httpbin.example.com \
    2. --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
    3. --cacert httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem \
    4. https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418

    The httpbin service will return the418 I’m a Teapot code.

    • Delete the gateway’s secret and create a new one to change the ingressgateway’s credentials.
    1. $ kubectl -n istio-system delete secret httpbin-credential
    1. $ pushd mtls-go-example
    2. $ ./generate.sh httpbin.example.com <password>
    3. $ mkdir ../httpbin.new.example.com && mv 1_root 2_intermediate 3_application 4_client ../httpbin.new.example.com
    4. $ popd
    5. $ kubectl create -n istio-system secret generic httpbin-credential \
    6. --from-file=key=httpbin.new.example.com/3_application/private/httpbin.example.com.key.pem \
    7. --from-file=cert=httpbin.new.example.com/3_application/certs/httpbin.example.com.cert.pem
    • Access the httpbin service using curl
    1. $ curl -v -HHost:httpbin.example.com \
    2. --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
    3. --cacert httpbin.new.example.com/2_intermediate/certs/ca-chain.cert.pem \
    4. https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
    5. ...
    6. HTTP/2 418
    7. ...
    8. -=[ teapot ]=-
    10.    _...._
    11.  .'  _ _ `.
    12. | ."` ^ `". _,
    13. \_;`"---"`|//
    14.   |       ;/
    15.   \_     _/
    16.     `"""`
    • If you try to access httpbin with the previous certificate chain, the attempt now fails.
    1. $ curl -v -HHost:httpbin.example.com \
    2. --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
    3. --cacert httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem \
    4. https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
    5. ...
    6. * TLSv1.2 (OUT), TLS handshake, Client hello (1):
    7. * TLSv1.2 (IN), TLS handshake, Server hello (2):
    8. * TLSv1.2 (IN), TLS handshake, Certificate (11):
    9. * TLSv1.2 (OUT), TLS alert, Server hello (2):
    10. * SSL certificate problem: unable to get local issuer certificate

    Configure a TLS ingress gateway for multiple hosts

    You can configure an ingress gateway for multiple hosts,httpbin.example.com and helloworld-v1.example.com, for example. The ingress gatewayretrieves unique credentials corresponding to a specific credentialName.

    • To restore the credentials for httpbin, delete its secret and create it again.
    1. $ kubectl -n istio-system delete secret httpbin-credential
    2. $ kubectl create -n istio-system secret generic httpbin-credential \
    3. --from-file=key=httpbin.example.com/3_application/private/httpbin.example.com.key.pem \
    4. --from-file=cert=httpbin.example.com/3_application/certs/httpbin.example.com.cert.pem
    • Start the helloworld-v1 sample
    1. $ cat <<EOF | kubectl apply -f -
    2. apiVersion: v1
    3. kind: Service
    4. metadata:
    5.   name: helloworld-v1
    6.   labels:
    7.     app: helloworld-v1
    8. spec:
    9.   ports:
    10.   - name: http
    11.     port: 5000
    12.   selector:
    13.     app: helloworld-v1
    14. ---
    15. apiVersion: apps/v1
    16. kind: Deployment
    17. metadata:
    18.   name: helloworld-v1
    19. spec:
    20.   replicas: 1
    21.   selector:
    22.     matchLabels:
    23.       app: helloworld-v1
    24.       version: v1
    25.   template:
    26.     metadata:
    27.       labels:
    28.         app: helloworld-v1
    29.         version: v1
    30.     spec:
    31.       containers:
    32.       - name: helloworld
    33.         image: istio/examples-helloworld-v1
    34.         resources:
    35.           requests:
    36.             cpu: "100m"
    37.         imagePullPolicy: IfNotPresent #Always
    38.         ports:
    39.         - containerPort: 5000
    40. EOF
    • Create a secret for the ingress gateway. If you created the httpbin-credentialsecret already, you can now create the helloworld-credential secret.
    1. $ pushd mtls-go-example
    2. $ ./generate.sh helloworld-v1.example.com <password>
    3. $ mkdir ../helloworld-v1.example.com && mv 1_root 2_intermediate 3_application 4_client ../helloworld-v1.example.com
    4. $ popd
    5. $ kubectl create -n istio-system secret generic helloworld-credential \
    6. --from-file=key=helloworld-v1.example.com/3_application/private/helloworld-v1.example.com.key.pem \
    7. --from-file=cert=helloworld-v1.example.com/3_application/certs/helloworld-v1.example.com.cert.pem
    • Define a gateway with two server sections for port 443. Set the value ofcredentialName on each port to httpbin-credential and helloworld-credentialrespectively. Set TLS mode to SIMPLE.
    1. $ cat <<EOF | kubectl apply -f -
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: Gateway
    4. metadata:
    5.   name: mygateway
    6. spec:
    7.   selector:
    8.     istio: ingressgateway # use istio default ingress gateway
    9.   servers:
    10.   - port:
    11.       number: 443
    12.       name: https-httpbin
    13.       protocol: HTTPS
    14.     tls:
    15.       mode: SIMPLE
    16.       credentialName: "httpbin-credential"
    17.     hosts:
    18.     - "httpbin.example.com"
    19.   - port:
    20.       number: 443
    21.       name: https-helloworld
    22.       protocol: HTTPS
    23.     tls:
    24.       mode: SIMPLE
    25.       credentialName: "helloworld-credential"
    26.     hosts:
    27.     - "helloworld-v1.example.com"
    28. EOF
    • Configure the gateway’s traffic routes. Define the correspondingvirtual service.
    1. $ cat <<EOF | kubectl apply -f -
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: VirtualService
    4. metadata:
    5.   name: helloworld-v1
    6. spec:
    7.   hosts:
    8.   - "helloworld-v1.example.com"
    9.   gateways:
    10.   - mygateway
    11.   http:
    12.   - match:
    13.     - uri:
    14.         exact: /hello
    15.     route:
    16.     - destination:
    17.         host: helloworld-v1
    18.         port:
    19.           number: 5000
    20. EOF
    • Send an HTTPS request to helloworld-v1.example.com:
    1. $ curl -v -HHost:helloworld-v1.example.com \
    2. --resolve helloworld-v1.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
    3. --cacert helloworld-v1.example.com/2_intermediate/certs/ca-chain.cert.pem \
    4. https://helloworld-v1.example.com:$SECURE_INGRESS_PORT/hello
    5. HTTP/2 200
    • Send an HTTPS request to httpbin.example.com and still get a teapot in return:
    1. $ curl -v -HHost:httpbin.example.com \
    2. --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
    3. --cacert httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem \
    4. https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
    5.     -=[ teapot ]=-
    7.        _...._
    8.      .'  _ _ `.
    9.     | ."` ^ `". _,
    10.     \_;`"---"`|//
    11.       |       ;/
    12.       \_     _/
    13.         `"""`

    Configure a mutual TLS ingress gateway

    You can extend your gateway’s definition to supportmutual TLS. Changethe credentials of the ingress gateway by deleting its secret and creating a new one.The server uses the CA certificate to verifyits clients, and we must use the name cacert to hold the CA certificate.

    1. $ kubectl -n istio-system delete secret httpbin-credential
    2. $ kubectl create -n istio-system secret generic httpbin-credential  \
    3. --from-file=key=httpbin.example.com/3_application/private/httpbin.example.com.key.pem \
    4. --from-file=cert=httpbin.example.com/3_application/certs/httpbin.example.com.cert.pem \
    5. --from-file=cacert=httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem
    • Change the gateway’s definition to set the TLS mode to MUTUAL.
    1. $ cat <<EOF | kubectl apply -f -
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: Gateway
    4. metadata:
    5.  name: mygateway
    6. spec:
    7.  selector:
    8.    istio: ingressgateway # use istio default ingress gateway
    9.  servers:
    10.  - port:
    11.      number: 443
    12.      name: https
    13.      protocol: HTTPS
    14.    tls:
    15.      mode: MUTUAL
    16.      credentialName: "httpbin-credential" # must be the same as secret
    17.    hosts:
    18.    - "httpbin.example.com"
    19. EOF
    • Attempt to send an HTTPS request using the prior approach and see how it fails:
    1. $ curl -v -HHost:httpbin.example.com \
    2. --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
    3. --cacert httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem \
    4. https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
    5. * TLSv1.3 (OUT), TLS handshake, Client hello (1):
    6. * TLSv1.3 (IN), TLS handshake, Server hello (2):
    7. * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
    8. * TLSv1.3 (IN), TLS handshake, Request CERT (13):
    9. * TLSv1.3 (IN), TLS handshake, Certificate (11):
    10. * TLSv1.3 (IN), TLS handshake, CERT verify (15):
    11. * TLSv1.3 (IN), TLS handshake, Finished (20):
    12. * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
    13. * TLSv1.3 (OUT), TLS handshake, Certificate (11):
    14. * TLSv1.3 (OUT), TLS handshake, Finished (20):
    15. * TLSv1.3 (IN), TLS alert, unknown (628):
    16. * OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0
    • Pass a client certificate and private key to curl and resend the request.Pass your client’s certificate with the —cert flag and your private keywith the —key flag to curl.
    1. $ curl -v -HHost:httpbin.example.com \
    2. --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
    3. --cacert httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem \
    4. --cert httpbin.example.com/4_client/certs/httpbin.example.com.cert.pem \
    5. --key httpbin.example.com/4_client/private/httpbin.example.com.key.pem \
    6. https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
    8.     -=[ teapot ]=-
    10.        _...._
    11.      .'  _ _ `.
    12.     | ."` ^ `". _,
    13.     \_;`"---"`|//
    14.       |       ;/
    15.       \_     _/

    • Instead of creating a httpbin-credential secret to hold all the credentials, you cancreate two separate secrets:

      • httpbin-credential holds the server’s key and certificate
      • httpbin-credential-cacert holds the client’s CA certificate and must have the -cacert suffixCreate the two separate secrets with the following commands:
    1. $ kubectl -n istio-system delete secret httpbin-credential
    2. $ kubectl create -n istio-system secret generic httpbin-credential  \
    3. --from-file=key=httpbin.example.com/3_application/private/httpbin.example.com.key.pem \
    4. --from-file=cert=httpbin.example.com/3_application/certs/httpbin.example.com.cert.pem
    5. $ kubectl create -n istio-system secret generic httpbin-credential-cacert  \
    6. --from-file=cacert=httpbin.example.com/2_intermediate/certs/ca-chain.cert.pem

    Troubleshooting

    • Inspect the values of the INGRESS_HOST and SECURE_INGRESS_PORT environmentvariables. Make sure they have valid values, according to the output of thefollowing commands:
    1. $ kubectl get svc -n istio-system
    2. $ echo INGRESS_HOST=$INGRESS_HOST, SECURE_INGRESS_PORT=$SECURE_INGRESS_PORT
    • Check the log of the istio-ingressgateway controller for error messages:
    1. $ kubectl logs -n istio-system $(kubectl get pod -l istio=ingressgateway \
    2. -n istio-system -o jsonpath='{.items[0].metadata.name}') -c istio-proxy

    • If using macOS, verify you are using curl compiled with the LibreSSLlibrary, as described in the Before you begin section.

    • Verify that the secrets are successfully created in the istio-systemnamespace:

    1. $ kubectl -n istio-system get secrets

    httpbin-credential and helloworld-credential should show in the secretslist.

    • Check the logs to verify that the ingress gateway agent has pushed thekey/certificate pair to the ingress gateway.
    1. $ kubectl logs -n istio-system $(kubectl get pod -l istio=ingressgateway \
    2. -n istio-system -o jsonpath='{.items[0].metadata.name}') -c ingress-sds

    The log should show that the httpbin-credential secret was added. If using mutualTLS, then the httpbin-credential-cacert secret should also appear.Verify the log shows that the gateway agent receives SDS requests from theingress gateway, that the resource’s name is httpbin-credential, and that the ingress gatewayobtained the key/certificate pair. If using mutual TLS, the log should showkey/certificate was sent to the ingress gateway,that the gateway agent received the SDS request with the httpbin-credential-cacertresource name, and that the ingress gateway obtained the root certificate.

    Cleanup

    • Delete the gateway configuration, the virtual service definition, and the secrets:
    1. $ kubectl delete gateway mygateway
    2. $ kubectl delete virtualservice httpbin
    3. $ kubectl delete --ignore-not-found=true -n istio-system secret httpbin-credential \
    4. helloworld-credential
    5. $ kubectl delete --ignore-not-found=true virtualservice helloworld-v1
    • Delete the directories of the certificates and the repository used to generate them:
    1. $ rm -rf httpbin.example.com helloworld-v1.example.com mtls-go-example
    • Remove the file you used for redeployment of the ingress gateway.
    1. $ rm -f $HOME/istio-ingressgateway.yaml
    • Shutdown the httpbin and helloworld-v1 services:
    1. $ kubectl delete service --ignore-not-found=true helloworld-v1
    2. $ kubectl delete service --ignore-not-found=true httpbin

    See also

    Istio as a Proxy for External Services

    Configure Istio ingress gateway to act as a proxy for external services.

    Deploy a Custom Ingress Gateway Using Cert-Manager

    Describes how to deploy a custom ingress gateway using cert-manager manually.

    Configuring Istio Ingress with AWS NLB

    Describes how to configure Istio ingress with a network load balancer on AWS.

    Ingress Gateway without TLS Termination

    Describes how to configure SNI passthrough for an ingress gateway.

    Ingress Gateways

    Describes how to configure an Istio gateway to expose a service outside of the service mesh.

    Kubernetes Ingress with Cert-Manager

    Demonstrates how to obtain Let's Encrypt TLS certificates for Kubernetes Ingress automatically using Cert-Manager.