Use Vault-CRD with NGINX Ingress Controller Part I
Last week I announced my OpenSource project Vault-CRD to share secrets that are Stored in HashiCorp Vault with Kubernetes.
Here you can find the corresponding Blog post: Vault-CRD
In this Blog post I’ll show you how to use the Vault-CRD to dynamically update NGINX Ingress Controller if the certificate changes in Vault.
The Blog post is splitted into two parts.
- Part I: The first part will show you how to use a PKI Secret Engine to generate NGINX Ingress Controller certificates.
- Part II: The second part of this tutorial will show you how to setup and use Key-Value type for certificates that NGINX Ingress Controller can use
Setup
For a simple setup I’ve deployed a NGINX container in Kubernetes:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: nginx
name: nginx
namespace: vault-crd
spec:
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:alpine
ports:
- name: http
containerPort: 80
restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
labels:
app: nginx
name: nginx
namespace: vault-crd
spec:
selector:
app: nginx
type: ClusterIP
ports:
- port: 80
protocol: TCP
targetPort: 80
As you can see it’s a nginx:alpine-image that will be deployed as a deployment with 1 replication and a service that is connected via the selector to it. The Ingress for exposing this container to the world is also very easy:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
name: ingress
namespace: vault-crd
spec:
rules:
- host: test-url.koudingspawn.de
http:
paths:
- backend:
serviceName: nginx
servicePort: 80
path: /
tls:
- secretName: ingress-ssl-certificate
hosts:
- test-url.koudingspawn.de
Ingress will be configured to watch for test-url.koudingspawn.de Hosts in HTTP request and will proxy requests to the nginx service on port 80.
Now the interesting part is the tls section inside the ingress definition. If you apply this now to Kubernetes the Ingress controller will fail with an exception, that there is no secret secret/ingress-ssl-certificate:
W0505 18:14:02.283761 5 backend_ssl.go:48] error obtaining PEM from secret vault-crd/ingress-ssl-certificate: error retrieving secret vault-crd/ingress-ssl-certificate: secret vault-crd/ingress-ssl-certificate was not found
Now comes the interresting part of Vault-CRD.
Share the Certificate with Kubernetes via PKI
In Vault there is a build in Secret Type PKI. This one is a complete certificate authority we can use to generate self signed certificates. For more information how to generate a PKI please read the instructions from HashiCorp Vault. Or you can use the simple one I’ve created for this example:
$ mountPath="testpki"
$ vault secrets enable -path=$mountPath -description=$mountPath pki
$ vault secrets tune -max-lease-ttl=8760h $mountPath
$ vault write $mountPath/root/generate/internal \
common_name=$mountPath \
ttl=8760h
$ vault write $mountPath/roles/testrole \
allow_any_name=true \
max_ttl=8760h
Now we can generate our Vault-CRD to automatically issue certificates:
apiVersion: "koudingspawn.de/v1"
kind: Vault
metadata:
name: ingress-ssl-certificate
namespace: vault-crd
spec:
path: "testpki/issue/testrole"
type: "PKI"
pkiConfiguration:
commonName: "test-url.koudingspawn.de"
ttl: "720h"
What will happen now
As you can see inside the Vault-CRD logs, that it detects a new Vault-resource in namespace vault-crd with name ingress-ssl-certificate.
Now Vault-CRD tries to access your Vault and issues a new certificate with common name test-url.koudingspawn.de and a time to life of 30 days (720h). After this it generates a Secret with the same name as it’s own, in the vault-crd namespace.
After this there is now a Secret that the Ingress Controller can use to expose your service to the world.
The interresting part is here the naming. I’ve named the ingress resource tls secret name with ingress-ssl-certificate and the Vault resource also ingress-ssl-certificate. And as already told Vault-CRD now generates a secret with the same name.
Now you’ll see that the URL can be accessed and has a specific fingerprint for the certificate.
$ openssl s_client -servername test-url.koudingspawn.de -connect test-url.koudingspawn.de:30443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin
SHA1 Fingerprint=62:98:DE:39:66:7C:A4:72:DF:FC:80:3A:71:27:3F:AD:E7:87:4A:98
Refresh of a certificate
If the 30 days are near to expire Vault-CRD will detect this and issues a new Certificate.
When a refresh happens NGINX will detect the changed secret and updates itself to use the new certificate:
I0505 18:35:02.732775 5 store.go:375] secret vault-crd/ingress-ssl-certificate was updated and it is used in ingress annotations. Parsing...
I0505 18:35:02.736128 5 backend_ssl.go:59] updating secret vault-crd/ingress-ssl-certificate in the local store
I0505 18:35:02.736653 5 controller.go:168] backend reload required
I0505 18:35:02.914816 5 controller.go:177] ingress backend successfully reloaded...
I0505 18:35:28.340035 5 backend_ssl.go:173] updating local copy of ssl certificate vault-crd/ingress-ssl-certificate with missing intermediate CA certs
I0505 18:35:28.340265 5 controller.go:168] backend reload required
I0505 18:35:28.479019 5 controller.go:177] ingress backend successfully reloaded...
In Part two of this tutorial I’ll show you how to use Vault-CRD with certificates that are not from a Vault PKI Secret Engine. Part II
