Setup an etcd cluster
etcd is an distributed key value store developed and maintained by the CoreOS team. It is used for example by Kubernetes to store the cluster state in it. When operating a Kubernetes cluster, it is therefore important that the etcd is highly available in order to remain accessible in case of failure of individual nodes. In case of network splitting it is very important to have an odd number of members in a cluster. In case of failures the cluster is more tolerant against failures because the members can keep the quorum about the state inside the etcd.
So in our case we will deploy an etcd cluster with three nodes. It is also recommended to operate the cluster in multiple geographically separated locations in this tutorial we will deploy the cluster only in region Frankfurt (fra1).
The most difficult part of this is to tell the individual nodes what other nodes exist, so that they can find themselves in a cluster. In our example we will use a simple approach by adding an read only DigitalOcean API key to the nodes, so they can contact the API to ask for droplets with the tag etcd.
The first step is to write a short shell script to list all Droplets with the tag etcd:
do_response=$(curl -X GET -H "Content-Type: application/json" -H "Authorization: Bearer $DO_TOKEN" "https://api.digitalocean.com/v2/droplets?tag_name=etcd")
After you received them from the DigitalOcean API we will receive the etcd members via jq, a command line JSON processor:
etcd_member_hosts=$(echo ${do_response} | jq '.droplets[].name' -r)
Now in the variable etcd_member_hosts the droplets are listed by name and we can iterate over them to build the list of etcd members:
for member_host in $etcd_member_hosts; do
ETCD_INITIAL_CLUSTER="${ETCD_INITIAL_CLUSTER},${member_host}=https://${member_host}:2380"
done
It is important to note, that our DigitalOcean Droplets are named like: etcd-server-
ETCD_INITIAL_CLUSTER="etcd-server-0.koudingspawn.de=https://etcd-server-0.koudingspawn.de:2380,etcd-server-1.koudingspawn.de=https://etcd-server-1.koudingspawn.de:2380,etcd-server-2.koudingspawn.de=https://etcd-server-2.koudingspawn.de:2380,
This tells the cluster, that there are three members in the cluster, additionally you have to define your own ETCD_NAME. This ETCD_NAME must be the same as one of the members in the ETCD_INITIAL_CLUSTER:
ETCD_NAME=etcd-server-0.koudingspawn.de
It must be specified under which address the etcd cluster listens for clients and peers to communicate with it, therefore you have to define thet ETCD_LISTEN_CLIENT_URLS, ETCD_LISTEN_PEER_URLS, ETCD_ADVERTISE_CLIENT_URLS, ETCD_ADVERTISE_PEER_URLS and ETCD_INITIAL_ADVERTISE_PEER_URLS. The LISTEN_URLS are for listening and the ADVERTISE_URLS are to tell other peers where they can access this member.
ETCD_ADVERTISE_CLIENT_URLS="https://etcd-server-0.koudingspawn.de:2379"
ETCD_ADVERTISE_PEER_URLS="https://etcd-server-0.koudingspawn.de:2380"
ETCD_LISTEN_CLIENT_URLS="https://<public-ip>:2379,https://127.0.1.1:2379"
ETCD_LISTEN_PEER_URLS="https://<public-ip>:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://etcd-server-0.koudingspawn.de:2380
Maybe you have already seen, that the endpoints are TLS secured via https. So you have to generate self signed certificates for peer and client to server communication and one certificate for authentication of clients. Therefore please follow the instructions available via: https://coreos.com/os/docs/latest/generate-self-signed-certificates.html
Here are some snippets out of my certificates (client, server, peer):
// client:
...
"CN": "etcd client",
"hosts": [""],
...
// server:
...
"CN": "etcd server",
"hosts": [
"*.koudingspawn.de"
],
...
// peer:
...
"CN": "etcd peer",
"hosts": [
"*.koudingspawn.de"
],
...
Now you can copy these certificates to a directory on your server, like /etc/etcd and add them as variables to our script to dynamically append nodes to the etcd cluster:
ETCD_CERT_FILE="/etc/etcd/server.pem"
ETCD_KEY_FILE="/etc/etcd/server-key.pem"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ca.pem"
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_CERT_FILE="/etc/etcd/peer.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/peer-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ca.pem"
Now the endpoints are secured via TLS and client certificate authentication is enabled.
The complete script looks like this:
#!/bin/bash
# used network interface
PUBLIC_NETWORK_INTERFACE="eth0"
do_response=$(curl -X GET -H "Content-Type: application/json" -H "Authorization: Bearer $do_token" "https://api.digitalocean.com/v2/droplets?tag_name=etcd")
## collect list of all etcd members
etcd_member_hosts=$(echo ${do_response} | jq '.droplets[].name' -r)
ETCD_INITIAL_CLUSTER=''
echo $etcd_member_hosts
# calculate initial cluster list
for member_host in $etcd_member_hosts; do
ETCD_INITIAL_CLUSTER="${ETCD_INITIAL_CLUSTER},${member_host}=https://${member_host}:2380"
done
# get instance url and peer name that is ip with underscores in int
public_peer_hostname="$(hostname).koudingspawn.de"
public_peer_ip=$(ifconfig $PUBLIC_NETWORK_INTERFACE | grep 'inet addr' | cut -d: -f2 | awk '{print $1}')
peer_name=$(hostname)
# remove leading ,
ETCD_INITIAL_CLUSTER=$(echo $ETCD_INITIAL_CLUSTER | cut -c 2-)
## List of all etcd environment variables
cat > "/etc/etcd/etcd-members" <<EOF
ETCD_INITIAL_CLUSTER="$ETCD_INITIAL_CLUSTER"
ETCD_NAME="$public_peer_hostname"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-1"
ETCD_ADVERTISE_CLIENT_URLS="https://$public_peer_hostname:2379"
ETCD_ADVERTISE_PEER_URLS="https://$public_peer_hostname:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://$public_peer_hostname:2380"
ETCD_LISTEN_CLIENT_URLS="https://$public_peer_ip:2379,https://127.0.1.1:2379"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_CERT_FILE="/etc/etcd/server.pem"
ETCD_KEY_FILE="/etc/etcd/server-key.pem"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ca.pem"
ETCD_LISTEN_PEER_URLS="https://$public_peer_ip:2380"
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_CERT_FILE="/etc/etcd/peer.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/peer-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ca.pem"
EOF
exit 0
To let this run automatically after Server startup we add this to the services list and make it a oneshot service:
[Unit]
Description=Find etcd members by digitalocean api
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
ExecStart=/etc/etcd/prepare_cluster.sh
RemainAfterExit=true
StandardOutput=journal
Now we can download etcd and copy etcd and etcdctl to the /usr/bin directory. After this please add the following service:
[Unit]
Description=etcd
Requires=preparecluster.service
After=preparecluster.service
[Service]
EnvironmentFile=/etc/etcd/etcd-members
ExecStart=/usr/bin/etcd
Restart=always
RestartSec=15
[Install]
WantedBy=multi-user.target
First important part here is, that the etcd requires the preparecluster service. After this service has finished it will load the generated etcd-members file as EnvironmentFile and start the etcd.
After we did this on each node our cluster should be healthy. You can check this by running etcdctl cluster-health. This should look like this:
$ etcdctl --key-file /etc/etcd/client-key.pem --cert-file /etc/etcd/client.pem --ca-file /etc/etcd/ca.pem --endpoints https://`hostname`.koudingspawn:2379 cluster-health
member 9b8076232d4d3ac9 is healthy: got healthy result from https://etcd-server-0.koudingspawn.de:2379
member dcf2164c23324761 is healthy: got healthy result from https://etcd-server-1.koudingspawn.de:2379
member df993b20651f2465 is healthy: got healthy result from https://etcd-server-2.koudingspawn.de:2379
Now our cluster is up and running and we can run operations on it. Please add to each statement the –key-file, –cert-file, –ca-file and –endpoints variables.
$ etcdctl mkdir somedata
$ etcdctl ls
/somedata
$ etcdctl set /somedata/file "THIS IS SPARTA"
THIS IS SPARTA
$ etcdctl get /somedata/file
THIS IS SPARTA
$ etcdctl rm /somedata/file
PrevNode.Value: THIS IS SPARTA
