In this second part of the tutorial series on Rancher Labs’ K3s project, we will explore how to configure a three-node etcd cluster that will be used by a highly available, multinode K3s cluster.
The etcd project is one of the most popular open source projects in the cloud native ecosystem. It’s a Cloud Native Computing Foundation-incubating project that has already been a core building block of Kubernetes infrastructure.
By the end of this tutorial, you will have a TLS enabled, three-node etcd cluster deployed which acts as an external datastore of a highly available K3s cluster with multiple masters.
Make sure you have three Linux hosts with static IP addresses. In my lab, I am running four Intel NUC mini PCs running Ubuntu 18.04 with IP addresses ranging from 10.0.0.60 to 10.0.0.63. We will install etcd on the hosts with IP addresses 10.0.0.60, 10.0.0.61, and 10.0.0.62. You will have to replace these IP addresses with your own set of addresses.
Downloading etcd Binaries
On each Linux host, run the below commands to download and install the latest version of the binaries:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | ETCD_VER=v3.4.10 DOWNLOAD_URL=https://storage.googleapis.com/etcd rm –f /tmp/etcd–${ETCD_VER}–linux–amd64.tar.gz rm –rf /tmp/etcd–download–test mkdir –p /tmp/etcd–download–test curl –L ${DOWNLOAD_URL}/${ETCD_VER}/etcd–${ETCD_VER}–linux–amd64.tar.gz –o /tmp/etcd–${ETCD_VER}–linux–amd64.tar.gz tar xzvf /tmp/etcd–${ETCD_VER}–linux–amd64.tar.gz –C /tmp/etcd–download–test —strip–components=1 rm –f /tmp/etcd–${ETCD_VER}–linux–amd64.tar.gz chmod +x /tmp/etcd–download–test/etcd chmod +x /tmp/etcd–download–test/etcdctl #Verify the downloads /tmp/etcd–download–test/etcd —version /tmp/etcd–download–test/etcdctl version #Move them to the bin folder sudo mv /tmp/etcd–download–test/etcd /usr/local/bin sudo mv /tmp/etcd–download–test/etcdctl /usr/local/bin |
Generating and Distributing the Certificates
We will use cfssl tool from Cloudflare to generate the certificates and keys. If you using Mac as your workstation, you can install it using Homebrew.
1 | brew install cfssl |
Make a directory called certs and run the below commands to generate the CA certificate and server certificate and key combinations for each host.
1 | mkdir certs && cd certs |
First, let’s create the CA certificate that’s going to be used by all the etcd servers and the clients.
1 2 | echo ‘{“CN”:”CA”,”key”:{“algo”:”rsa”,”size”:2048}}’ | cfssl gencert –initca – | cfssljson –bare ca – echo ‘{“signing”:{“default”:{“expiry”:”43800h”,”usages”:[“signing”,”key encipherment”,”server auth”,”client auth”]}}}’ > ca–config.json |
This results in three files – ca-key.pem, ca.pem, and ca.csr
Next, we will generate the certificate and key for the first node.
1 2 3 | export NAME=node–1 export ADDRESS=10.0.0.60,$NAME echo ‘{“CN”:”‘$NAME‘”,”hosts”:[“”],”key”:{“algo”:”rsa”,”size”:2048}}’ | cfssl gencert –config=ca–config.json –ca=ca.pem –ca–key=ca–key.pem –hostname=“$ADDRESS” – | cfssljson –bare $NAME |
Repeat the steps for the next two nodes.
1 2 3 | export NAME=node–2 export ADDRESS=10.0.0.61,$NAME echo ‘{“CN”:”‘$NAME‘”,”hosts”:[“”],”key”:{“algo”:”rsa”,”size”:2048}}’ | cfssl gencert –config=ca–config.json –ca=ca.pem –ca–key=ca–key.pem –hostname=“$ADDRESS” – | cfssljson –bare $NAME |
1 2 3 | export NAME=node–3 export ADDRESS=10.0.0.62,$NAME echo ‘{“CN”:”‘$NAME‘”,”hosts”:[“”],”key”:{“algo”:”rsa”,”size”:2048}}’ | cfssl gencert –config=ca–config.json –ca=ca.pem –ca–key=ca–key.pem –hostname=“$ADDRESS” – | cfssljson –bare $NAME |
Don’t forget to replace the IP addresses and node names with your own combination.
At this point, we have the certificates and keys generated for the CA and all the three nodes.
It’s time to distribute these certificates to each node of the cluster.
Run the below command to copy the certificates to the respective nodes by replacing the username and the IP address.
1 2 3 4 5 6 | HOST=10.0.0.60 USER=ubuntu scp ca.pem $USER@$HOST:etcd–ca.crt scp node–1.pem $USER@$HOST:server.crt scp node–1–key.pem $USER@$HOST:server.key |
SSH into each node and run the below commands to move the certificates into an appropriate directory.
1 2 3 4 5 6 7 | HOST=10.0.0.60 USER=ubuntu ssh $USER@$HOST sudo mkdir –p /etc/etcd sudo mv * /etc/etcd sudo chmod 600 /etc/etcd/server.key |
We are done with the generation and distribution of certificates on each node. In the next step, we will create the configuration file and the Systemd unit file per each node.
Configuring and Starting the etcd Cluster
On node 1, create a file called etcd.conf in /etc/etcd directory with the below contents:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | ETCD_NAME=node–1 ETCD_LISTEN_PEER_URLS=“https://10.0.0.60:2380” ETCD_LISTEN_CLIENT_URLS=“https://10.0.0.60:2379” ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster” ETCD_INITIAL_CLUSTER=“node-1=https://10.0.0.60:2380,node-2=https://10.0.0.61:2380,node-3=https://10.0.0.62:2380” ETCD_INITIAL_ADVERTISE_PEER_URLS=“https://10.0.0.60:2380” ETCD_ADVERTISE_CLIENT_URLS=“https://10.0.0.60:2379” ETCD_TRUSTED_CA_FILE=“/etc/etcd/etcd-ca.crt” ETCD_CERT_FILE=“/etc/etcd/server.crt” ETCD_KEY_FILE=“/etc/etcd/server.key” ETCD_PEER_CLIENT_CERT_AUTH=true ETCD_PEER_TRUSTED_CA_FILE=“/etc/etcd/etcd-ca.crt” ETCD_PEER_KEY_FILE=“/etc/etcd/server.key” ETCD_PEER_CERT_FILE=“/etc/etcd/server.crt” ETCD_DATA_DIR=“/var/lib/etcd” |
For node 2, use the below content.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | ETCD_NAME=node–2 ETCD_LISTEN_PEER_URLS=“https://10.0.0.61:2380” ETCD_LISTEN_CLIENT_URLS=“https://10.0.0.61:2379” ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster” ETCD_INITIAL_CLUSTER=“node-1=https://10.0.0.60:2380,node-2=https://10.0.0.61:2380,node-3=https://10.0.0.62:2380” ETCD_INITIAL_ADVERTISE_PEER_URLS=“https://10.0.0.61:2380” ETCD_ADVERTISE_CLIENT_URLS=“https://10.0.0.61:2379” ETCD_TRUSTED_CA_FILE=“/etc/etcd/etcd-ca.crt” ETCD_CERT_FILE=“/etc/etcd/server.crt” ETCD_KEY_FILE=“/etc/etcd/server.key” ETCD_PEER_CLIENT_CERT_AUTH=true ETCD_PEER_TRUSTED_CA_FILE=“/etc/etcd/etcd-ca.crt” ETCD_PEER_KEY_FILE=“/etc/etcd/server.key” ETCD_PEER_CERT_FILE=“/etc/etcd/server.crt” ETCD_DATA_DIR=“/var/lib/etcd” |
Finally, create the configuration file for the last node.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | ETCD_NAME=node–3 ETCD_LISTEN_PEER_URLS=“https://10.0.0.62:2380” ETCD_LISTEN_CLIENT_URLS=“https://10.0.0.62:2379” ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster” ETCD_INITIAL_CLUSTER=“node-1=https://10.0.0.60:2380,node-2=https://10.0.0.61:2380,node-3=https://10.0.0.62:2380” ETCD_INITIAL_ADVERTISE_PEER_URLS=“https://10.0.0.62:2380” ETCD_ADVERTISE_CLIENT_URLS=“https://10.0.0.62:2379” ETCD_TRUSTED_CA_FILE=“/etc/etcd/etcd-ca.crt” ETCD_CERT_FILE=“/etc/etcd/server.crt” ETCD_KEY_FILE=“/etc/etcd/server.key” ETCD_PEER_CLIENT_CERT_AUTH=true ETCD_PEER_TRUSTED_CA_FILE=“/etc/etcd/etcd-ca.crt” ETCD_PEER_KEY_FILE=“/etc/etcd/server.key” ETCD_PEER_CERT_FILE=“/etc/etcd/server.crt” ETCD_DATA_DIR=“/var/lib/etcd” |
Please don’t forget to replace the IP addresses specific to your network.
With the configuration in place, it’s time for us to create the systemd unit file on each node.
Create the file, etcd.service at /lib/systemd/system with the below content:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | [Unit] Description=etcd key–value store Documentation=https://github.com/etcd-io/etcd After=network.target [Service] Type=notify EnvironmentFile=/etc/etcd/etcd.conf ExecStart=/usr/local/bin/etcd Restart=always RestartSec=10s LimitNOFILE=40000 [Install] WantedBy=multi–user.target |
Since the configuration per node is moved into the dedicated file (/etc/etcd/etcd.conf), the unit file remains the same for all the nodes.
We are now ready to start the service. Run the below command on each node to start the etcd cluster.
1 2 3 | sudo systemctl daemon–reload sudo systemctl enable etcd sudo systemctl start etcd |
Make sure that the etcd service is up and running with no errors.
1 | sudo systemctl status etcd |
Testing and Validating the Cluster
SSH into one of the nodes to connect to the cluster through the etcdctl CLI.
1 | etcdctl —endpoints https://10.0.0.60:2379 –cert /etc/etcd/server.crt –cacert /etc/etcd/etcd-ca.crt –key /etc/etcd/server.key put foo bar |
We inserted a key into the etcd database. Let’s see if we can retrieve it.
1 | etcdctl —endpoints https://10.0.0.60:2379 –cert /etc/etcd/server.crt –cacert /etc/etcd/etcd-ca.crt –key /etc/etcd/server.key get foo |
Next, let’s use the API endpoint to check the health of the cluster.
1 | curl —cacert /etc/etcd/etcd–ca.crt —cert /etc/etcd/server.crt —key /etc/etcd/server.key https://10.0.0.60:2379/health |
Finally, let’s ensure that all the nodes are participating in the cluster.
1 | etcdctl —endpoints https://10.0.0.60:2379 –cert /etc/etcd/server.crt –cacert /etc/etcd/etcd-ca.crt –key /etc/etcd/server.key member list |
Congratulations! You now have a secure, distributed, highly available etcd cluster that’s ready for a production-grade K3s cluster environment.
In the next part of this tutorial, I will walk you through the steps of installing and configuring a 4-node K3s cluster with a highly available control plane. Stay tuned!
Janakiram MSV’s Webinar series, “Machine Intelligence and Modern Infrastructure (MI2)” offers informative and insightful sessions covering cutting-edge technologies. Sign up for the upcoming MI2 webinar at http://mi2.live.
InApps is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Unit.
The Cloud Native Computing Foundation is a sponsor of InApps.
Feature image by analogicus from Pixabay.
