• May 30, 2023
  • 166 views
  • 0 Comments

Kubernetes is a powerful open-source container orchestration system that enables the ability to automate deployments and manage containerized workloads and services. Kubernetes automates operational tasks in software deployments and allows users to schedule and run containers on clusters the can utilize both physical or virtual machines in public cloud, private cloud and hybrid environments. Kubernetes is awesome when it comes to management of containers that can improve resources utilization and reduce costs. Kubernetes also is known for optimize hardware resources, improving DevOps productivity, and manage changes with containerized applications.

Now lets go over a few things before we setup
All though I am confident instructions will work if you are using virutalbox, vmware workstations, or hyvervisor or a later version of VMware ESXI. In this lab I am using the following

  • VMware ESXI  6.5, with 2 machines 1 master and 1 worker node
  • OS installed on both VMs is ubuntu 20.04
  • Using kubeadm, to initialize and provision the kubernetes cluster
  • Installing Flannel as the container network interface

 

We need to get a few things setup before Install kubeadm and initializing our cluster

First we want to disable swap because if not turned off swap can cause issues with kubelet

# use command below to disable swap
sudo swapoff -a

This is optional but I added the IPs for my nodes into my /etc/hosts

# use command below to edit the /etc/hosts file
vi /etc/hosts

# Add lines and change the ips below to IPs 
# to the IP your using for your nodes in your Cluster  
172.30.9.30 master1
172.30.9.31 worker1
# if you have more than one worker node
172.30.9.32 worker2

Open following Ports on the firewall

# Ports below we need to be opened on the master nodes only 
sudo ufw allow ssh
sudo ufw enable
sudo ufw allow 6443/tcp
sudo ufw allow 2379/tcp
sudo ufw allow 2380/tcp
sudo ufw allow 10250/tcp
sudo ufw allow 10257/tcp
sudo ufw allow 10259/tcp
sudo ufw reload

# Ports below we need to be opened on the worker nodes only
sudo ufw allow ssh
sudo ufw enable
sudo ufw allow 10250/tcp
sudo ufw allow 30000:32767/tcp
sudo ufw reload

Load the br_netfilter module to enable traffic communication between Kubernetes pods across the cluster nodes.

# Enable kernel modules
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

# Add some settings to sysctl
sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

# Reload sysctl
sudo sysctl --system

Installing docker runtime

You want to install Dockers into each of your Kubernetes nodes for running Pods. Docker is the most popular option, so docker was what I also went with when setting up this cluster. All the commands are shown below:

# Update the Ubuntu 
sudo apt -y update

# Installing Docker 
sudo apt -y install docker.io

# Installing curl 
sudo apt install -y apt-transport-https ca-certificates curl

# make the keyrings direcotry 
sudo mkdir /etc/apt/keyrings

# make the dockers directory if it does not exist
sudo mkdir /etc/docker

# want to change Docker daemon to use systemd for the management of the container’s cgroups. To do this, run the following command below
sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{ "exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts":
{ "max-size": "100m" },
"storage-driver": "overlay2"
}
EOF

# then restart docker services 
sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker

Installing Kubeadm

Kubeadm is a tool that helps to install and configure your Kubernetes cluster. To install kubeadm, perform the following in Ubuntu.

# Add and install repo packages
curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://dl.k8s.io/apt/doc/apt-key.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

# update ubuntu 
sudo apt -y update 

# installing kubeadm kubectl and kubelet
sudo apt -y install kubeadm kubelet kubectl kubernetes-cni

# mark hold kubeadm kubectl and kubelet so its not updated along with other packages if someone runs a apt update 
sudo apt-mark hold kubelet kubeadm kubectl

Initialize Kubernetes on Master Node

Command below only use while setting up the master node

# go edit the kuelet configu file 
vi /etc/default/kubelet

# add following line
KUBELET_EXTRA_ARGS="--cgroup-driver=cgroupfs"


# edit the kubeadm conf file 
sudo vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

# add following line below under service 
Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false"

# restart both the kubelet and the daemon 
systemctl daemon-reload

systemctl restart kubelet

# next run the initialize with the CIDR of 10.244.0.0/16
kubeadm init --pod-network-cidr=10.244.0.0/16

# After you run the kubeadm init 
# it will output the following command to join your worker node to the cluster 
# store the token into a safe place 
# example below
kubeadm join 172.30.9.30:6443 --token kce5ji.5r6wis7xaqzkbw2v 
        --discovery-token-ca-cert-hash sha256:06699b8fb404dbddb98614549c499c8002bb1f65bae41e6e61f5dded761ff60e


# now create your kube config file using commands below 
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Installing Flannel

There are many networking solutions available for Kubernetes. You can find the full list and links here: Cluster Networking .

In our case we are installing Flannel and you want to only install Flannel on the master node not the worker node
Run commands below

# install flannel using command below
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

# list all in all the namespaces 
kubectl get pods --all-namespaces

# now the coreDNS pods take about a minute or 2 to provision
# after a minute or 2 rerun command above

If all node Show as running your pretty much done and now you can proceed with adding your worker node

Setting up worker node

For the work node you want to rerun all the commands above again except for the kubeadm init and the flannel commands
here are the command to run on the worker node below:

disable swap because if not turned off swap can cause issues with kubelet

# use command below to disable swap
sudo swapoff -a

Once again This is optional but I added the IPs for my nodes into my /etc/hosts

# use command below to edit the /etc/hosts file
vi /etc/hosts

# Add lines and change the ips below to IPs 
# to the IP your using for your nodes in your Cluster  
172.30.9.30 master1
172.30.9.31 worker1
# if you have more than one worker node
172.30.9.32 worker2

Open following Ports on the firewall for the worker node

# Ports below we need to be opened on the master nodes only 
sudo ufw allow ssh
sudo ufw enable
sudo ufw allow 6443/tcp
sudo ufw allow 2379/tcp
sudo ufw allow 2380/tcp
sudo ufw allow 10250/tcp
sudo ufw allow 10257/tcp
sudo ufw allow 10259/tcp
sudo ufw reload

# Ports below we need to be opened on the worker nodes only
sudo ufw allow ssh
sudo ufw enable
sudo ufw allow 10250/tcp
sudo ufw allow 30000:32767/tcp
sudo ufw reload

Load the br_netfilter module to enable traffic communication between Kubernetes pods across the cluster nodes.

# Enable kernel modules
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

# Add some settings to sysctl
sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

# Reload sysctl
sudo sysctl --system

Installing docker runtime

You want to install Dockers into each of your Kubernetes nodes for running Pods. Docker is the most popular option, so docker was what I also went with when setting up this cluster. All the commands are shown below:

# Update the Ubuntu 
sudo apt -y update

# Installing Docker 
sudo apt -y install docker.io

# Installing curl 
sudo apt install -y apt-transport-https ca-certificates curl

# make the keyrings direcotry 
sudo mkdir /etc/apt/keyrings

# make the dockers directory if it does not exist
sudo mkdir /etc/docker

# want to change Docker daemon to use systemd for the management of the container’s cgroups. To do this, run the following command below
sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{ "exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts":
{ "max-size": "100m" },
"storage-driver": "overlay2"
}
EOF

# then restart docker services 
sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker

Installing Kubeadm

Kubeadm is a tool that helps to install and configure your Kubernetes cluster. To install kubeadm, perform the following in Ubuntu.

# Add and install repo packages
curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://dl.k8s.io/apt/doc/apt-key.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

# update ubuntu 
sudo apt -y update 

# installing kubeadm kubectl and kubelet
sudo apt -y install kubeadm kubelet kubectl kubernetes-cni

# mark hold kubeadm kubectl and kubelet so its not updated along with other packages if someone runs a apt update 
sudo apt-mark hold kubelet kubeadm kubectl

Use commands below to join the worker node to the cluster

# take the token that was earlier
kubeadm join 172.30.9.30:6443 --token kce5ji.5r6wis7xaqzkbw2v 
        --discovery-token-ca-cert-hash sha256:06699b8fb404dbddb98614549c499c8002bb1f65bae41e6e61f5dded761ff60e

# if you do not have or remember the token 
# use command below to create new join token:
kubeadm token create --print-join-command

# example new join token below copy and paste into the worker nodes 
kubeadm join 172.30.9.30:6443 --token ssi5ij.4c3dsmg62v9zivg7 
        --discovery-token-ca-cert-hash sha256:b79bb89808e2c64f4e8a81a498eaf1370b7d9dbdc2027d314d0159933f0dff0f


# once server joins the cluster run command 
kubectl get nodes

# example output
root@master1:~# kubectl get nodes
NAME      STATUS   ROLES           AGE   VERSION
master1   Ready    control-plane   8d    v1.27.01
worker1   Ready    <none>          8d    v1.27.01


# list all pods in all namespaces  
root@master1:~# kubectl get pods --all-namespaces
NAMESPACE      NAME                              READY   STATUS    RESTARTS        AGE
kube-flannel   kube-flannel-ds-kmf4n             1/1     Running   2 (46h ago)     8d
kube-flannel   kube-flannel-ds-pspjz             1/1     Running   1 (7d10h ago)   8d
kube-system    coredns-565d847f94-54dct          1/1     Running   1 (7d10h ago)   8d
kube-system    coredns-565d847f94-mxqvn          1/1     Running   1 (7d10h ago)   8d
kube-system    etcd-master1                      1/1     Running   1 (7d10h ago)   8d
kube-system    kube-apiserver-master1            1/1     Running   1 (7d10h ago)   8d
kube-system    kube-controller-manager-master1   1/1     Running   1 (7d10h ago)   8d
kube-system    kube-proxy-hcft7                  1/1     Running   1 (7d10h ago)   8d
kube-system    kube-proxy-hkf88                  1/1     Running   1 (7d10h ago)   8d
kube-system    kube-scheduler-master1            1/1     Running   1 (7d10h ago)   8d

Testing Kubernetes Cluster

Congrats if you’ve made it this far in this tutorial now its time to test our Cluster out to see if everything works

# Now create your manifest file running below command 
vi deployment.yaml


# add following lines below into the file
apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx
spec:
 replicas: 2
 selector:
  matchLabels:
    app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        resources: {}
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
 name: nginx
spec:
 type: NodePort
 ports:
  - targetPort: 80
    port: 80
    nodePort: 30008
 selector:
  app: nginx



# after lines been added run following command 
kubectl create -f deployment.yaml


# check if the pods are running and to see what node the nginx app is running
root@master1:~# kubectl get pods -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP            NODE      NOMINATED NODE   READINESS GATES
nginx-cd55c47f5-2g77w     1/1     Running   0          22h   10.244.1.18   worker1   <none>           <none>
nginx-cd55c47f5-k5n25     1/1     Running   0          22h   10.244.1.17   worker1   <none>           <none>


# list your services 
root@master1:~# kubectl get services
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        8d
nginx        NodePort    10.104.192.0     <none>        80:30008/TCP   22h

Since the nginx app is running on our worker1 node in my case I know my worker node has the IP of 172.30.9.31 and the port we specified for our pod to communicate on is port 30008
if you do not know the ip of your worker node login to your worker node run command ip a or ifconfig to get the IP of your worker node

then in your browser  http://your-node-ip:30008
and your nginx should appear in your browser like in example below

if the nginx page comes up congrats you’ve successfully setup your kubernetes cluster