CKS Exam Series #4 Crash that Apiserver !
Kubernetes CKS Example Exam Question Series

#####################################
THIS CHALLENGE WON’T BE UPDATED HERE AND MOVED TO:
https://killercoda.com/killer-shell-cks
######################################
Content
- Create Cluster & Security Best Practices
- Pods, Secrets and ServiceAccounts
- Immutable Pods
- Crash that Apiserver & check logs
- ImagePolicyWebhook / AdmissionController
- Users and CertificateSigningRequests
- ServiceAccount Token Mounting
- Role Based Access Control (RBAC)
- Role Based Access Control (RBAC) v2
- Container Hardening
- NetworkPolicies (Default Deny + Allowlist)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
→ Check out the FULL CKS COURSE on Udemy ←
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Rules!
- Be fast, avoid creating yaml manually from scratch
- Use only kubernetes.io/docs for help.
- Check our solution after you did yours. You probably have a better one!
Todays Task: Crash the Apiserver and check logs
You should be very comfortable changing the Apiserver config. You will probably mess something up when altering Apiserver config, and this isn’t a bad thing if you know where to check for logs!
- Configure the Apiserver manifest with a new argument
--this-is-very-wrong
. Check if the Pod comes back up and what logs this causes. Fix the Apiserver again. - Change the existing Apiserver manifest argument to:
—-etcd-servers=this-is-very-wrong.
Check what the logs say, and fix it again. - Change the Apiserver manifest and add invalid YAML. Check what the logs say, and fix again.
.
.
.
.
.
Solution

alias k=kubectl
Log locations to check:
/var/log/pods
/var/log/containers
docker ps
+docker logs
crictl ps
+crictl logs
(in case when Docker isn’t used)- kubelet logs:
/var/log/syslog
orjournalctl -u kubelet
1. Add unknown argument
cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/kube-apiserver.yaml.ori # always make a backup !vim /etc/kubernetes/manifests/kube-apiserver.yaml
Edit:
...
spec:
containers:
- command:
- kube-apiserver
- --this-is-very-wrong
- --advertise-address=10.156.0.53
- --allow-privileged=true
...
Wait for it to come back:
k -n kube-system get pod # nothing there
Check for logs:
cd /var/log/podsls -lh | grep apiservertail -f kube-system_kube-apiserver-cks-master_7aef8559c5d7d59259044bb444b01ac3/kube-apiserver/4.log

And we remove the unknown flag
to fix the Apiserver again:
cp ~/kube-apiserver.yaml.ori /etc/kubernetes/manifests/kube-apiserver.yaml # smart people use a backup

2. Misconfigure ETCD connection
cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/kube-apiserver.yaml.ori # always make a backup !vim /etc/kubernetes/manifests/kube-apiserver.yaml
Edit:
...
spec:
containers:
- command:
- kube-apiserver
...
- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
- --etcd-servers=this-is-very-wrong
- --insecure-port=0
...
The Apiserver needs to communicate with ETCD:

We can see the process running:

We see the container running:

OR USE
crictl ps
andcrictl logs
if Docker isn’t used
We can check container logs: docker logs 829f1d829a43
:

And we fix the Apiserver again:
cp ~/kube-apiserver.yaml.ori /etc/kubernetes/manifests/kube-apiserver.yaml # clever people use a backup
3. Invalid Apiserver Manifest YAML
This will cause the kubelet not being able to even create a Container. So we cannot check Pod or Container logs.
cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/kube-apiserver.yaml.ori # do you do backup?vim /etc/kubernetes/manifests/kube-apiserver.yaml
Edit to invalid YAML:
apiVersion: v1
kind: Pod
metadata:
...
name: kube-apiserver
namespace: kube-system
specTHIS IS VERY ::::: WRONG
containers:
- command:
- kube-apiserver
...
Hello apiserver process?
ps aux | grep apiserver # nada
Pod logs?
find /var/log/pods/ | grep apiserver # nichts
Container?
docker ps | grep apiserver # YUdGb1lTRWhJUW89Cg==
OR USE
crictl ps
andcrictl logs
if Docker isn’t used
Kubelet syslog logs (/var/log/syslog
)?
Jan 2 18:41:00 cks-controlplane kubelet[7000]: E0102 18:41:00.573469 7000 file.go:187] Can't process manifest file "/etc/kubernetes/manifests/kube-apiserver.yaml": /etc/kubernetes/manifests/kube-apiserver.yaml: couldn't parse as pod(yaml: line 13: mapping values are not allowed in this context), please check config file
Or also possible:
journalctl -u kubelet | tail
Now fix it again to make the apiserver happy:
cp ~/kube-apiserver.yaml.ori /etc/kubernetes/manifests/kube-apiserver.yaml # wise people use a backup
.
.
.
.
.
You have a different solution?
Let us know by leaving a comment below!
Recap: don’t panic

- If the Apiserver doesn’t come back up, don’t freak out. There are most probably logs that give you some more information!
- Always make a backup before changing K8s component configs.
- Be comfortable checking Pod/Container/Process/Syslog/Kubelet logs.
— — — The END — — —
So much for this session. See you at the next one and happy learning!
Ready to join Killer Shell?
FULL CKS COURSE

…or the CKS SIMULATOR
