Contents
Based on http://tracpath.com/works/devops/how-to-install-the-kubernetes-kubeadm/
- Distro: ubuntu 18.04 LTS
(VirtualBox) Add an internal network between VMs:
http://qiita.com/areaz_/items/c9075f7a0b3e147e92f2#%E3%82%B2%E3%82%B9%E3%83%88os%E3%81%AE%E5%8B%95%E4%BD%9C%E7%A2%BA%E8%AA%8D
Shutdown a VM
Setting -> Network -> Adapter 2
Check "Enable Network Adapter"
Attached to: "Internal Network"
Reboot the VM
SSH into the VM (kube-host01 should be 172.168.0.2)
$ sudo vi /etc/network/interfaces
+ auto enp0s8
+ iface enp0s8 inet static
+ address 172.168.0.1
+ netmask 255.255.255.0
Reboot the VMOperate the following installation on both kube-master and kube-host01 (nfs-common is for Subpath):
$ sudo su -
# apt-get update && apt-get install -y apt-transport-https
# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
# echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
# add-apt-repository ppa:gluster/glusterfs-4.1
# apt-get update
# apt-get install -y docker.io nfs-common glusterfs-clientTo install the latest packages of Kubernetes:
# apt-get install -y kubelet kubeadm kubectlIf you want to install previous release of Kubernetes, check available releases with:
# apt-cache policy kubelet
kubelet:
Installed: (none)
Candidate: 1.10.0-00
Version table:
1.10.0-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.9.6-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.9.5-00 500
[..]
#Then you can specify the release like:
# apt-get install -y kubelet=1.9.6-00 kubeadm=1.9.6-00 kubectl=1.9.6-00(Flannel) Operate the following commands:
# kubeadm init --pod-network-cidr=10.244.0.0/16
[..]
Your Kubernetes master has initialized successfully!
[..]
You can now join any number of machines by running the following on each node
as root:
kubeadm join --token 22ac74.4d061109507a992b 172.27.138.55:644310.244.0.0/16 comes from kube-flannel.yml which contains:
"Network": "10.244.0.0/16",(Other) Operate the following commands:
# kubeadm init
[..]
Your Kubernetes master has initialized successfully!
[..]
You can now join any number of machines by running the following on each node
as root:
kubeadm join --token 22ac74.4d061109507a992b 172.27.138.55:6443
#The above output needs to be operated on kube-host01 to join into the cluster.
If using VirtualBox, need to specify the internal ip address like:
# kubeadm init --apiserver-advertise-address 172.168.0.1Operate the following commands:
$ sudo cp /etc/kubernetes/admin.conf $HOME/
$ sudo chown $(id -u):$(id -g) $HOME/admin.conf
$ export KUBECONFIG=$HOME/admin.conf
$ echo "export KUBECONFIG=$HOME/admin.conf" >> $HOME/.bashrc(Flannel) Configure network setting for pod2pod communication:
$ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
$ kubectl apply -f kube-flannel.yml(Weave) Configure network setting for pod2pod communication:
$ kubectl apply -f https://git.io/weave-kube-1.6Check the valid installation:
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-kube-manager 1/1 Running 0 1h
kube-system kube-apiserver-kube-manager 1/1 Running 0 1h
kube-system kube-controller-manager-kube-manager 1/1 Running 0 1h
kube-system kube-dns-692378583-3gbgp 0/3 ContainerCreating 0 1h
kube-system kube-proxy-4rbvg 1/1 Running 0 1h
kube-system kube-scheduler-kube-manager 1/1 Running 0 1h
kube-system weave-net-cjf25 2/2 Running 0 51s
$Confirm the STATUS becomes Ready:
$ kubectl get nodes
NAME STATUS AGE VERSION
k8s-master Ready 1m v1.7.3Make the manager schedulable:
$ kubectl describe nodes | grep Tain
Taints: node-role.kubernetes.io/master:NoSchedule
$ kubectl taint nodes <master nodename: k8s-master> node-role.kubernetes.io/master:NoSchedule-
node "k8s-master" untainted
$ kubectl describe nodes | grep Tain
Taints: <none>
$To make the mount propagation work proerly, edit /etc/systemd/system/multi-user.target.wants/docker.service like:
[Service]
- MountFlags=slave
+ MountFlags=sharedThis is required to pass e2e test "[sig-storage] CSI Volumes CSI plugin test using CSI driver: hostPath".
To add a node into k8s cluster, operate the following command on a node (not manager):
# kubeadm join --token 22ac74.4d061109507a992b 172.27.138.55:6443Check the node joins into the cluster with the command on the manager:
$ kubectl get nodes
NAME STATUS AGE VERSION
kube-host01 Ready 51s v1.6.6
kube-manager Ready 1h v1.6.6
$Get a kubeadm token on k8s-master:
$ TOKEN=`sudo kubeadm token list | grep authentication | awk '{print $1}'`
$ echo $TOKEN
c3cf19.89e62945a88d7a91If you cannot get a token, need to recreate with:
$ sudo kubeadm token createGet a discovery token on k8s-master:
$ DISCOVERY_TOKEN=`openssl x509 -pubkey \
-in /etc/kubernetes/pki/ca.crt | openssl rsa \
-pubin -outform der 2>/dev/null | openssl dgst \
-sha256 -hex | sed 's/^.* //'`
$ echo $DISCOVERY_TOKEN
b3bb83c24673649bf1909e9144929a64569b1a7988df97323a9a3449c3b4c1e6Get an endpoint on k8s-master:
$ ENDPOINT=`cat admin.conf | grep server | sed s@" server: https://"@@`
$ echo $ENDPOINT
192.168.1.105:6443Use the token and the discovery token on k8s-node to add a new node on the node:
# TOKEN=c3cf19.89e62945a88d7a91
# DISCOVERY_TOKEN=b3bb83c24673649bf1909e9144929a64569b1a7988df97323a9a3449c3b4c1e6
# ENDPOINT=192.168.1.105:6443
#
# kubeadm join --token ${TOKEN} ${ENDPOINT} \
--discovery-token-ca-cert-hash sha256:${DISCOVERY_TOKEN}Install metrics-server on k8s-master:
$ git clone https://github.com/kubernetes-incubator/metrics-server
$ cd metrics-server/
$ kubectl create -f deploy/kubernetes/Integrate standalone-cinder of the external cloud-provider-openstack for Dynamic Volume Provisioning
NOTE: It is not necessary to add options (--cloud-provider, --cloud-config) to kube-controller-manager and other processes at all.
Use manifests as samples from https://github.com/oomichi/try-kubernetes/tree/master/manifests/standalone-cinder-external
Add RBAC for standalone-cinder deployment:
$ kubectl create -f rbac.yamlChange hostAliases, OS_AUTH_URL and other OS*** env values of deployment.yaml for your environment.
Deploy standalone-cinder:
$ kubectl create -f deployment.yamlAdd default StorageClass:
$ kubectl create -f storage-class.yamlVerify Dynamic Volume Provisioning works fine:
$ kubectl create -f pvc.yaml
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cinder-claim Bound pvc-af01ada4-9cf4-11e8-a146-fa163e420595 1Gi RWO gold 31s
$Change the listening address of kube-scheduler to 0.0.0.0:
$ sudo vi /etc/kubernetes/manifests/kube-scheduler.yaml
--- ./kube-scheduler.yaml.orig 2018-08-22 02:30:16.060204589 +0000
+++ /etc/kubernetes/manifests/kube-scheduler.yaml 2018-08-22 02:30:38.160555932 +0000
@@ -13,7 +13,7 @@
containers:
- command:
- kube-scheduler
- - --address=127.0.0.1
+ - --address=0.0.0.0
- --kubeconfig=/etc/kubernetes/scheduler.conf
- --leader-elect=true
image: k8s.gcr.io/kube-scheduler-amd64:v1.11.1Change the listening address of kube-controller-manager to 0.0.0.0:
$ sudo vi /etc/kubernetes/manifests/kube-controller-manager.yaml
--- kube-controller-manager.yaml 2019-01-08 23:39:48.721525219 +0000
+++ /etc/kubernetes/manifests/kube-controller-manager.yaml 2019-01-08 23:40:09.266003213 +0000
@@ -13,7 +13,7 @@
containers:
- command:
- kube-controller-manager
- - --address=127.0.0.1
+ - --address=0.0.0.0
- --allocate-node-cidrs=true
- --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
- --authorization-kubeconfig=/etc/kubernetes/controller-manager.confStorageObjectInUseProtection sets the PV protection finalizer flag before a PV object appears on k8s API. "PV protection" e2e test requires this behavior.
Add StorageObjectInUseProtection to --enable-admission-plugins option:
$ sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
- --advertise-address=192.168.1.102
- --allow-privileged=true
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- - --enable-admission-plugins=NodeRestriction
+ - --enable-admission-plugins=NodeRestriction,StorageObjectInUseProtection
- --enable-bootstrap-token-auth=true
- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crtJust specify '--v=8' option on kubectl command like:
$ kubectl --v=8 get nodes
[..] GET https://172.27.138.55:6443/api/v1/nodes
[..] Request Headers:
[..] Accept: application/json
[..] User-Agent: kubectl/v1.6.6 (linux/amd64) kubernetes/7fa1c17
[..] Response Status: 200 OK in 21 milliseconds
[..] Response Headers:
[..] Content-Type: application/json
[..] Date: Wed, 28 Jun 2017 00:33:39 GMT
[..] Response Body: {"kind":"NodeList","apiVersion":"v1",
"metadata":{"selfLink":"/api/v1/nodes","resourceVersion":"7254"},
"items":[{"metadata":{"name":"kube-host01","selfLink":"/api/v1/nodeskube-host01",
"uid":"a354969d-5b98-11e7-9e55-1866da463eb0",
"resourceVersion":"7244","creationTimestamp":"2017-06-28T00:27:59Z",
"labels":{"beta.kubernetes.io/arch":"amd64",
"beta.kubernetes.io/os":"linux",
"kubernetes.io/hostname":"kube-host01"} ..This requires metrics-server is enabled. How to know resource work load for each node is:
$ kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master 113m 5% 1271Mi 33%
k8s-node01 116m 5% 902Mi 23%Need to install golang 1.13.7 which is not provided from ubuntu 18.04 as the default. So we need to do the following process for that:
$ wget https://go.dev/dl/go1.18.2.linux-amd64.tar.gz
$ sudo tar -C /usr/local/ -xzf go1.18.2.linux-amd64.tar.gz
$ export PATH=$PATH:/usr/local/go/bin
$ echo "export PATH=$PATH:/usr/local/go/bin" >> $HOME/.bashrc
$ sudo ln -s /usr/local/go/bin/go /usr/local/bin/goSet GOPATH as parmanent setting:
$ mkdir ${HOME}/go
$ echo "export GOPATH=${HOME}/go" >> ${HOME}/.bashrcInstall some building packages:
$ sudo apt-get install -y docker.io gcc makeDownload k8s source code:
$ go get k8s.io/test-infra
$ go get k8s.io/kubernetesCheck out the same version as the target k8s cluster:
$ cd $GOPATH/src/k8s.io/kubernetes
$ kubectl version
Client Version: version.Info{
Major:"1", Minor:"11", GitVersion:"v1.11.1",
GitCommit:"b1b29978270dc22fecc592ac55d903350454310a",
GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1",
GitCommit:"b1b29978270dc22fecc592ac55d903350454310a",
GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
$
$ git tag -l
v0.10.0
..
v1.11.1
..
$
$ git checkout refs/tags/v1.11.1
$ git checkout -b tag-v1.11.1Build e2e test binary. (NOTE: When changing the e2e code, we need to build the binary again to apply the changes):
# The docker daemon runs as root user, not docker user. So it is necessary to specify `sudo`
$ cd ${HOME}/go/src/k8s.io/test-infra/kubetest
$ GO111MODULE=on go install ./kubetest
$ export PATH=${PATH}:${HOME}/go/bin
$ cd ${HOME}/go/src/k8s.io/kubernetes
$ sudo ${HOME}/go/bin/kubetest --buildRun e2e test:
$ export KUBECONFIG=$HOME/admin.conf
$ export KUBERNETES_CONFORMANCE_TEST=true
$ kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\]"Confirm which tests will run without actual tests (Need to specify provider to avoid an error):
$ kubetest --provider=skeleton --test --test_args="--ginkgo.dryRun=true --ginkgo.focus=\[Conformance\]"
[..]
[k8s.io] Docker Containers
should use the image defaults if command and args are blank [Conformance]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/docker_containers.go:35
~SS
------------------------------
[k8s.io] EmptyDir volumes
should support (non-root,0644,tmpfs) [Conformance] [sig-storage]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/empty_dir.go:85
~SS
------------------------------
[sig-apps] ReplicaSet
should serve a basic image on each replica with a public image [Conformance]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/replica_set.go:82
~S
------------------------------
[sig-network] Services
should provide secure master service [Conformance]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:71
~
Ran 149 of 652 Specs in 0.072 seconds
SUCCESS! -- 0 Passed | 0 Failed | 0 Pending | 503 Skipped PASS
Ginkgo ran 1 suite in 519.123083ms
Test Suite Passed
2017/08/09 15:38:12 util.go:133: Step './hack/ginkgo-e2e.sh --ginkgo.dryRun=true --ginkgo.focus=\[Conformance\]' finished in 937.615925ms
2017/08/09 15:38:12 e2e.go:80: Done
$Specify a single test with regex:
$ kubetest --provider=skeleton --test --test_args="--ginkgo.focus=1\spod\sto\s2\spods"If changing e2e code, we need to specify --check-version-skew=false to skip checking versions of both server and e2e client:
$ kubetest --provider=skeleton --test --test_args="--ginkgo.focus=from\s3\sto\s5$" --check-version-skew=falseInstall bazel:
$ sudo apt-get install pkg-config zip g++ zlib1g-dev unzip python
$ wget https://github.com/bazelbuild/bazel/releases/download/0.21.0/bazel-0.21.0-installer-linux-x86_64.sh
$ chmod 755 bazel-0.21.0-installer-linux-x86_64.sh
$ ./bazel-0.21.0-installer-linux-x86_64.sh --user
$
$ export PATH="$PATH:$HOME/bin"
$ echo "export PATH=$PATH:$HOME/bin" >> $HOME/.bashrcRun unit tests on kubernetes/test-infra:
$ bazel test //..- https://github.com/kubernetes/test-infra#building-and-testing-the-test-infra
- http://qiita.com/lucy/items/e4f21c507d3fd2c0ffe9
with make:
$ make testwith bazel:
$ bazel test //...As https://github.com/kubernetes/helm#install :
$ wget https://storage.googleapis.com/kubernetes-helm/helm-v2.9.1-linux-amd64.tar.gz
$ tar -zxvf helm-v2.9.1-linux-amd64.tar.gz
$ sudo mv linux-amd64/helm /usr/local/bin/
$ helm initVerify helm:
$ helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
$Add permission to deploy tiller:
$ kubectl create serviceaccount --namespace kube-system tiller
$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
$ kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'Install Spinnaker:
$ wget https://raw.githubusercontent.com/kubernetes/charts/master/stable/spinnaker/values.yaml
$ helm install -n kubelive -f values.yaml stable/spinnaker
Error: timed out waiting for the condition
$
$ helm ls --all kubelive
NAME REVISION UPDATED STATUS CHART NAMESPACE
kubelive 1 Tue May 15 21:36:52 2018 FAILED spinnaker-0.4.1 default
$
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kubelive-create-bucket-j97wn 0/1 CrashLoopBackOff 5 10m
kubelive-jenkins-86bcb6c4b5-h4bqx 0/1 Pending 0 10m
kubelive-minio-5d78b95d9c-pkpss 0/1 Pending 0 10m
kubelive-redis-5667b84965-k4nmz 0/1 Pending 0 10m
kubelive-spinnaker-clouddriver-85997f4b64-q97qq 0/1 Running 0 10m
kubelive-spinnaker-deck-86c48f7594-vxmnt 1/1 Running 0 10m
kubelive-spinnaker-echo-8ccc9956c-prk58 1/1 Running 0 10m
kubelive-spinnaker-front50-6859bf64bb-cn9bd 0/1 CrashLoopBackOff 6 10m
kubelive-spinnaker-gate-5468cccbc7-n2ncw 0/1 CrashLoopBackOff 6 10m
$
$ kubectl logs kubelive-create-bucket-j97wn
mc: <ERROR> Unable to initialize new config from the provided credentials.
Get http://kubelive-minio:9000/probe-bucket-sign/?location=: dial tcp: lookup kubelive-minio on 10.96.0.10:53: no such host
$Easy one:
$ kubectl get pods -n=default
NAME READY STATUS RESTARTS AGE
pod-00 1/1 Running 0 51s
pod-01 1/1 Running 0 1m
pod-name 1/1 Running 0 18m
$
$ kubectl get pods --sort-by=.status.startTime -n=default
NAME READY STATUS RESTARTS AGE
pod-name 1/1 Running 0 18m
pod-01 1/1 Running 0 55s
pod-00 1/1 Running 0 42s
$
$ kubectl get pods --sort-by=.metadata.name -n=default
NAME READY STATUS RESTARTS AGE
pod-00 1/1 Running 0 2m
pod-01 1/1 Running 0 2m
pod-name 1/1 Running 0 20m
$Easy one:
$ kubectl create -f manifests/pod-01.yamlCreate a pod with some changes by edit without any chages of the original manifest file:
$ kubectl create -f manifests/pod-01.yaml --edit -o jsonCreate a deployment with external network access:
$ kubectl run nginx --image nginx --replicas=3
$ kubectl expose deployment nginx --port=80 --target-port=80
$ kubectl create -f manifests/ingress-nginx.yaml
$ kubectl describe ingress
Name: test-ingress
Namespace: default
Address:
Default backend: nginx:80 (10.244.0.25:80,10.244.0.26:80,10.244.0.27:80)
Rules:
Host Path Backends
---- ---- --------
* * nginx:80 (10.244.0.25:80,10.244.0.26:80,10.244.0.27:80)
Annotations:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 17s nginx-ingress-controller Ingress default/test-ingress
$On this environment, ingress-nginx-controller is used and the setting is:
$ kubectl get services -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default-http-backend ClusterIP 10.102.0.178 <none> 80/TCP 2h
ingress-nginx NodePort 10.101.145.191 <none> 80:31454/TCP,443:31839/TCP 2h
$So NodePort is configured and the host's 31454/TCP is proxied to 80/TCP of the ingress. You can get nginx page like:
$ curl http://localhost:31454
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
..On this environment, etcd is running as a pod on kube-system namespace:
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
etcd-k8s-v109-flannel-master 1/1 Running 0 1d
..
$The manifest is /etc/kubernetes/manifests/etcd.yaml and we can see the endpoint (http://127.0.0.1:2379) in this case:
$ sudo cat /etc/kubernetes/manifests/etcd.yaml
..
- command:
- etcd
- --data-dir=/var/lib/etcd
- --listen-client-urls=http://127.0.0.1:2379
- --advertise-client-urls=http://127.0.0.1:2379
..Install etcdctl command (The ubuntu package is too old and doesn't support the snapshot feature):
$ mkdir foo
$ cd foo
$ wget https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz
$ tar -zxvf etcd-v3.2.18-linux-amd64.tar.gz
$ cd etcd-v3.2.18-linux-amd64Create a snapshot:
$ ETCDCTL_API=3 ./etcdctl --endpoints http://127.0.0.1:2379 snapshot save snapshot.db
Snapshot saved at snapshot.db
$Check pods where live and the node:
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-foo-74cd78d68f-4jwsq 1/1 Running 0 1m 10.244.0.30 k8s-v109-flannel-master
nginx-foo-74cd78d68f-5jl55 1/1 Running 0 1m 10.244.1.7 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-9cts2 1/1 Running 0 1m 10.244.1.5 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-9gtwx 1/1 Running 0 1m 10.244.1.6 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-b7zmx 1/1 Running 0 1m 10.244.1.4 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-d97pw 1/1 Running 0 1m 10.244.0.29 k8s-v109-flannel-master
nginx-foo-74cd78d68f-j27qf 1/1 Running 0 1m 10.244.0.28 k8s-v109-flannel-master
nginx-foo-74cd78d68f-j45c8 1/1 Running 0 1m 10.244.1.2 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-l4mwq 1/1 Running 0 1m 10.244.0.31 k8s-v109-flannel-master
nginx-foo-74cd78d68f-wnb4c 1/1 Running 0 1m 10.244.1.3 k8s-v109-flannel-worker
$
$ kubectl describe node k8s-v109-flannel-worker | grep Taints
Taints: <none>
$Even if making the node tainted with NoSchedule, the pods still exist in the node:
$ kubectl taint nodes k8s-v109-flannel-worker key=value:NoSchedule
node "k8s-v109-flannel-worker" tainted
$ kubectl describe node k8s-v109-flannel-worker | grep Taints
Taints: key=value:NoSchedule
$
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-foo-74cd78d68f-4jwsq 1/1 Running 0 5m 10.244.0.30 k8s-v109-flannel-master
nginx-foo-74cd78d68f-5jl55 1/1 Running 0 5m 10.244.1.7 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-9cts2 1/1 Running 0 5m 10.244.1.5 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-9gtwx 1/1 Running 0 5m 10.244.1.6 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-b7zmx 1/1 Running 0 5m 10.244.1.4 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-d97pw 1/1 Running 0 5m 10.244.0.29 k8s-v109-flannel-master
nginx-foo-74cd78d68f-j27qf 1/1 Running 0 5m 10.244.0.28 k8s-v109-flannel-master
nginx-foo-74cd78d68f-j45c8 1/1 Running 0 5m 10.244.1.2 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-l4mwq 1/1 Running 0 5m 10.244.0.31 k8s-v109-flannel-master
nginx-foo-74cd78d68f-wnb4c 1/1 Running 0 5m 10.244.1.3 k8s-v109-flannel-worker
$After making the node tainted with NoExecute, the pods go away from the node:
$ kubectl taint nodes k8s-v109-flannel-worker key=value:NoExecute
node "k8s-v109-flannel-worker" tainted
$ kubectl describe node k8s-v109-flannel-worker | grep Taints
Taints: key=value:NoExecute
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-foo-74cd78d68f-48q4p 1/1 Running 0 17s 10.244.0.37 k8s-v109-flannel-master
nginx-foo-74cd78d68f-4jwsq 1/1 Running 0 8m 10.244.0.30 k8s-v109-flannel-master
nginx-foo-74cd78d68f-9q6f8 1/1 Running 0 17s 10.244.0.34 k8s-v109-flannel-master
nginx-foo-74cd78d68f-d97pw 1/1 Running 0 8m 10.244.0.29 k8s-v109-flannel-master
nginx-foo-74cd78d68f-j27qf 1/1 Running 0 8m 10.244.0.28 k8s-v109-flannel-master
nginx-foo-74cd78d68f-jlxng 1/1 Running 0 17s 10.244.0.36 k8s-v109-flannel-master
nginx-foo-74cd78d68f-k5rl9 1/1 Running 0 17s 10.244.0.32 k8s-v109-flannel-master
nginx-foo-74cd78d68f-l4mwq 1/1 Running 0 8m 10.244.0.31 k8s-v109-flannel-master
nginx-foo-74cd78d68f-sg52l 1/1 Running 0 17s 10.244.0.33 k8s-v109-flannel-master
nginx-foo-74cd78d68f-vzspf 1/1 Running 0 17s 10.244.0.35 k8s-v109-flannel-master
$Remove the taint after this try:
$ kubectl taint nodes k8s-v109-flannel-worker key-Encode a plain password with base64:
$ echo -n "mypassword" | base64
bXlwYXNzd29yZA==
$Create a secret:
$ cat manifests/secret-01.yaml
apiVersion: v1
kind: Secret
metadata:
name: secret-01
type: Opaque
data:
password: bXlwYXNzd29yZA==
$
$ kubectl create -f manifests/secret-01.yamlCreate a pod with the secret as a file:
$ kubectl create -f manifests/pod-using-secret-as-file.yamlConfirm the password in the pod:
$ kubectl exec -it pod-using-secret-as-file /bin/bash
(login the pod)
#
# ls /etc/foo/
password
# cat /etc/foo/password
mypasswordCreate a pod with the secret as a variable:
$ kubectl create -f manifests/pod-using-secret-as-variable.yamlConfirm the password in the pod:
$ kubectl exec -it pod-using-secret-as-variable /bin/bash
(login the pod)
#
# echo $SECRET_PASSWORD
mypasswordCreate a deployment with a little old nginx (v1.7.9):
$ kubectl create -f manifests/nginx-deployment.yaml
$ kubectl describe deployment/nginx-deployment | grep Image
Image: nginx:1.7.9
$Check the strategy (in this case (the default), that is RollingUpdate and the upgrade happens immediately just after setting the image):
$ kubectl describe deployment/nginx-deployment | grep StrategyType
StrategyType: RollingUpdate
$Check the ReplicaSet name and the pod names:
$ kubectl get rs
NAME DESIRED CURRENT READY AGE
nginx-deployment-75675f5897 3 3 3 6s
$
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-deployment-75675f5897-9mhmv 1/1 Running 0 36s
nginx-deployment-75675f5897-kpgtr 1/1 Running 0 36s
nginx-deployment-75675f5897-plq92 1/1 Running 0 36s
$Set a newer nginx image (v1.9.1):
$ kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1
$ kubectl describe deployment/nginx-deployment | grep Image
Image: nginx:1.9.1
$Then check the status of the upgrade:
$ kubectl rollout status deployment/nginx-deployment
Waiting for rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for rollout to finish: 2 old replicas are pending termination...
Waiting for rollout to finish: 1 old replicas are pending termination...
Waiting for rollout to finish: 1 old replicas are pending termination...
deployment "nginx-deployment" successfully rolled out
$Conform new created ReplicaSet and pods. The old ReplicaSet doesn't have any pods now and new pods only exist:
$ kubectl get rs
NAME DESIRED CURRENT READY AGE
nginx-deployment-75675f5897 0 0 0 3m
nginx-deployment-c4747d96c 3 3 3 1m
$
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-deployment-c4747d96c-fbsw6 1/1 Running 0 2m
nginx-deployment-c4747d96c-gvqg2 1/1 Running 0 1m
nginx-deployment-c4747d96c-jfvvl 1/1 Running 0 1m
$Check the history of a deployment:
$ kubectl rollout history deployment/nginx-deployment
deployments "nginx-deployment"
REVISION CHANGE-CAUSE
1 <none>
2 <none>
$Show the detail of each revision:
$ kubectl rollout history deployment/nginx-deployment --revision=2
deployments "nginx-deployment" with revision #2
Pod Template:
Labels: app=nginx
pod-template-hash=1520898311
Containers:
nginx:
Image: nginx:1.9.1
Port: 80/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
$
$ kubectl rollout history deployment/nginx-deployment --revision=1
deployments "nginx-deployment" with revision #1
Pod Template:
Labels: app=nginx
pod-template-hash=2710681425
Containers:
nginx:
Image: nginx:1.7.9
Port: 80/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
$Rolling-back the deployment:
$ kubectl rollout undo deployment/nginx-deploymentConfirm the rolling-back succeeded:
$ kubectl rollout history deployment/nginx-deployment
deployments "nginx-deployment"
REVISION CHANGE-CAUSE
2 <none>
3 <none>
$ kubectl rollout history deployment/nginx-deployment --revision=3
deployments "nginx-deployment" with revision #3
Pod Template:
Labels: app=nginx
pod-template-hash=2710681425
Containers:
nginx:
Image: nginx:1.7.9
Port: 80/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
$
$ kubectl describe deployment/nginx-deployment | grep Image
Image: nginx:1.7.9
$https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
Check what service works on the cluster:
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 2d
nginx-deployment ClusterIP 10.99.52.90 <none> 80/TCP 24s
$Create a pod for verifying DNS works:
$ kubectl create -f manifests/pod-busybox.yaml
$ kubectl exec -it pod-busybox sh
(login the pod)
wget http://nginx-deployment
Connecting to nginx-deployment (10.99.52.90:80)
index.html 100% |********************************************************************************************************************************************| 612 0:00:00 ETA
/ # cat index.html
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
..
#As the above, DNS works fine and the service nginx-deployment can be looked up from a pod as the same name.
A pod also can be looked up by "pod-ip-address.my-namespace.pod.cluster.local" like:
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
pod-01 1/1 Running 0 19m 10.244.0.48 k8s-v109-flannel-master
$ kubectl exec -it pod-busybox sh
/ #
/ # ping 10-244-0-48.default.pod.cluster.local
PING 10-244-0-48.default.pod.cluster.local (10.244.0.48): 56 data bytes
64 bytes from 10.244.0.48: seq=0 ttl=64 time=0.033 ms
64 bytes from 10.244.0.48: seq=1 ttl=64 time=0.064 mshttps://kubernetes.io/docs/concepts/workloads/pods/init-containers/
Create a pod with init-containers:
$ kubectl create -f manifests/pod-init-container.yamlCheck the pod status, it waits for end of init process:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pod-init-container 0/1 Init:0/2 0 30s
$Check logs of each containers, init-containers start on the order of the manifest. That means 2nd init-container also wait for 1st one's finishes:
$ kubectl logs pod-init-container -c myapp-container
Error from server (BadRequest): container "myapp-container" in pod "pod-init-container" is waiting to start: PodInitializing
$
$ kubectl logs pod-init-container -c init-myservice
waiting for myservice
nslookup: can't resolve 'myservice'
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
waiting for myservice
nslookup: can't resolve 'myservice'
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
waiting for myservice
$
$ kubectl logs pod-init-container -c init-mydb
Error from server (BadRequest): container "init-mydb" in pod "pod-init-container" is waiting to start: PodInitializing
$Create services for making end of init process:
$ kubectl create -f manifests/services-for-init-containers.yaml
service "myservice" created
service "mydb" created
$
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pod-init-container 0/1 PodInitializing 0 4m
$
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pod-init-container 1/1 Running 0 5m
$Then the pod outputs the message to show the end as its command in the manifest:
$ kubectl logs pod-init-container
The app is running!
$Create a daemonset:
$ kubectl create -f manifests/daemonset.yamlCheck the existence:
$ kubectl get ds -n kube-system
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
fluentd-elasticsearch 1 1 1 1 1 <none> 1m
..
$(Non-recommended way) Enforce kubelet boot on an environment with swap:
$ sudo diff -u /etc/systemd/system/kubelet.service.d/10-kubeadm.conf.orig /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
sudo: unable to resolve host k8s-v109-flannel-worker
--- /etc/systemd/system/kubelet.service.d/10-kubeadm.conf.orig 2018-04-05 21:28:10.278748887 +0000
+++ /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 2018-04-05 21:32:14.191449307 +0000
@@ -6,5 +6,6 @@
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
+Environment="KUBELET_SWAP_ARGS=--fail-swap-on=false"
ExecStart=
-ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS
+ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS $KUBELET_SWAP_ARGS
$
$ sudo rebootSwapoff on lxcfs (lxcfs is a simple file system to implement nest-cgroup for systemd environments which are defact init of Linux kernel today):
$ diff -u /usr/share/lxcfs/lxc.mount.hook.orig /usr/share/lxcfs/lxc.mount.hook
--- /usr/share/lxcfs/lxc.mount.hook.orig 2018-04-05 21:55:21.626302043 +0000
+++ /usr/share/lxcfs/lxc.mount.hook 2018-04-05 21:57:05.956673664 +0000
@@ -7,6 +7,7 @@
if [ -d /var/lib/lxcfs/proc/ ]; then
for entry in /var/lib/lxcfs/proc/*; do
[ -e "${LXC_ROOTFS_MOUNT}/proc/$(basename $entry)" ] || continue
+ [ $entry != "swap" ] || continue
mount -n --bind $entry ${LXC_ROOTFS_MOUNT}/proc/$(basename $entry)
done
fi