Skip to content

loodse/kubectl-hacking

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits

fzf

fzf

 
 

kubeone-cluster

kubeone-cluster

 
 

pics

pics

 
 

powerline-go

powerline-go

 
 

.gitignore

.gitignore

 
 

README.md

README.md

 
 

presentation-notes.md

presentation-notes.md

 
 

Repository files navigation

title author patat
$ kubectl hacking - my journey with kate
Tobias Schneck, Loodse GmbH
wrap pandocExtensions margins theme
true
patat_extensions
autolink_bare_uris
left right
5
5
syntaxHighlighting code codeBlock
decVal
bold
onDullRed
dullBlack
onDullWhite
dullBlack
onDullWhite

My journey to Kubernetes

Java programmer -> Testautomation -> Docker -> OpenShift -> Kubernetes

                              ..              
                              ..              
                           ........           
                   ...  ..++..++..++..  ...   
                    ..+++..  .++.  ..+++..    
                     .+.++.. .++. ..++.+.     
                    .+.  .++++++++++.  .+.    
                    ++.  ..++....++..  .++    
                  ..++.+++++++..+++++++.++..  
                .....++..  .++++++.  ..++.....
                     .++.  ++.  .++  .++.     
                      ..+..+.    .+..+..      
                         .+.++++++.+.         
                        .+.        .+.        
                        +.          .+

📧 mailto:tobi@loodse.com | 🐦 https://twitter.com/toschneck | 🐙 https://github.com/toschneck

kubectl - what's this about?

kubectl - what's this about?

basics - bash config

  • Enable bash auto completion (update compatible)

    # add to ~/.bashrc
source <(kubectl completion bash)
alias k=kubectl
complete -F __start_kubectl k

  • beautify your bash prompt -> powerline go

    1. Install binary go get -v -u github.com/justjanne/powerline-go
    2. ensure powerline-go is available in the path: export PATH=$PATH:$GOPATH/bin
    3. config ~/.bashrc -> source ./powerline-go/.bashrc

basics - kubectl help?

  • kubectl [command] --help is very helpful!
  • kubectl explain [object][.field][...] information about spec fields!
  • kubectl api-resources shows available objects - with CRDs!

Examples

kubectl get --help
#options for all commands
kubectl options

kubectl explain pod.spec.containers.ports
kubectl explain svc.spec.type

kubectl api-resources --api-group=apps
kubectl api-resources -o wide

basics - How to manage multiple cluster?

  1. $KUBECONFIG environment variable or --kubeconfig flag

  2. Merge multiple config files and use kubectl set-context

# list and change context
kubectl config get-contexts
kubectl config use-context CONTEXT_NAME

# set the used default namespace
kubectl config set-context --current --namespace=default

# merge multiple config files
KUBECONFIG=conf1:conf2 kubectl config view --flatten > merged.conf

Attention: set-context modifies context values! use-context change the current context!

Tooling fast context switching with kubectx and kubens

Special thx to Ahmet Alp Balkan

Installation

cd $HOME/bin
wget https://raw.githubusercontent.com/ahmetb/kubectx/master/kubectx
wget https://raw.githubusercontent.com/ahmetb/kubectx/master/kubens
chmod 755 kubectx kubens
wget https://github.com/junegunn/fzf-bin/releases/download/0.18.0/fzf-0.18.0-linux_amd64.tgz
tar xf fzf-0.18.0-linux_amd64.tgz
# → kubectx, kubens, fzf

...

Tooling - Usage kubectx, kubens

#### CONTEXT switching ####
# fuzzy search list
kubectx
# direct select e.g. `default` context 
kubectx default
# select last context
kubectx -

#### NAMESPACE switching ####
# fuzzy search list
kubens
# direct select e.g. `kube-system` namespace 
kubens kube-system
# select last namespace
kubens -

Tooling - fuzzy search fzf

  • Search a file the fuzzy way

    fzf
fzf --preview 'cat {}'  # with preview

  • Key binding → add ./fzf/.fzf.bash to your ~/.bashrc

    [ -f ~/.fzf.bash ] && source ~/.fzf.bash

  • Pipe kubectl output, e.g. logs, config

    kubectl logs POD_NAME | fzf
kubectl get pod POD_NAME -o yaml | fzf

kubectl output parameter

  • --v=9 Debug verbosity 0-10

  • -o wide, -o yaml shows more important information about an object

  • --show-labels and --label-columns=k8s-app structure your output

  • -l k8s-app=my-app, --field-selector=status.phase=Running select objects

  • -A (new) using all namespaces

  • -o json | jq 'expresion' combine JSON and jq to get more details (useful for scripting)

  • jsonpath=JSONPATH_EXP powerful one line helper to get multiple valuesx of a json output

  • kubectl describe OBJECT shows information and events

kubectl output parameter

Examples

# all runnings pods
kubectl get pod --field-selector=status.phase=Running -A
 
# node kernel version
kubectl get nodes -o json | jq '.items[].status.nodeInfo.kernelVersion' -r

# all used images
kubectl get pods -A \
  -o jsonpath='{range .items[*]}{@.metadata.name}{" "}{@.spec.containers[*].image}{"\n"}{end}'

# Check which nodes are ready
JSONPATH='{range .items[*]}{"\n---\n"}{@.metadata.name}: 
{"\n"}{range @.status.conditions[*]}{@.type}={@.status}; {"\n"}{end}{end}' \
 && kubectl get nodes -o jsonpath="$JSONPATH"

Quick wins - let kubectl help you!

use run for resource creation

  • inline paramaters: --image=image, --env="key=value", --port=port, --replicas=replicas, --label="myapp=app1"
  • inline manifest overrides: --overrides JSON
  • --generatortrigger different kind of object creation:
  • (alternative) --restart Never/OnFailure or --schedule=CRON
kubectl run --generator=run-pod/v1 --image nginx:1.13 web
kubectl run --generator=run-pod/v1 --image nginx:1.13 --overrides='{"spec": {"hostNetwork": true}} web-host'

kubectl run # without flag creates a deployment
kubectl run --restart=Never  # creates a Pod
kubectl run --restart=OnFailure # creates a job
kubectl run --restart=OnFailure -schedule="* * * * *" # creates a cronjob

K8s doc ref: https://kubernetes.io/docs/reference/kubectl/conventions/#generators

use run for quick debugging

  • run ... -- argument pass the arguments directly to the container 
    # start a simple web image
kubectl run --image=loodse/demo-www --port 80 web-deployment
kubectl get pods -o wide

# verify usage by a debug container
kubectl run --image=nicolaka/netshoot --generator=run-pod/v1 --rm -it -- bash
# ... inside the container: wget WEB_DEPLOYMENT_POD_IP -O -

NOTE: most --generators deprecated, but replacement kubectl create don't support -it 😢

Quick wins - let kubectl help you!

create templates

  • run --restart / create combined with --dry-run -o yaml and creates a template for common resources

    # create a deployment yaml file
kubectl create deployment web-template --image=loodse/demo-www --dry-run -o yaml > dep.create.yaml
kubectl run --image=loodse/demo-www --port 80 --dry-run -o yaml web-template > dep.run.yaml

# job with 10 sleep 
kubectl run --image=busybox --restart=OnFailure --dry-run -o yaml job -- /bin/sleep 10 > job.run.yaml

  • --export get a pod's YAML without cluster specific information

#deployment
kubectl get deployment web-template -o yaml --export > dep.export.yaml
vim dep.export.yaml 
kubectl apply -f dep.export.yaml   

#service
kubectl get service web-template --export -o yaml > svc.export.yaml

Quick wins - let kubectl help you!

Modify resources

  • Use inplace editor functionality

    • KUBE_EDITOR sets the local editor
    • kubectl edit TYP OBJECT open in cluster resource

  • Use apply for mutable objects, replace for immutable objects. Note: You can use -f FOLDER for using multi manifests!

    kubectl apply -f dep.yaml
# delete resource and recreates it
kubectl replace --force -f pod.yaml

  • Use scaling functions

    • kubectl autoscale deployment foo --min=2 --max=10 add HPA
    • kubectl scale deployment --replicas=10 web-deployment scales up

Quick wins - let kubectl help you!

use expose for service creation

Can reference pod (po), service (svc), replicationcontroller (rc), deployment (deploy), replicaset (rs).

  • --port listing port to match at referenced resource
  • --type type of Service: ClusterIP (default), NodePort, LoadBalancer, ExternalName
  • --target-port port at the service
  • --selector specify label selector
kubectl expose deployment web-deployment --type=LoadBalancer --port=80
kubectl get nodes -o wide
kubectl get nodes --selector=kubernetes.io/role!=master \
  -o jsonpath={.items[0].status.addresses[?\(@.type==\"ExternalIP\"\)].address}
  • Combine with port-forward for quick testing or debugging
    • can target pod, deployment, service + localport:remoteport for port mapping
kubectl port-forward svc/web-deployment 8080:80 &
curl localhost:8080

Quick wins - let kubectl help you!

Modify resources

  • Manipulate current objects, e.g. the image value 
    # use set for common modification
kubectl set image deployment/web-deployment web-deployment=loodse/demo-www
kubectl set env deployment/web-deployment TEST=val

# use patch for all other, e.g. service type
kubectl patch svc/web-deployment -p '{"spec":{"type":"NodePort"}}' 

# Update a container's image; spec.containers[*].name is required because it's a merge key
kubectl patch pod/podname -p \
 '{"spec":{"containers":[{"name":"web-deployment","image":"loodse/demo-www"}]}}'

Extend kubectl with plugins

  • Enable kubectl plugin manager krew

    # add to ~/.bashrc
# export KREW_ROOT=/path/to/krew-folder
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

  • Plugin management

kubectl-krew search
kubectl-krew install view-secret
kubectl-krew upgrade

Extend kubectl with plugins

  • Example: decode base64 secrets
kubectl get secret
kubectl view-secret default-token-976rc namespace
  • Example: tree view
kubectl tree deployment web-template
NAMESPACE  NAME                                   READY  REASON  AGE
default    Deployment/web-template                -              44m
default    └─ReplicaSet/web-template-655c987cb8   -              44m
default      └─Pod/web-template-655c987cb8-5dzrh  True           44m

Use kubectl the fuzzy way with fubectl

  • bash/zsh wrapper based on kubectl, jq and fzf

  • Interactive search and interaction with Kubernetes objects

    • support's also CRD's
    • No in-cluster installation needed

  • Installation

    curl -LO https://rawgit.com/kubermatic/fubectl/master/fubectl.source
# add to `~/.bashrc`
[ -f <path-to>/fubectl.source ] && source <path-to>/fubectl.source  
khelp
# [k] like g for git but 233% as effective!
# [ka] get all pods in namespace
# [kall] get all pods in cluster
# [kwa] watch all pods in the current namespace
# [kwall] watch all pods in cluster
# [kp] open kubernetes dashboard with proxy
# [kwatch] watch resource
# [kdebug] start debugging container in the current namespace
# [kube_ctx_name] get the current context
# [kube_ctx_namespace] get current namespace
# [kget] get a resource by its YAML
# [ked] edit a resource by its YAML
# [kdes] describe resource
# [kdel] delete resource
# [klog] fetch log from container
# [kex] execute command in container
# [kfor] port-forward a container port to your local machine
# [ksearch] search for string in resources
# [kcl] context list
# [kcs] context set
# [kcns] context set default namespace
# [kwns] watch pods in a namespace
# [ktree] prints a tree of k8s objects (kubectl tree plugin needs to be installed)
# [kbash] create a pod (ubuntu) with a bash
# [konsole] create root shell (alpine) on a node

Troubleshooting

  • Take a look for objects in state Pending, Error, CrashLoopBackOff and inspect: kubectl describe

  • Use port-forward to test different connections, e.g. service or pod

  • Use prepared debug container for e.g. network debugging

    kubectl run --image=nicolaka/netshoot --restart=Never --rm -it -- bash
# with fubectl
kdebug     #start netshoot container
konsole    #start shell on HOST of node!

  • top for resource usage, requires metrics-server

    kubectl top node
kubectl top pod

Troubleshooting

  • Reproduce the event and stream all matching logs, e.g. with label name=myLabel

    • kubectl logs -f -l name=myLabel --all-containers
    • kubectl get events -A

  • exec into running container

    • kubectl exec my-pod -- ls -la /
    • kubectl exec my-pod -it -- sh

  • Inspect the whole cluster state

    # download the state
kubectl cluster-info dump --output-directory=./output/cluster-state

# diagnose it
tree ./output/cluster-state

grep -r Error output/cluster-state
grep -C 5 -r Error output/cluster-state

Cluster Inspection Tools

++ need no running components in the cluster

k9s

  • provides a curses based terminal UI
  • interactive view similar to htop

popeye

  • Kubernetes Cluster Sanitizer
  • Find errors and warnings

Cluster Management by Cluster API

  • Manage Cluster's by CRDs to abstract of the provider (cloud/on-prem)
  • Currently mostly used for machine creation, see as e.g. machine-controller implementations
    • Used by e.g. HA cluster management tool kubeOne
  • Immutable machine objects handle cluster nodes similar to pods 
    Deployment -> ReplicaSet -> Pod -> Container
MachineDeployment -> MachineSet -> Machine -> Node
  • Independent management Layer between the clouds

Cluster Management by Cluster API

Basic API objects for automation for the patch, resource and access management

# see the machine definition
kubectl describe machine -n kube-system MACHINE_NAME
kubectl get machinedeployment,machineset,machine,node -n kube-system
kubectl tree machinedeployment pool-1 -n kube-system

# scale machines
kubectl scale md --replicas=3 pool-1
# update e.g. kubernetes version, machine size, ...
kubectl edit machinedeployment

# machine to node reference:
kubectl get machine -n kube-system \
  -o jsonpath='{range .items[*]}{@.metadata.name}{" >> "}{@.status.nodeRef.name}{"\n"}{end}}'

Manage VMs with kubevirt

Questions?

I'm happy to answer!

Slides?

Take a look at https://github.com/loodse/kubectl-hacking

Something to add?

Open a pull request 😉

Thx for your attention!

References:

About

$ kubectl hacking - my journey with kate

Resources

Readme
Activity
Custom properties

Stars

38 stars

Watchers

5 watching

Forks

11 forks
Report repository

Releases

2 tags

Packages

No packages published

Languages