How to access Kubernetes Cluster
All projects deployed on the SIP are running on Kubernetes. For many operations such as creating or updating a new deployment you can use the sipctl command line tool, but it is sometimes useful (especially for debugging) to directly access your application using kubectl the standard management utility of Kubernetes.
Terms for Kubernetes
| Term | What is it | Official Documentation |
|---|---|---|
| kubectl | Kubectl is the cli management tool provided by the Kubernetes project. | https://kubernetes.io/docs/tasks/tools/install-kubectl/ |
| pod | A pod is a single unit of deployment inside a Kubernetes cluster. You can think of a pod as one instance of your application. In the VSETH context a pod usually contains the application container, plus some additional helper containers (aka sidecars) that are for example needed to access the databases. Fortunately as a developer the SIP takes care of setting this up for you. | https://kubernetes.io/docs/concepts/workloads/pods/ |
| ingress | An ingress in Kubernetes is an object / rule that manages how external traffic is distributed inside your cluster. For example: Imange you have a pod running in your cluster, by default this pod is not exposed to the "outside" world. Using an ingress you can specify that traffic for https://wiki.vseth.ethz.ch reaches the pod where the VSETH Wiki is running in. You can think of an ingress as a configuration of the cluster wide reverse proxy. | https://kubernetes.io/docs/concepts/services-networking/ingress/ |
| gcloud | Is the command line utility for the google cloud sdk. | https://cloud.google.com/sdk |
Setup
Install the necessary tools
To access the SIP Kubernetes cluster you need to install kubectl (described above) and the google cloud sdk. VSETH uses the Google Cloud Platform for Kubernetes Hosting, the google cloud platform sdk contains the gcloud cli which can be used to authenticate you as a developer and generate access credentials to the Kubernetes cluster.
The installation of these tools is of course platform specific, therefore we link the guides to the official documentation here:
On linux systems it is probably the easiest way to install the tools using Snap.
You can check that the installation was successful by running:
➜ ~ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-09-30T19:31:27Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-gke.401", GitCommit:"eb94c181eea5290e9da1238db02cfef263542f5f", GitTreeState:"clean", BuildDate:"2020-09-09T00:57:35Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}
➜ ~ gcloud version
Google Cloud SDK 312.0.0
alpha 2020.09.25
beta 2020.09.25
bq 2.0.61
core 2020.09.25
gsutil 4.53
kubectl 1.15.11
Version number may be different on your system, which is not a problem as long as you see a similar output, it probably means that you installed the tools correctly.
Authenticate
For your organization to be able to access the Kubernetes Cluster directly, the VSETH ISG needs to give you access. Please check the list of organizations that were already setup to check whether you should have access.
To get credentials for the google cluster you can run:
gcloud auth login gcloud config set project aerial-reef-202214 gcloud container clusters get-credentials --region europe-west4-b vseth-k8s-redzone
The glcoud auth login command should open a browser window, where you need to login with your VSETH GSuite Account (for example lukasre@gsuite.vseth.ch). Logging in with your personal Google Account will not work.
Kubectl
Kubectl is a very extensive utility which you can use to manage all aspects of a Kubernetes Cluster. Since most of the configuration of a deployment is done through the SIP system, developers are restricted to use a subset of the kubectl operations, so they can not accidentally break something. Here is a list of the permissions that are currently configured for developers (on the respective namespaces that they have access to):
todo
This list is Work in progress, so if you need additional permissions to perform an operation as a developer, please contact us a vseth-it-support@ethz.ch and we can check, whether we can add the permission.
The Kubernetes access neees to be initially set up for your organization, you can find a list of the organizations that have been configured here (you need to login to see this page).
Examples
Some examples about what you might find useful to do with kubectl:
List all pods in namespace
To list all the pods that are currently running in a namespace you can run:
kubectl get pod -n $namespace
Which will produce an output similar to this:
➜ ~ kubectl get pod -n vseth-prod NAME READY STATUS RESTARTS AGE buecherboerse-deployment-5694c54b99-jjxzs 2/2 Running 0 12d confluence-deployment-6cf6b5556c-j8lpf 2/2 Running 0 6d18h jira-deployment-8565bf77c8-kt2cq 2/2 Running 0 164m static-deployment-64fd5997cd-knsnh 1/1 Running 0 6d6h static-deployment-64fd5997cd-rm9lt 1/1 Running 0 164m static-deployment-64fd5997cd-txprq 1/1 Running 0 164m
Fields:
- NAME: This is the name of the pod, it is unique for every pod that is created. So if a pod ist restarted a new name will be assigned. Also in the example above you can see that you have an application running with multiple replicas every pod gets a unique name.
- READY: #containers ready / #containers total inside the pod. for example in the example above the buecherboerse pod consists of two containers inside a single pod: in one container the actual application is running and the other is a sidecar container which is used to connect to the database. In the example above all containers inside all pods are running smoothly.
- STATUS: The phase of the pod, an overview over all the phases in a pods lifecycle can be found here.
- RESTARTS: The number of restarts that the pod already has "endured". A high number is usually bad and means that the pod is crashing frequently.
- AGE: How long ago this pod was created.
Display the logs of a running pod
For debugging, particularly useful: you can display and also follow the logs of a running container inside a pod. To display the logs of a particular pod, you first have to get the name of the pod (unique name of the pod), for example using the command from above:
kubectl logs -n vseth-prod buecherboerse-deployment-5694c54b99-jjxzs -c buecherboerse-app
The options for this command:
- -n the namespace the pod is located in
- -c the name of the container of which you want to access logs. You can also specify --all-containers=true to get the logs of all containers in a single pod.
- -f "follow" will keep the command running and display log messages as they come in.
Here you can see an example usage of the command:
➜ ~ kubectl logs -n api-prod peopleapi-deployment-7cb9886bcd-l677v -c peopleapi-app 2020-09-30T16:00:31.769 INFO [cinit] Starting up 2020-09-30T16:00:31.809 INFO [cinit] Config is at /etc/cinit.d/ 2020-09-30T16:00:31.809 INFO [cinit] Parsing config 2020-09-30T16:00:32.013 INFO [cinit] Perform analysis on programs 2020-09-30T16:00:32.013 INFO [cinit] Spawning processes 2020-09-30T16:00:32.017 INFO [cinit] Starting peopleapi 2020-09-30T16:00:32.018 INFO [cinit] Started child 6 2020-09-30T16:00:32.018 INFO [cinit] Starting base-info 2020-09-30T16:00:32.018 INFO [cinit] Started child 7 2020-09-30T16:00:32.339 INFO [base-info] Container built on Tue 08 Sep 2020 02:10:30 AM CEST 2020-09-30T16:00:32.339 INFO [base-info] Name: 2020-09-30T16:00:32.339 INFO [base-info] URL: 2020-09-30T16:00:32.410 INFO [base-info] Current directory: /app 2020-09-30T16:00:32.410 INFO [base-info] Files: 2020-09-30T16:00:32.445 INFO [base-info] total 16 2020-09-30T16:00:32.445 INFO [base-info] drwxr-xr-x 1 app-user app-user 4096 Sep 9 22:42 . 2020-09-30T16:00:32.445 INFO [base-info] drwxr-xr-x 1 root root 4096 Sep 30 16:00 .. 2020-09-30T16:00:32.445 INFO [base-info] -rwxr-xr-x 1 root root 2049 Sep 8 02:09 base-setup.sh 2020-09-30T16:00:32.445 INFO [base-info] drwxr-xr-x 4 root root 4096 Sep 9 22:41 peopleapi 2020-09-30T16:00:32.445 INFO [cinit] Child base-info exited successfully 2020-09-30T16:00:40.046 INFO [peopleapi] log4j:WARN No appenders could be found for logger (org.apache.commons.beanutils.converters.BooleanConverter). 2020-09-30T16:00:40.047 INFO [peopleapi] log4j:WARN Please initialize the log4j system properly. 2020-09-30T16:00:40.048 INFO [peopleapi] log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 2020-09-30T16:00:44.972 INFO [peopleapi] HV000001: Hibernate Validator 6.1.3.Final 2020-09-30T16:00:45.887 INFO [peopleapi] Server started, listening on 9111
Get your output in yaml or json format
You can specify to give the output in yaml or json format, which is more convenient for further processing (for example with jq) by specifying the -o json or -o yaml flag, for example:
➜ ~ kubectl get pod -n vseth-prod -o json
{
....
}
the json output was removed from the example, because it is quite big.
Open a shell in a running container
For debugging it can be useful to open a shell inside a container.
Keep in mind that everything you do inside a container will be reverted as soon as the container restarts (which can happen at any moment), so do not use this approach to configure your application in any way.
You can execute a command inside a running container using the following command:
kubectl exec -it -n vseth-prod buecherboerse-deployment-5694c54b99-jjxzs -c buecherboerse-app -- bash
- -it means that the command is interactive, so it basically open an interactive session. If you do not use this flag you can just execute a single command.
- -n the namespace of the pod
- -c the container inside the pod you want to execute the command in
- bash the command that should be executed inside the container, please note that the respective command needs to be installed inside your image. Bash is installed in all containers based on the vseth base image, but might not be installed on some alpine images etc.
Below you can see an example of opening a bash inside a running container and the executing ps to see a list of all the running processes:
➜ ~ kubectl exec -it -n vseth-prod buecherboerse-deployment-5694c54b99-jjxzs -c buecherboerse-app -- bash root@buecherboerse-deployment-5694c54b99-jjxzs:/app/var/htdocs# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 5624 2932 ? Ss Sep23 1:01 /bin/cinit --config /etc/cinit.d/ root 19 0.0 0.2 253828 30772 ? Ss Sep23 0:48 php-fpm: master process (/etc/php/7.3/fpm/php-fpm.conf) www-data 20 0.0 0.0 11540 5460 ? S Sep23 0:00 nginx: master process /usr/bin/openresty -g daemon off; -p /etc/nginx www-data 21 0.0 0.0 12664 4420 ? S Sep23 1:17 nginx: worker process www-data 285 0.0 0.1 254280 24300 ? S 15:56 0:00 php-fpm: pool www www-data 286 0.0 0.1 254092 19012 ? S 15:56 0:00 php-fpm: pool www www-data 287 0.0 0.1 254288 25604 ? S 15:59 0:00 php-fpm: pool www root 305 0.0 0.0 6996 3640 pts/1 Ss 17:20 0:00 bash root 311 0.0 0.0 10636 3052 pts/1 R+ 17:20 0:00 ps aux root@buecherboerse-deployment-5694c54b99-jjxzs:/app/var/htdocs# exit ➜ ~
Get detailed information about resources (pod, ingress etc.)
The kubectl describe command can be used to get detailed, human-readable information about Kubernetes Resources such as pods or ingresses:
kubectl describe pod -n vseth-prod buecherboerse-deployment-5694c54b99-jjxzs
An example output of the describe command is shown here:
Delete a running pod
Currently it is not possible to restart an application using the sipctl. The easiest way to do this is just to delete the pod using the kubectl delete command.
kubectl delete pod -n vseth-prod buecherboerse-deployment-5694c54b99-jjxzs
After deleting the pod Kubernetes will automatically create a new POD.