1.查看集群資源狀況

集群整體狀態： kubectl cluster-info 更多集群信息： kubectl cluster-info dump 查看資源信息 kubectl describe <資源> <名稱> 查看資源信息 kubectl get pod <Pod名稱> --watch
　　　　　　　　　　　　　　　　-w 監控著

?kubectl api-resources

[root@k8s-m1 ~]# kubectl api-resources NAME SHORTNAMES APIGROUP NAMESPACED KIND bindings true Binding componentstatuses cs false ComponentStatus configmaps cm true ConfigMap endpoints ep true Endpoints events ev true Event limitranges limits true LimitRange namespaces ns false Namespace nodes no false Node persistentvolumeclaims pvc true PersistentVolumeClaim persistentvolumes pv false PersistentVolume pods po true Pod podtemplates true PodTemplate replicationcontrollers rc true ReplicationController resourcequotas quota true ResourceQuota secrets true Secret serviceaccounts sa true ServiceAccount services svc true Service mutatingwebhookconfigurations admissionregistration.k8s.io false MutatingWebhookConfiguration validatingwebhookconfigurations admissionregistration.k8s.io false ValidatingWebhookConfiguration customresourcedefinitions crd,crds apiextensions.k8s.io false CustomResourceDefinition apiservices apiregistration.k8s.io false APIService controllerrevisions apps true ControllerRevision daemonsets ds apps true DaemonSet deployments deploy apps true Deployment replicasets rs apps true ReplicaSet statefulsets sts apps true StatefulSet tokenreviews authentication.k8s.io false TokenReview localsubjectaccessreviews authorization.k8s.io true LocalSubjectAccessReview selfsubjectaccessreviews authorization.k8s.io false SelfSubjectAccessReview selfsubjectrulesreviews authorization.k8s.io false SelfSubjectRulesReview subjectaccessreviews authorization.k8s.io false SubjectAccessReview horizontalpodautoscalers hpa autoscaling true HorizontalPodAutoscaler cronjobs cj batch true CronJob jobs batch true Job certificatesigningrequests csr certificates.k8s.io false CertificateSigningRequest leases coordination.k8s.io true Lease bgpconfigurations crd.projectcalico.org false BGPConfiguration bgppeers crd.projectcalico.org false BGPPeer blockaffinities crd.projectcalico.org false BlockAffinity clusterinformations crd.projectcalico.org false ClusterInformation felixconfigurations crd.projectcalico.org false FelixConfiguration globalnetworkpolicies crd.projectcalico.org false GlobalNetworkPolicy globalnetworksets crd.projectcalico.org false GlobalNetworkSet hostendpoints crd.projectcalico.org false HostEndpoint ipamblocks crd.projectcalico.org false IPAMBlock ipamconfigs crd.projectcalico.org false IPAMConfig ipamhandles crd.projectcalico.org false IPAMHandle ippools crd.projectcalico.org false IPPool kubecontrollersconfigurations crd.projectcalico.org false KubeControllersConfiguration networkpolicies crd.projectcalico.org true NetworkPolicy networksets crd.projectcalico.org true NetworkSet endpointslices discovery.k8s.io true EndpointSlice events ev events.k8s.io true Event ingresses ing extensions true Ingress nodes metrics.k8s.io false NodeMetrics pods metrics.k8s.io true PodMetrics ingressclasses networking.k8s.io false IngressClass ingresses ing networking.k8s.io true Ingress networkpolicies netpol networking.k8s.io true NetworkPolicy runtimeclasses node.k8s.io false RuntimeClass poddisruptionbudgets pdb policy true PodDisruptionBudget podsecuritypolicies psp policy false PodSecurityPolicy clusterrolebindings rbac.authorization.k8s.io false ClusterRoleBinding clusterroles rbac.authorization.k8s.io false ClusterRole rolebindings rbac.authorization.k8s.io true RoleBinding roles rbac.authorization.k8s.io true Role priorityclasses pc scheduling.k8s.io false PriorityClass csidrivers storage.k8s.io false CSIDriver csinodes storage.k8s.io false CSINode storageclasses sc storage.k8s.io false StorageClass volumeattachments storage.k8s.io false VolumeAttachment View Code

NAMESPACED 表示是不是可以被命名空間隔離

查看pod

[root@k8s-m1 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE nginx-f89759699-qjjjb 1/1 Running 0 9h [root@k8s-m1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-f89759699-qjjjb 1/1 Running 0 9h

? [root@k8s-m1 ~]# kubectl describe pod nginx-f89759699-qjjjb?

[root@k8s-m1 ~]# kubectl describe pod nginx-f89759699-qjjjb Name: nginx-f89759699-qjjjb Namespace: default Priority: 0 Node: k8s-n1/10.0.0.24 Start Time: Thu, 30 Jul 2020 21:29:42 +0800 Labels: app=nginxpod-template-hash=f89759699 Annotations: cni.projectcalico.org/podIP: 10.244.215.73/32cni.projectcalico.org/podIPs: 10.244.215.73/32 Status: Running IP: 10.244.215.73 IPs:IP: 10.244.215.73 Controlled By: ReplicaSet/nginx-f89759699 Containers:nginx:Container ID: docker://5cad37326a2ec8b9ac91e83910f664f0587723c6c863222eae702d94755d5b99Image: nginxImage ID: docker-pullable://nginx@sha256:0e188877aa60537d1a1c6484b8c3929cfe09988145327ee47e8e91ddf6f76f5cPort: <none>Host Port: <none>State: RunningStarted: Thu, 30 Jul 2020 21:30:09 +0800Ready: TrueRestart Count: 0Environment: <none>Mounts:/var/run/secrets/kubernetes.io/serviceaccount from default-token-dvcjp (ro) Conditions:Type StatusInitialized TrueReady TrueContainersReady TruePodScheduled True Volumes:default-token-dvcjp:Type: Secret (a volume populated by a Secret)SecretName: default-token-dvcjpOptional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300snode.kubernetes.io/unreachable:NoExecute for 300s Events:Type Reason Age From Message---- ------ ---- ---- -------Warning FailedScheduling 15m (x353 over 9h) default-scheduler 0/3 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 2 node(s) had taint {node.kubernetes.io/unreachable: }, that the pod didn't tolerate.Normal Scheduled 9m57s default-scheduler Successfully assigned default/nginx-f89759699-qjjjb to k8s-n1Normal Pulling 9m56s kubelet, k8s-n1 Pulling image "nginx"Normal Pulled 9m30s kubelet, k8s-n1 Successfully pulled image "nginx"Normal Created 9m30s kubelet, k8s-n1 Created container nginxNormal Started 9m30s kubelet, k8s-n1 Started container nginx View Code # 查看組件的狀態
[root@k8s-m1 ~]# kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health":"true"}

# 查看節點 [root@k8s-m1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-m1 Ready master 6d23h v1.18.0 k8s-n1 Ready <none> 6d23h v1.18.0 k8s-n2 Ready <none> 6d23h v1.18.0

2.監控集群資源利用率

Metrics Server 是一個集群范圍的資源使用情況的數據聚合器。作為一個應用部署在集群中。 Metric server，從毎個節點上 Kubelet AP收集指標，通過 Kubernetes聚合器注冊在 Master APiServer中

默認執行 kubectl top node? 會報錯，需要安裝??Metrics Server?

Metrics Server 的架構

Metrics server部署

wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml

vi?components.yaml

Metrics Server 鏡像在國外需要拉到docker hub 方便拉取(鏡像的地址 默認會去拉國外的image會卡在那里，這里改成國內的)
修改參數 --kubelet-insecure-tls # 允許不安全的tls --kubelet-preferred-address-types=InternalIP # 通過 InternalIP IP訪問 以IP的形式去連接kubelet

?修改后如下圖所示

項目地址 https://github.com/kubernetessigs/metrics-server

1.19+ 在執行yaml文件的時候會有一個警告Warning: apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created這里改一下API的接口 sed 's#apiregistration.k8s.io/v1beta1#apiregistration.k8s.io/v1#g' components.yaml -i heapster 用于匯總所有node節點資源利用率 metrics-serverhpa pod水平擴展 vpa pod橫向擴展- --kubelet-preferred-address-types=InternalIP 以node ip連接kubelet - --kubelet-insecure-tls 跳過tls檢查kubectl get apiservice 查看apiserver聚合層注冊信息

?工作流程：kubectl top -> apiserver -> metrics-server pod -> kubelet(cadvisor) -> cgroups

apiserver聚合層：動態注冊，安全代理，方便第三方應用接入，統一接入

查看Node資源消耗： kubectl top node <node name> 查看Pod資源消耗： kubectl top pod <pod name>[root@k8s-m1 ~]# kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% k8s-m1 108m 5% 1734Mi 45% k8s-n1 47m 2% 480Mi 12% k8s-n2 45m 2% 414Mi 10% [root@k8s-m1 ~]# kubectl top pod NAME CPU(cores) MEMORY(bytes) nginx-f89759699-qjjjb 0m 3Mi

?系統底層 對pod 的資源限制

3.管理K8s組件日志

K8S 系統的組件日志
K8S Cluster里面部署的應用程序日志
? 標準輸出
? 日志文件

systemd 守護進程管理的組件 journalctl -u kubeletPod部罟的組件 kubectl logs 組件名字 -n kube-system系統日志 /var/log/messages
查看組件日志 [root@k8s-m1 chp3]# kubectl logs etcd-k8s-m1 -n kube-system

輸出日志
[root@k8s-m1 ~]# kubectl logs etcd-k8s-m1 -n kube-system > etcd-k8s-m1.log

4.管理K8s應用日志

標準輸出路徑 /var/log/docker/containers/<container-id>/<container-id>-json.log
查看容器標準輸出日志 kubectl logs <Pod名稱> kubectl logs -f <Pod名稱> kubectl logs -f <Pod名稱> -c <容器名稱> 容器內應用日志路徑

將日志文件通過數據卷掛載到宿主機目錄，這樣一來，就可以在宿主機上統一查看日志了。

• 1、deamonset方式在每個節點部署一個日志采集pod完成講解的兩個目錄采集
• 2、sidecar在pod部署一個日志采集容器，通過數據卷共享業務容器日志目錄

練習

將日志掛載到本地

[root@k8s-m1 chp3]# cat pod2.yml apiVersion: v1 kind: Pod metadata:name: my-pod spec:containers:- name: webimage: lizhenliang/nginx-phpvolumeMounts:- name: logsmountPath: /usr/local/nginx/logsvolumes:- name: logshostPath:path: /tmp/logstype: Directory [root@k8s-n2 ~]# tail -f /tmp/logs/access.log 10.244.42.128 - - [30/Jul/2020:23:18:07 +0800] "GET / HTTP/1.1" 403 146 "-" "curl/7.29.0" 10.244.42.128 - - [30/Jul/2020:23:18:08 +0800] "GET / HTTP/1.1" 403 146 "-" "curl/7.29.0" 10.244.42.128 - - [30/Jul/2020:23:18:09 +0800] "GET / HTTP/1.1" 403 146 "-" "curl/7.29.0" 10.244.42.128 - - [30/Jul/2020:23:18:10 +0800] "GET / HTTP/1.1" 403 146 "-" "curl/7.29.0"

放到kubelet 的日志卷

[root@k8s-m1 chp3]# cat pod-kube-v.yml apiVersion: v1 kind: Pod metadata:name: my-pod-1 spec:containers:- name: webimage: lizhengliang/nginx-phpvolumeMounts:- name: logsmountPath: /usr/local/nginx/logsvolumes:- name: logsemptyDir: {} 1、查看pod日志，并將日志中Error的行記錄到指定文件 ? pod名稱：web ? 文件：/opt/webkubectl run web --image=nginx -n cka kubectl get pod -n cka -o wide kubectl logs web -n cka | grep "\[error\]" kubectl logs web -n cka | grep "\[error\]" > /opt/web.log2、查看指定標簽使用cpu最高的pod，并記錄到到指定文件 ? 標簽：app=web ? 文件：/opt/cpukubectl run web1 --image=nginx -l app=web -n cka kubectl run web2 --image=nginx -l app=web -n cka kubectl run web3 --image=nginx -l app=web -n cka kubectl top pod -n cka -l app=web --sort-by=cpu > /opt/cpu.log 與50位技術專家面對面20年技術見證，附贈技術全景圖

總結

以上是生活随笔為你收集整理的第3章：Kubernetes监控与日志管理的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。