API Server簡(jiǎn)介 k8s API Server提供了k8s各類(lèi)資源對(duì)象(pod,RC,Service等)的增刪改查及watch等HTTP Rest接口,是整個(gè)系統(tǒng)的數(shù)據(jù)總線和數(shù)據(jù)中心。
kubernetes API Server的功能:
提供了集群管理的REST API接口(包括認(rèn)證授權(quán)、數(shù)據(jù)校驗(yàn)以及集群狀態(tài)變更); 提供其他模塊之間的數(shù)據(jù)交互和通信的樞紐(其他模塊通過(guò)API Server查詢或修改數(shù)據(jù),只有API Server才直接操作etcd); 是資源配額控制的入口; 擁有完備的集群安全機(jī)制. kube-apiserver工作原理圖
如何訪問(wèn)kubernetes API k8s通過(guò)kube-apiserver這個(gè)進(jìn)程提供服務(wù),該進(jìn)程運(yùn)行在單個(gè)k8s-master節(jié)點(diǎn)上。默認(rèn)有兩個(gè)端口。
k8s通過(guò)kube-apiserver這個(gè)進(jìn)程提供服務(wù),該進(jìn)程運(yùn)行在單個(gè)k8s-master節(jié)點(diǎn)上。默認(rèn)有兩個(gè)端口。
1.?本地端口 該端口用于接收HTTP請(qǐng)求; 該端口默認(rèn)值為8080,可以通過(guò)API Server的啟動(dòng)參數(shù)“--insecure-port”的值來(lái)修改默認(rèn)值; 默認(rèn)的IP地址為“l(fā)ocalhost”,可以通過(guò)啟動(dòng)參數(shù)“--insecure-bind-address”的值來(lái)修改該IP地址; 非認(rèn)證或授權(quán)的HTTP請(qǐng)求通過(guò)該端口訪問(wèn)API Server。 2.安全端口 該端口默認(rèn)值為6443,可通過(guò)啟動(dòng)參數(shù)“--secure-port”的值來(lái)修改默認(rèn)值; 默認(rèn)IP地址為非本地(Non-Localhost)網(wǎng)絡(luò)端口,通過(guò)啟動(dòng)參數(shù)“--bind-address”設(shè)置該值; 該端口用于接收HTTPS請(qǐng)求; 用于基于Tocken文件或客戶端證書(shū)及HTTP Base的認(rèn)證; 用于基于策略的授權(quán); 默認(rèn)不啟動(dòng)HTTPS安全訪問(wèn)控制。 kubernetes API訪問(wèn)方式 Kubernetes REST API可參考https://kubernetes.io/docs/api-reference/v1.6/
1.?curl curl localhost:8080/api
curl localhost:8080/api/v1/pods
curl localhost:8080/api/v1/services
curl localhost:8080/api/v1/replicationcontrollers
2.?Kubectl Proxy Kubectl Proxy代理程序既能作為API Server的反向代理,也能作為普通客戶端訪問(wèn)API Server的代理。通過(guò)master節(jié)點(diǎn)的8080端口來(lái)啟動(dòng)該代理程序。
kubectl proxy --port=8080 &
具體見(jiàn)kubectl proxy --help
3.?kubectl客戶端 命令行工具kubectl客戶端,通過(guò)命令行參數(shù)轉(zhuǎn)換為對(duì)API Server的REST API調(diào)用,并將調(diào)用結(jié)果輸出。
命令格式:kubectl [command] [options]
具體可參考Kubernetes常用命令
4.?編程方式調(diào)用 使用場(chǎng)景:
1、運(yùn)行在Pod里的用戶進(jìn)程調(diào)用kubernetes API,通常用來(lái)實(shí)現(xiàn)分布式集群搭建的目標(biāo)。
2、開(kāi)發(fā)基于kubernetes的管理平臺(tái),比如調(diào)用kubernetes API來(lái)完成Pod、Service、RC等資源對(duì)象的圖形化創(chuàng)建和管理界面。可以使用kubernetes提供的Client Library。
具體可參考https://github.com/kubernetes/client-go。
通過(guò)API Server訪問(wèn)Node、Pod和Service k8s API Server最主要的REST接口是資源對(duì)象的增刪改查,另外還有一類(lèi)特殊的REST接口—k8s Proxy API接口,這類(lèi)接口的作用是代理REST請(qǐng)求,即kubernetes API Server把收到的REST請(qǐng)求轉(zhuǎn)發(fā)到某個(gè)Node上的kubelet守護(hù)進(jìn)程的REST端口上,由該kubelet進(jìn)程負(fù)責(zé)響應(yīng)。
1.?Node相關(guān)接口 關(guān)于Node相關(guān)的接口的REST路徑為:/api/v1/proxy/nodes/{name},其中{name}為節(jié)點(diǎn)的名稱(chēng)或IP地址。
/api/v1/proxy/nodes/{name}/pods/????#列出指定節(jié)點(diǎn)內(nèi)所有Pod的信息
/api/v1/proxy/nodes/{name}/stats/???#列出指定節(jié)點(diǎn)內(nèi)物理資源的統(tǒng)計(jì)信息
/api/v1/prxoy/nodes/{name}/spec/????#列出指定節(jié)點(diǎn)的概要信息
這里獲取的Pod信息來(lái)自Node而非etcd數(shù)據(jù)庫(kù),兩者時(shí)間點(diǎn)可能存在偏差。如果在kubelet進(jìn)程啟動(dòng)時(shí)加--enable-debugging-handles=true參數(shù),那么kubernetes Proxy API還會(huì)增加以下接口:
/api/v1/proxy/nodes/{name}/run??????#在節(jié)點(diǎn)上運(yùn)行某個(gè)容器
/api/v1/proxy/nodes/{name}/exec?????#在節(jié)點(diǎn)上的某個(gè)容器中運(yùn)行某條命令
/api/v1/proxy/nodes/{name}/attach???#在節(jié)點(diǎn)上attach某個(gè)容器
/api/v1/proxy/nodes/{name}/portForward???#實(shí)現(xiàn)節(jié)點(diǎn)上的Pod端口轉(zhuǎn)發(fā)
/api/v1/proxy/nodes/{name}/logs?????#列出節(jié)點(diǎn)的各類(lèi)日志信息
/api/v1/proxy/nodes/{name}/metrics??#列出和該節(jié)點(diǎn)相關(guān)的Metrics信息
/api/v1/proxy/nodes/{name}/runningpods??#列出節(jié)點(diǎn)內(nèi)運(yùn)行中的Pod信息
/api/v1/proxy/nodes/{name}/debug/pprof??#列出節(jié)點(diǎn)內(nèi)當(dāng)前web服務(wù)的狀態(tài),包括CPU和內(nèi)存的使用情況
2.?Pod相關(guān)接口 /api/v1/proxy/namespaces/{namespace}/pods/{name}/{path:*}??????#訪問(wèn)pod的某個(gè)服務(wù)接口
/api/v1/proxy/namespaces/{namespace}/pods/{name}???????????????#訪問(wèn)Pod
#以下寫(xiě)法不同,功能一樣
/api/v1/namespaces/{namespace}/pods/{name}/proxy/{path:*}??????#訪問(wèn)pod的某個(gè)服務(wù)接口
/api/v1/namespaces/{namespace}/pods/{name}/proxy???????????????#訪問(wèn)Pod
3.?Service相關(guān)接口 /api/v1/proxy/namespaces/{namespace}/services/{name}
Pod的proxy接口的作用:在kubernetes集群之外訪問(wèn)某個(gè)pod容器的服務(wù)(HTTP服務(wù)),可以用Proxy API實(shí)現(xiàn),這種場(chǎng)景多用于管理目的,比如逐一排查Service的Pod副本,檢查哪些Pod的服務(wù)存在異常問(wèn)題。
集群功能模塊之間的通信 kubernetes API Server作為集群的核心,負(fù)責(zé)集群各功能模塊之間的通信,集群內(nèi)各個(gè)功能模塊通過(guò)API Server將信息存入etcd,當(dāng)需要獲取和操作這些數(shù)據(jù)時(shí),通過(guò)API Server提供的REST接口(GET\LIST\WATCH方法)來(lái)實(shí)現(xiàn),從而實(shí)現(xiàn)各模塊之間的信息交互。
1.?kubelet與API Server交互 每個(gè)Node節(jié)點(diǎn)上的kubelet定期就會(huì)調(diào)用API Server的REST接口報(bào)告自身狀態(tài),API Server接收這些信息后,將節(jié)點(diǎn)狀態(tài)信息更新到etcd中。kubelet也通過(guò)API Server的Watch接口監(jiān)聽(tīng)Pod信息,從而對(duì)Node機(jī)器上的POD進(jìn)行管理。
監(jiān)聽(tīng)信息
kubelet動(dòng)作
備注
新的POD副本被調(diào)度綁定到本節(jié)點(diǎn) 執(zhí)行POD對(duì)應(yīng)的容器的創(chuàng)建和啟動(dòng)邏輯 ? POD對(duì)象被刪除 刪除本節(jié)點(diǎn)上相應(yīng)的POD容器 ? 修改POD信息 修改本節(jié)點(diǎn)的POD容器 ?
2.?kube-controller-manager與API Server交互 kube-controller-manager中的Node Controller模塊通過(guò)API Server提供的Watch接口,實(shí)時(shí)監(jiān)控Node的信息,并做相應(yīng)處理。
3.?kube-scheduler與API Server交互 Scheduler通過(guò)API Server的Watch接口監(jiān)聽(tīng)到新建Pod副本的信息后,它會(huì)檢索所有符合該P(yáng)od要求的Node列表,開(kāi)始執(zhí)行Pod調(diào)度邏輯。調(diào)度成功后將Pod綁定到目標(biāo)節(jié)點(diǎn)上。
?
?
API Server參數(shù)介紹 API Server 主要是和 etcd 打交道,并且對(duì)外提供 HTTP 服務(wù),以及進(jìn)行安全控制,因此它的命令行提供的參數(shù)也主要和這幾個(gè)方面有關(guān)。下面是一些比較重要的參數(shù)以及說(shuō)明(不同版本參數(shù)可能會(huì)有不同):
參數(shù)含義默認(rèn)值 –advertise-address 通過(guò)該 ip 地址向集群其他節(jié)點(diǎn)公布 api server 的信息,必須能夠被其他節(jié)點(diǎn)訪問(wèn) nil –allow-privileged 是否允許 privileged 容器運(yùn)行 false –admission-control 準(zhǔn)入控制 AlwaysAdmit –authorization-mode 授權(quán)模式 ,安全接口上的授權(quán) AlwaysAllow –bind-address HTTPS 安全接口的監(jiān)聽(tīng)地址 0.0.0.0 –secure-port HTTPS 安全接口的監(jiān)聽(tīng)端口 6443 –cert-dir TLS 證書(shū)的存放目錄 /var/run/kubernetes –etcd-prefix 信息存放在 etcd 中地址的前綴 “/registry” –etcd-servers 逗號(hào)分割的 etcd server 地址 [] –insecure-bind-address HTTP 訪問(wèn)的地址 127.0.0.1 –insecure-port HTTP 訪問(wèn)的端口 8080 –log-dir 日志存放的目錄 ? –service-cluster-ip-range service 要使用的網(wǎng)段,使用 CIDR 格式,參考 kubernetes 中 service 的定義 ?
API Server安裝和運(yùn)行 API Server 是通過(guò)提供的?kube-apiserver?二進(jìn)制文件直接運(yùn)行的,下面的例子指定了 service 分配的 ip 范圍,etcd 的地址,和對(duì)外提供服務(wù)的 ip 地址:
/usr/bin/kube-apiserver \ --service-cluster-ip-range=10.20.0.1/24 \ --etcd-servers=http://127.0.0.1:2379 \ --advertise-address=192.168.8.100 \ --bind-address=192.168.8.100 \ --insecure-bind-address=192.168.8.100 \ --v=4 直接訪問(wèn)?8080?端口,API Server 會(huì)返回它提供了哪些接口:
[root@localhost vagrant]# curl http://192.168.8.100:8080 { "paths": [ "/api", "/api/v1", "/apis", "/apis/apps", "/apis/apps/v1alpha1", "/apis/autoscaling", "/apis/autoscaling/v1", "/apis/batch", "/apis/batch/v1", "/apis/batch/v2alpha1", "/apis/extensions", "/apis/extensions/v1beta1", "/apis/policy", "/apis/policy/v1alpha1", "/apis/rbac.authorization.k8s.io", "/apis/rbac.authorization.k8s.io/v1alpha1", "/healthz", "/healthz/ping", "/logs/", "/metrics", "/swaggerapi/", "/ui/", "/version" ] } 而目前最重要的路徑是?/api/v1,里面包含了 kubernetes 所有資源的操作,比如下面的 nodes:
? ~ http http://192.168.8.100:8080/api/v1/nodes HTTP/1.1 200 OK Content-Length: 112 Content-Type: application/json Date: Thu, 08 Sep 2016 08:14:45 GMT { "apiVersion": "v1", "items": [], "kind": "NodeList", "metadata": { "resourceVersion": "12", "selfLink": "/api/v1/nodes" } } ?API 以 json 的形式返回,會(huì)通過(guò)?apiVersion?來(lái)說(shuō)明 API 版本號(hào),kind?說(shuō)明請(qǐng)求的是什么資源。不過(guò)這里面的內(nèi)容是空的,因?yàn)槟壳斑€沒(méi)有任何 kubelet 節(jié)點(diǎn)接入到我們的 API Server。對(duì)應(yīng)的,pod 也是空的:
? ~ http http://192.168.8.100:8080/api/v1/pods HTTP/1.1 200 OK Content-Length: 110 Content-Type: application/json Date: Thu, 08 Sep 2016 08:18:53 GMT { "apiVersion": "v1", "items": [], "kind": "PodList", "metadata": { "resourceVersion": "12", "selfLink": "/api/v1/pods" } } 添加節(jié)點(diǎn) 添加節(jié)點(diǎn)也非常簡(jiǎn)單,啟動(dòng) kubelet 的時(shí)候使用 --api-servers 指定要接入的 API Server 就行。kubelet 啟動(dòng)之后,會(huì)把自己注冊(cè)到指定的 API Server,然后監(jiān)聽(tīng) API 對(duì)應(yīng) pod 的變化,根據(jù) API 中 pod 的實(shí)際信息來(lái)管理節(jié)點(diǎn)上 pod 的生命周期。
現(xiàn)在訪問(wèn)?/api/v1/nodes?就能看到已經(jīng)添加進(jìn)來(lái)的節(jié)點(diǎn):
? ~ http http://192.168.8.100:8080/api/v1/nodes HTTP/1.1 200 OK Content-Type: application/json Date: Thu, 08 Sep 2016 08:27:44 GMT Transfer-Encoding: chunked { "apiVersion": "v1", "items": [ { "metadata": { "annotations": { "volumes.kubernetes.io/controller-managed-attach-detach": "true" }, "creationTimestamp": "2016-09-08T08:23:01Z", "labels": { "beta.kubernetes.io/arch": "amd64", "beta.kubernetes.io/os": "linux", "kubernetes.io/hostname": "192.168.8.100" }, "name": "192.168.8.100", "resourceVersion": "65", "selfLink": "/api/v1/nodes/192.168.8.100", "uid": "74e16eba-759d-11e6-b463-080027c09e5b" }, "spec": { "externalID": "192.168.8.100" }, "status": { "addresses": [ { "address": "192.168.8.100", "type": "LegacyHostIP" }, { "address": "192.168.8.100", "type": "InternalIP" } ], "allocatable": { "alpha.kubernetes.io/nvidia-gpu": "0", "cpu": "1", "memory": "502164Ki", "pods": "110" }, "capacity": { "alpha.kubernetes.io/nvidia-gpu": "0", "cpu": "1", "memory": "502164Ki", "pods": "110" }, "conditions": [ { "lastHeartbeatTime": "2016-09-08T08:27:36Z", "lastTransitionTime": "2016-09-08T08:23:01Z", "message": "kubelet has sufficient disk space available", "reason": "KubeletHasSufficientDisk", "status": "False", "type": "OutOfDisk" }, { "lastHeartbeatTime": "2016-09-08T08:27:36Z", "lastTransitionTime": "2016-09-08T08:23:01Z", "message": "kubelet has sufficient memory available", "reason": "KubeletHasSufficientMemory", "status": "False", "type": "MemoryPressure" }, { "lastHeartbeatTime": "2016-09-08T08:27:36Z", "lastTransitionTime": "2016-09-08T08:24:56Z", "message": "kubelet is posting ready status", "reason": "KubeletReady", "status": "True", "type": "Ready" } ], "daemonEndpoints": { "kubeletEndpoint": { "Port": 10250 } }, "images": [ { "names": [ "172.16.1.41:5000/nginx:latest" ], "sizeBytes": 425626718 }, { "names": [ "172.16.1.41:5000/hyperkube:v0.18.2" ], "sizeBytes": 207121551 }, { "names": [ "172.16.1.41:5000/etcd:v3.0.4" ], "sizeBytes": 43302056 }, { "names": [ "172.16.1.41:5000/busybox:latest" ], "sizeBytes": 1092588 }, { "names": [ "172.16.1.41:5000/google_containers/pause:0.8.0" ], "sizeBytes": 241656 } ], "nodeInfo": { "architecture": "amd64", "bootID": "48955926-11dd-4ad3-8bb0-2585b1c9215d", "containerRuntimeVersion": "docker://1.10.3", "kernelVersion": "3.10.0-123.13.1.el7.x86_64", "kubeProxyVersion": "v1.3.1-beta.0.6+fbf3f3e5292fb0", "kubeletVersion": "v1.3.1-beta.0.6+fbf3f3e5292fb0", "machineID": "b9597c4ae5f24494833d35e806e00b29", "operatingSystem": "linux", "osImage": "CentOS Linux 7 (Core)", "systemUUID": "823EB67A-057E-4EFF-AE7F-A758140CD2F7" } } } ], "kind": "NodeList", "metadata": { "resourceVersion": "65", "selfLink": "/api/v1/nodes" } } ?我們可以看到,kubelet 收集了很多關(guān)于自身節(jié)點(diǎn)的信息,這些信息也會(huì)不斷更新。這些信息里面不僅包含節(jié)點(diǎn)的系統(tǒng)信息(系統(tǒng)架構(gòu),操作系統(tǒng)版本,內(nèi)核版本等)、還有鏡像信息(節(jié)點(diǎn)上有哪些已經(jīng)下載的 docker 鏡像)、資源信息(Memory 和 Disk 的總量和可用量)、以及狀態(tài)信息(是否正常,可以分配 pod等)。
和 API Server 通信 編寫(xiě)的 yaml 文件轉(zhuǎn)換成 json 格式,保存到文件里。主要注意的是,我們指定了 nodeName 的名字,這個(gè)名字必須和之前通過(guò)?/api/v1/nodes?得到的結(jié)果中?metadata.labels.kubernetes.io/hostname?保持一致:
[root@localhost vagrant]# cat nginx_pod.yml apiVersion: v1 kind: Pod metadata: name: nginx-server spec: NodeName: 192.168.8.100 containers: - name: nginx-server image: 172.16.1.41:5000/nginx ports: - containerPort: 80 volumeMounts: - mountPath: /var/log/nginx name: nginx-logs - name: log-output image: 172.16.1.41:5000/busybox command: - bin/sh args: [-c, 'tail -f /logdir/access.log'] volumeMounts: - mountPath: /logdir name: nginx-logs volumes: - name: nginx-logs emptyDir: {} ?
使用 curl 執(zhí)行 POST 請(qǐng)求,設(shè)置頭部?jī)?nèi)容為?application/json,傳過(guò)去文件中的 json 值,可以看到應(yīng)答(其中?status?為?pending,表示以及接收到請(qǐng)求,正在準(zhǔn)備處理):
# curl -s -X POST -H "Content-Type: application/json" http://192.168.8.100:8080/api/v1/namespaces/default/pods --data @nginx_pod.json { "kind": "Pod", "apiVersion": "v1", "metadata": { "name": "nginx-server", "namespace": "default", "selfLink": "/api/v1/namespaces/default/pods/nginx-server", "uid": "888e95d0-75a9-11e6-b463-080027c09e5b", "resourceVersion": "573", "creationTimestamp": "2016-09-08T09:49:28Z" }, "spec": { "volumes": [ { "name": "nginx-logs", "emptyDir": {} } ], "containers": [ { "name": "nginx-server", "image": "172.16.1.41:5000/nginx", "ports": [ { "containerPort": 80, "protocol": "TCP" } ], "resources": {}, "volumeMounts": [ { "name": "nginx-logs", "mountPath": "/var/log/nginx" } ], "terminationMessagePath": "/dev/termination-log", "imagePullPolicy": "Always" } ], "restartPolicy": "Always", "terminationGracePeriodSeconds": 30, "dnsPolicy": "ClusterFirst", "nodeName": "192.168.8.100", "securityContext": {} }, "status": { "phase": "Pending" } } 返回中包含了我們提交 pod 的信息,并且添加了?status、metadata?等額外信息。
等一段時(shí)間去查詢 pod,就可以看到 pod 的狀態(tài)已經(jīng)更新了:
? http http://192.168.8.100:8080/api/v1/namespaces/default/pods HTTP/1.1 200 OK Content-Type: application/json Date: Thu, 08 Sep 2016 09:51:29 GMT Transfer-Encoding: chunked { "apiVersion": "v1", "items": [ { "metadata": { "creationTimestamp": "2016-09-08T09:49:28Z", "name": "nginx-server", "namespace": "default", "resourceVersion": "592", "selfLink": "/api/v1/namespaces/default/pods/nginx-server", "uid": "888e95d0-75a9-11e6-b463-080027c09e5b" }, "spec": { "containers": [ { "image": "172.16.1.41:5000/nginx", "imagePullPolicy": "Always", "name": "nginx-server", "ports": [ { "containerPort": 80, "protocol": "TCP" } ], "resources": {}, "terminationMessagePath": "/dev/termination-log", "volumeMounts": [ { "mountPath": "/var/log/nginx", "name": "nginx-logs" } ] }, { "args": [ "-c", "tail -f /logdir/access.log" ], "command": [ "bin/sh" ], "image": "172.16.1.41:5000/busybox", "imagePullPolicy": "Always", "name": "log-output", "resources": {}, "terminationMessagePath": "/dev/termination-log", "volumeMounts": [ { "mountPath": "/logdir", "name": "nginx-logs" } ] } ], "dnsPolicy": "ClusterFirst", "nodeName": "192.168.8.100", "restartPolicy": "Always", "securityContext": {}, "terminationGracePeriodSeconds": 30, "volumes": [ { "emptyDir": {}, "name": "nginx-logs" } ] }, "status": { "conditions": [ { "lastProbeTime": null, "lastTransitionTime": "2016-09-08T09:49:28Z", "status": "True", "type": "Initialized" }, { "lastProbeTime": null, "lastTransitionTime": "2016-09-08T09:49:44Z", "status": "True", "type": "Ready" }, { "lastProbeTime": null, "lastTransitionTime": "2016-09-08T09:49:44Z", "status": "True", "type": "PodScheduled" } ], "containerStatuses": [ { "containerID": "docker://8b79eeea60f27b6d3f0a19cbd1b3ee3f83709bcf56574a6e1124c69a6376972d", "image": "172.16.1.41:5000/busybox", "imageID": "docker://sha256:8c566faa3abdaebc33d40c1b5e566374c975d17754c69370f78c00c162c1e075", "lastState": {}, "name": "log-output", "ready": true, "restartCount": 0, "state": { "running": { "startedAt": "2016-09-08T09:49:43Z" } } }, { "containerID": "docker://96e64cdba7b05d4e30710a20e958ff5b8f1f359c8d16d32622b36f0df0cb353c", "image": "172.16.1.41:5000/nginx", "imageID": "docker://sha256:51d764c1fd358ce81fd0e728436bd0175ff1f3fd85fc5d1a2f9ba3e7dc6bbaf6", "lastState": {}, "name": "nginx-server", "ready": true, "restartCount": 0, "state": { "running": { "startedAt": "2016-09-08T09:49:36Z" } } } ], "hostIP": "192.168.8.100", "phase": "Running", "podIP": "172.17.0.2", "startTime": "2016-09-08T09:49:28Z" } } ], "kind": "PodList", "metadata": { "resourceVersion": "602", "selfLink": "/api/v1/namespaces/default/pods" } } 可以看到 pod 已經(jīng)在運(yùn)行,并且給分配了 ip:172.17.0.2,通過(guò) curl 也可以訪問(wèn)它的服務(wù):
[root@localhost vagrant]# curl -s http://172.17.0.2 | head -n 5 <!DOCTYPE html> <html> <head> <title>Welcome to nginx on Debian!</title> <style> kubectl -s http:
轉(zhuǎn)載于:https://www.cnblogs.com/Su-per-man/p/10942783.html
總結(jié)
以上是生活随笔 為你收集整理的k8s 组件介绍-API Server 的全部?jī)?nèi)容,希望文章能夠幫你解決所遇到的問(wèn)題。
如果覺(jué)得生活随笔 網(wǎng)站內(nèi)容還不錯(cuò),歡迎將生活随笔 推薦給好友。