Kubernets node
1. 准备环境¶
ip | 操作系统 | 角色 | 安装软件 | 主机名 |
---|---|---|---|---|
192.168.186.139 | centos7.6_x64 | master1 | docker | k8s-master01 |
192.168.186.141 | centos7.6_x64 | node1 | docker | k8s-node01 |
192.168.186.142 | centos7.6_x64 | node2 | docker | k8s-node02 |
1. 安装好docker 2. 配置docker加速器[dao cloud]
2. 部署node¶
2.1 原理图¶
Master apiserver启用TLS认证后,Node节点kubelet组件想要加入集群,必须使用CA签发的有效证书才能与apiserver通信,当Node节点很多时,签署证书是一件很繁琐的事情,因此有了TLS Bootstrapping机制,kubelet会以一个低权限用户自动向apiserver申请证书,kubelet的证书由apiserver动态签署。认证大致工作流程如图所示:

2.2 创建和分发证书¶
1. 将kubelet-bootstrap用户绑定到系统集群角色 kubectl create clusterrolebinding kubelet-bootstrap \ --clusterrole=system:node-bootstrapper \ --user=kubelet-bootstrap 2. 创建kubeconfig文件 3. 部署kubelet,kube-proxy组件
kubeconfig.sh 修改后
[root@k8s-master01 k8s-cert]# cat kubeconfig.sh # 创建 TLS Bootstrapping Token #BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ') BOOTSTRAP_TOKEN=0fb61c46f8991b718eb38d27b605b008 #cat > token.csv <<EOF #${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap" #EOF #---------------------- APISERVER=$1 SSL_DIR=$2 # 创建kubelet bootstrapping kubeconfig export KUBE_APISERVER="https://$APISERVER:6443" # 设置集群参数 kubectl config set-cluster kubernetes \ --certificate-authority=$SSL_DIR/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=bootstrap.kubeconfig # 设置客户端认证参数 kubectl config set-credentials kubelet-bootstrap \ --token=${BOOTSTRAP_TOKEN} \ --kubeconfig=bootstrap.kubeconfig # 设置上下文参数 kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=bootstrap.kubeconfig # 设置默认上下文 kubectl config use-context default --kubeconfig=bootstrap.kubeconfig #---------------------- # 创建kube-proxy kubeconfig文件 kubectl config set-cluster kubernetes \ --certificate-authority=$SSL_DIR/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-credentials kube-proxy \ --client-certificate=$SSL_DIR/kube-proxy.pem \ --client-key=$SSL_DIR/kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-context default \ --cluster=kubernetes \ --user=kube-proxy \ --kubeconfig=kube-proxy.kubeconfig kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
详细部署
kubectl create clusterrolebinding kubelet-bootstrap \ --clusterrole=system:node-bootstrapper \ --user=kubelet-bootstrap 生成kubeconfig证书 bash kubeconfig.sh 192.168.186.139 /root/k8s/k8s-cert/ cd /root/soft/kubernetes/server/bin scp -r bootstrap.kubeconfig kube-proxy.kubeconfig root@192.168.186.141:/opt/kubernetes/cfg/ scp -r bootstrap.kubeconfig kube-proxy.kubeconfig root@192.168.186.142:/opt/kubernetes/cfg/
kubeconfig是链接master相关信息,kublet需要bootstrap.kubeconfig,kubeproxy需要kube-proxy.kubeconfig
删除 kubectl delete clusterrolebinding kubelet-bootstrap 详细步骤
[root@k8s-master01 k8s]# kubectl create clusterrolebinding kubelet-bootstrap \ > --clusterrole=system:node-bootstrapper \ > --user=kubelet-bootstrap clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created [root@k8s-master01 k8s-cert]# cat bootstrap.kubeconfig apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2akNDQXFhZ0F3SUJBZ0lVS2p3bkhQeXVMdkhWMW0wWndYUENDUUUwSz E0d0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU1TURReE9EQTNNekV3TUZvWERUSTBNRFF4TmpBM016RXdNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbGFXcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByZFdKbGNtNWxkR1Z6Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBeTNoSTFFek5TakJsSzJBRWdpSkMKOVdvQzluRXorVDh3NldWTEZ2Y0N4bGEyeVpHSzFtaVNzOHpHMGk3SStiTW1CV2paV0tQaVdkbzlGQlJqVFJmNQpiR01nWHRKQ3JOUlY2WWhVeEM3UkFBVTlpMXp1OWswbkJSNFBKVngxSGg4TzJ2MStucXhyZ200YzRTUzVXNW5ICkZDNmlHOGpsYzVEbDA3QUx1ZG1seXpoVTlTcFFtQW9DY1pVbUFHR3c5NUNkam9pWFFHaVF5MnVVK1dTSHhYakoKVGhtSmx3MTZ6WWY0SGwzK2FFZ3lqRDE2LzNYbC9OYm1qTDhwTzRDWjBMbncwV3RCQTc5cWVTSWhXNkxGUVB4MwpENDNzQmJFUDRrWElRSUNSTnRLSUxYU0RrSVQrNUNIcjg4MTJoU2hpQzRqY2hFY1pmWk1YQXJQUXF2cHN2QlZHCnF3SURBUUFCbzJZd1pEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFqQWQKQmdOVkhRNEVGZ1FVYVJ6OEl5aUtCSGVnODdQQzhQZFovcjAxRytnd0h3WURWUjBqQkJnd0ZvQVVhUno4SXlpSwpCSGVnODdQQzhQZFovcjAxRytnd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFFZjQ2N0I4YldvWi9ERmhQN3pGCmFieFFySTRIaGFhd1ZoK3RXaUNGUFkwdTh6ZWJtZGtGS2NEY000Q3QvWlNzaldhZEhVdXQ4TkZianA1K1E5UGEKTnRUWUFKR3U3NnNpODlxSHpmTDhIdmJjZWpNNlE1V29mR3h3NkFPWVNjVDIzdmlibVZyZXQ0b1J0dng1TmRidApJcFZCY0NxL0FyMDZNb0VIaVRvaCtMejZWUjFGZlA4VEJMNThqVWs2Snh2dEVKbnNicldhQ21KcENwcSs4ZDJnCmtDemhzK1p5WUtvUElMK3JyUStycXYzMVpFSDF0a09QTHhCVFlUUDhwZEZqQUk3MFdlNE9RTkpjaWdRZ2hhaVgKb1ZiUTZsS2xYdUhUSkJOc0l2U0o0azlHL3pObGFGenZTUVFGWG03NEQrYXVsbEE4SEdPUUEvU0F4Tys3Q2p4Nwowbmc9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://192.168.186.139:6443 name: kubernetes contexts: - context: cluster: kubernetes user: kubelet-bootstrap name: default current-context: default kind: Config preferences: {} users: - name: kubelet-bootstrap user: token: 0fb61c46f8991b718eb38d27b605b008 [root@k8s-master01 k8s-cert]# cat kube-proxy.kubeconfig apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2akNDQXFhZ0F3SUJBZ0lVS2p3bkhQeXVMdkhWMW0wWndYUENDUUUwSz E0d0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU1TURReE9EQTNNekV3TUZvWERUSTBNRFF4TmpBM016RXdNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbGFXcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByZFdKbGNtNWxkR1Z6Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBeTNoSTFFek5TakJsSzJBRWdpSkMKOVdvQzluRXorVDh3NldWTEZ2Y0N4bGEyeVpHSzFtaVNzOHpHMGk3SStiTW1CV2paV0tQaVdkbzlGQlJqVFJmNQpiR01nWHRKQ3JOUlY2WWhVeEM3UkFBVTlpMXp1OWswbkJSNFBKVngxSGg4TzJ2MStucXhyZ200YzRTUzVXNW5ICkZDNmlHOGpsYzVEbDA3QUx1ZG1seXpoVTlTcFFtQW9DY1pVbUFHR3c5NUNkam9pWFFHaVF5MnVVK1dTSHhYakoKVGhtSmx3MTZ6WWY0SGwzK2FFZ3lqRDE2LzNYbC9OYm1qTDhwTzRDWjBMbncwV3RCQTc5cWVTSWhXNkxGUVB4MwpENDNzQmJFUDRrWElRSUNSTnRLSUxYU0RrSVQrNUNIcjg4MTJoU2hpQzRqY2hFY1pmWk1YQXJQUXF2cHN2QlZHCnF3SURBUUFCbzJZd1pEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFqQWQKQmdOVkhRNEVGZ1FVYVJ6OEl5aUtCSGVnODdQQzhQZFovcjAxRytnd0h3WURWUjBqQkJnd0ZvQVVhUno4SXlpSwpCSGVnODdQQzhQZFovcjAxRytnd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFFZjQ2N0I4YldvWi9ERmhQN3pGCmFieFFySTRIaGFhd1ZoK3RXaUNGUFkwdTh6ZWJtZGtGS2NEY000Q3QvWlNzaldhZEhVdXQ4TkZianA1K1E5UGEKTnRUWUFKR3U3NnNpODlxSHpmTDhIdmJjZWpNNlE1V29mR3h3NkFPWVNjVDIzdmlibVZyZXQ0b1J0dng1TmRidApJcFZCY0NxL0FyMDZNb0VIaVRvaCtMejZWUjFGZlA4VEJMNThqVWs2Snh2dEVKbnNicldhQ21KcENwcSs4ZDJnCmtDemhzK1p5WUtvUElMK3JyUStycXYzMVpFSDF0a09QTHhCVFlUUDhwZEZqQUk3MFdlNE9RTkpjaWdRZ2hhaVgKb1ZiUTZsS2xYdUhUSkJOc0l2U0o0azlHL3pObGFGenZTUVFGWG03NEQrYXVsbEE4SEdPUUEvU0F4Tys3Q2p4Nwowbmc9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://192.168.186.139:6443 name: kubernetes contexts: - context: cluster: kubernetes user: kube-proxy name: default current-context: default kind: Config preferences: {} users: - name: kube-proxy user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQzakNDQXNhZ0F3SUJBZ0lVY0EyOURoRG1nQllsS2tUMWdZSDhwQmRFdTlBd 0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU1TURReE9EQTNNekV3TUZvWERUSTVNRFF4TlRBM016RXdNRm93YkRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbGFVcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUm93R0FZRFZRUURFeEZ6ZVhOMFpXMDZhM1ZpClpTMXdjbTk0ZVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTjQ3OVE5OENWZy8KRmdiTjdaS3RxQm9ZdmY3YWRRcmtRTU5odnBzeFozc0hiVmFieVNWN0hwcU5BbHFyMytyVjc4R3Z5VWFaUHpIRgorQ0gwQXJyVE91YjFJcWhLOUlOU2svWnRValhxNkcvYXpoeStKRFE1ckFsT0VyZVZzUmQ2alVncHBDMVhxUlQzCmdBanBEeCtDaE9GcndHMkJBa2o1ZTk0VDVtOFNDWkthQWFDZWVieFZJSUVhTkxFdEtVUk9kaDU2cVpqbVJOUTAKWStGVEhKNHE2Rjd3dkpQODBTTVNGZ3BwbDB2RWV1V2dpK0Nmc0xpdjIrQW5zS25xRHpwTm9oR1g2QkxEQnRTQQpHM29hRzAvTWp0bVk0VFZDMU9xYlFnL3FJWERVTTZhREpTYUlJREZKL2U0TEJwa2F4RUJNbjNyZ3VGQ1lQaWxECnBUZUNsbWcydjdFQ0F3RUFBYU4vTUgwd0RnWURWUjBQQVFIL0JBUURBZ1dnTUIwR0ExVWRKUVFXTUJRR0NDc0cKQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFNQmdOVkhSTUJBZjhFQWpBQU1CMEdBMVVkRGdRV0JCUm1UdzJKZi9SdgpNV05TNU45OThQK2xFMTBrNFRBZkJnTlZIU01FR0RBV2dCUnBIUHdqS0lvRWQ2RHpzOEx3OTFuK3ZUVWI2REFOCkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQUM3NUEwQmdEZ1MrZ0F6VGVkNVYyNlV0RmlMQW1KTkQ4d1hQV1Mra0sKNDBMMGlyT1ZhVUtrZTVRMGUzYjJqTGx5NWpzYUNEVG03dVE3OXBuVUNSRVBQNUpkSzR3TUhPNy9YQ2ZRSmU5RAplZWI0cTh2UCtYdVFCT01jVlNFcHJSMXc5MzBIbTdsK2M5WFVRdHE0M2p3YWl3RXVWQ3UzTjlFYVMvSGFXM0pvClE0RDRqdEFGQUNsbDl6Y3JTYjd3RWhwMFpmaDJJYVlpUy9nUlVFTzAwWFJ6T1doUktac1pZaDZuOStBMXlnUUMKL1poSUlnR1Exdlp4d2U4RHkyY3M5ZUVHVHI3Yis0T3Z6RjNDMDB6Wmlyc2owbkFkVlFyR0N6cjErS0JHNzVBWgptRDFnVVdGNzVhZjN2VWo0WGIzT1NCZTNmU0c1WmpJUmduMXpGQlRKTlN4b213PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBM2p2MUQzd0pXRDhXQnMzdGtxMm9HaGk5L3RwMUN1UkF3M kcrbXpGbmV3ZHRWcHZKCkpYc2VtbzBDV3F2ZjZ0WHZ3YS9KUnBrL01jWDRJZlFDdXRNNjV2VWlxRXIwZzFLVDltMVNOZXJvYjlyT0hMNGsKTkRtc0NVNFN0NVd4RjNxTlNDbWtMVmVwRlBlQUNPa1BINEtFNFd2QWJZRUNTUGw3M2hQbWJ4SUprcG9Cb0o1NQp2RlVnZ1JvMHNTMHBSRTUySG5xcG1PWkUxRFJqNFZNY25pcm9YdkM4ay96Ukl4SVdDbW1YUzhSNjVhQ0w0Sit3CnVLL2I0Q2V3cWVvUE9rMmlFWmZvRXNNRzFJQWJlaG9iVDh5TzJaamhOVUxVNnB0Q0Qrb2hjTlF6cG9NbEpvZ2cKTVVuOTdnc0dtUnJFUUV5ZmV1QzRVSmcrS1VPbE40S1dhRGEvc1FJREFRQUJBb0lCQUNabitqd0kxWnlZbW5mbQprczRza01haHpBUUZRUUQyM1hKbXJBZ3FDNVlwZkczaFVtdlYwVDRvYkdXN0dtRjlRNGdYbHhOS1hLOS9aUmlKCkVRZTBoWk53ZmVMVHdWb0lwV1dMRXhBYVhyMGw5VVRtWDE5Zk1Db1RnZ3lVSkJ6SW95MzdkRkladWpEVGJSOGQKRiszOEluYktwbURHVU56SHNPNTZSZktnRFdOMlE2UDQ4RnJ4eWF6WHhJYjBWT1diNGxFK0I0T0FBZXVONzh6UwoyeWJROXoza3VsMzVlODVEcmpKbml1Z0xkTUxBVDlCdkNXRnZ4eXdIY29FbDFWZElnZFJmWE03RDJqd2NhRTYzCnZDOWFWazQ0SXBUVlBzc2pJUTFtVDEycStsYXl3SmY0TGcwczI2aVRjU3FhUDgwQzU2UXJtcjYreGtaaEx5VTAKbUovdEJCa0NnWUVBNTk2ZWFZd1hIdVdNRUlqMitmWnBoMUlmWWFydVhvYWpMNDVyQkU1YkpBK2lxcmd2cnVsYwpXTWpuVys2TVBmSFNaS3VYUUNzelNYaUVpejFOVk51Q3RKLzBoZkZia2pHazl0cHdwSUJwUnlLTWZCek5sWWtPCm0zK1JmNXNrQjRXWHpkdWozWmQ3TGlQQmFNQWwyOUdSbEtWNy9BY0hXanlZQmlBRkZpd1g5UHNDZ1lFQTlWeWkKeStLZTZkQ3ZQdjR5TmNVaEduRUlmZ0daTk96M1A1QVpyNnI5SnUzakp2KytSQmNZRFNIcWNsZjN0aGxZMU43bApseG83SlR6RXJVRUd1WERnMzFIZ2VZdE9ZNWdzUnlLcXpNUk05TXVQMWNXQ2FUQU5COVdwTjNkV0NES0dYZzZkCmJCMCsyNW9ITnhnSUtBMi9sK1YxWlUyOFd1elJlQ2tyNkk3UFJrTUNnWUJoV1JMVEozRFJsUGhBUFBETU0wdE0KK0FxYTI4UG1SY3FmZmNDcWR5ZEd0WlhLN1RkL3pSUHJaclhUNEF4Yk9Ycm1yeS82VGVqamNNamRHS2l0OXRjaQpkSUdaOXFKR2Q3ZFZ1SkpRVG1WazZ6bG1Ka1dlQVlQemZ4U2NLWXR2NlFPNTl2d09YYm5tdmpaR2YxMmxzNC9XCmc3L1JLVFpLQ1dTZU5iVk5BTWd5SHdLQmdRRGlra1pxaTd3L0lVdVNtZHozdGk5WllXTjhLREczbzlLMVNYWE8KdDlESThBZEFiZ3plaDR6WUk2ZUJLeVk5YTY2Ujg3cURDOS91Qk8yQkozajBLUDRlZWxjVkpjU2ZSMWdyNENGawpzU2gzTExxSHBybEVOUER6ZVNPbmFuVnhZR2FmMkZNYUVPK0lqZlYzdEtOamlUNlJIM3lHclgvdlhwd1huNzFDCkpRM1dUUUtCZ0NETE5EZDBrbkJjeXNScHRURjh0NzRBbGFBUE02eWxpVFNuRURLQVUyNzVhSHpTbHRSazFQR3cKRjdSUGN3MEE1V0Y3ckQyVXVMZTNsQ1NuUnBnamRHZ1JXc3ZTTlVybm9yZ0tIZEw3OHFYUHVHYW9hUUN3azFjZgpSaUF5ZEgvYVhmZjhLU0drd1Z0ZE9XdGJwS2pYOXdabmM0MWI1RS9HNE1Fd3A5OFlpWWJSCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg== 拷贝到node节点 [root@k8s-master01 k8s-cert]# scp -r bootstrap.kubeconfig kube-proxy.kubeconfig root@192.168.186.141:/opt/kubernetes/cfg/ root@192.168.186.141's password: bootstrap.kubeconfig 100% 2169 1.4MB/s 00:00 kube-proxy.kubeconfig 100% 6271 2.3MB/s 00:00 [root@k8s-master01 k8s-cert]# scp -r bootstrap.kubeconfig kube-proxy.kubeconfig root@192.168.186.142:/opt/kubernetes/cfg/ root@192.168.186.142's password: bootstrap.kubeconfig 100% 2169 1.4MB/s 00:00 kube-proxy.kubeconfig 100% 6271 3.8MB/s 00:00
2.3 部署node节点¶
kubelet.sh
[root@k8s-node01 ~]# cat kubelet.sh #!/bin/bash NODE_ADDRESS=$1 DNS_SERVER_IP=${2:-"10.0.0.2"} cat <<EOF >/opt/kubernetes/cfg/kubelet KUBELET_OPTS="--logtostderr=false \\ --log-dir=/opt/kubernetes/logs \\ --v=4 \\ --hostname-override=${NODE_ADDRESS} \\ --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\ --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\ --config=/opt/kubernetes/cfg/kubelet.config \\ --cert-dir=/opt/kubernetes/ssl \\ --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0" EOF cat <<EOF >/opt/kubernetes/cfg/kubelet.config kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 address: ${NODE_ADDRESS} port: 10250 readOnlyPort: 10255 cgroupDriver: cgroupfs clusterDNS: - ${DNS_SERVER_IP} clusterDomain: cluster.local. failSwapOn: false authentication: anonymous: enabled: true EOF cat <<EOF >/usr/lib/systemd/system/kubelet.service [Unit] Description=Kubernetes Kubelet After=docker.service Requires=docker.service [Service] EnvironmentFile=/opt/kubernetes/cfg/kubelet ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS Restart=on-failure KillMode=process [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable kubelet systemctl restart kubelet
proxy.sh
[root@k8s-node01 ~]# cat proxy.sh #!/bin/bash NODE_ADDRESS=$1 cat <<EOF >/opt/kubernetes/cfg/kube-proxy KUBE_PROXY_OPTS="--logtostderr=true \\ --v=4 \\ --hostname-override=${NODE_ADDRESS} \\ --cluster-cidr=10.0.0.0/24 \\ --proxy-mode=ipvs \\ --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig" EOF cat <<EOF >/usr/lib/systemd/system/kube-proxy.service [Unit] Description=Kubernetes Proxy After=network.target [Service] EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable kube-proxy systemctl restart kube-proxy
- 在master上操作
拷贝命令到node上
scp kubelet kube-proxy root@192.168.186.141:/opt/kubernetes/bin/ scp kubelet kube-proxy root@192.168.186.142:/opt/kubernetes/bin/
[root@k8s-master01 bin]# pwd /root/soft/kubernetes/server/bin [root@k8s-master01 bin]# scp kubelet kube-proxy root@192.168.186.141:/opt/kubernetes/bin/ root@192.168.186.141's password: kubelet 100% 108MB 61.1MB/s 00:01 kube-proxy 100% 33MB 45.2MB/s 00:00 [root@k8s-master01 bin]# scp kubelet kube-proxy root@192.168.186.142:/opt/kubernetes/bin/ root@192.168.186.142's password: kubelet 100% 108MB 65.4MB/s 00:01 kube-proxy 100% 33MB 48.8MB/s 00:00
- 在node上操作
详细步骤
unzip node.zip mkdir -p /opt/kubernetes/logs sh kubelet.sh 192.168.186.141 systemctl status kubelet ps axf|grep kubelet
[root@k8s-node01 ~]# ls anaconda-ks.cfg flannel.sh flannel-v0.10.0-linux-amd64.tar.gz node.zip README.md [root@k8s-node01 ~]# unzip node.zip Archive: node.zip inflating: proxy.sh inflating: kubelet.sh [root@k8s-node01 ~]# ls anaconda-ks.cfg flannel.sh flannel-v0.10.0-linux-amd64.tar.gz kubelet.sh node.zip proxy.sh README.md [root@k8s-node01 ~]# sh kubelet.sh 192.168.186.141 Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service. 检查进程 [root@k8s-node01 ~]# systemctl status kubelet ● kubelet.service - Kubernetes Kubelet Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2019-04-18 17:21:09 CST; 48s ago Main PID: 23949 (kubelet) Tasks: 11 Memory: 17.7M CGroup: /system.slice/kubelet.service └─23949 /opt/kubernetes/bin/kubelet --logtostderr=false --log-dir=/opt/kubernetes/logs --v=4 --hostname-override=192... Apr 18 17:21:09 k8s-node01 systemd[1]: Started Kubernetes Kubelet. [root@k8s-node01 ~]# ps axf|grep kubelet 25488 pts/0 S+ 0:00 | \_ grep --color=auto kubelet 23949 ? Ssl 0:00 /opt/kubernetes/bin/kubelet --logtostderr=false --log-dir=/opt/kubernetes/logs --v=4 --hostname-overri de=192.168.186.141 --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig --config=/opt/kubernetes/cfg/kubelet.config --cert-dir=/opt/kubernetes/ssl --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
授权加入.在master上操作
[root@k8s-master01 bin]# kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-ve2dlsj5vwoAxLhYmkait9Cc013EV6TuMc3mFmPlrMY 22m kubelet-bootstrap Pending [root@k8s-master01 bin]# kubectl certificate approve node-csr-ve2dlsj5vwoAxLhYmkait9Cc013EV6TuMc3mFmPlrMY certificatesigningrequest.certificates.k8s.io/node-csr-ve2dlsj5vwoAxLhYmkait9Cc013EV6TuMc3mFmPlrMY approved [root@k8s-master01 bin]# kubectl get node NAME STATUS ROLES AGE VERSION 192.168.186.141 Ready <none> 14s v1.13.4
安装 kube-proxy
sh proxy.sh 192.168.186.141 ps axf|grep proxy
[root@k8s-node01 ~]# sh proxy.sh 192.168.186.141 Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service. [root@k8s-node01 ~]# ps axf|grep proxy 26210 pts/0 S+ 0:00 | \_ grep --color=auto proxy 26019 ? Ssl 0:00 /opt/kubernetes/bin/kube-proxy --logtostderr=true --v=4 --hostname-override=192.168.186.141 --cluster- cidr=10.0.0.0/24 --proxy-mode=ipvs --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig
到目前为止node1已经OK,下面增加node
- node1上操作
scp -r /opt/kubernetes/ root @192.168.186.142:/opt/ scp /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.186.142:/usr/lib/systemd/system/
- node2上操作
cd /opt/kubernetes/ssl rm -rf * cd /opt/kubernetes/cfg sed -i 's@192.168.186.141@192.168.186.142@g' kubelet sed -i 's@192.168.186.141@192.168.186.142@g' kubelet.config sed -i 's@192.168.186.141@192.168.186.142@g' kube-proxy systemctl start kube-proxy systemctl status kube-proxy systemctl start kubelet.service systemctl status kubelet.service
详细步骤
在node1上操作 [root@k8s-node01 ~]# scp -r /opt/kubernetes/ root @192.168.186.142:/opt/ root@192.168.186.142's password: scp: /opt//kubernetes/bin/flanneld: Text file busy mk-docker-opts.sh 100% 2139 1.1MB/s 00:00 kubelet 100% 108MB 68.5MB/s 00:01 kube-proxy 100% 33MB 42.8MB/s 00:00 flanneld 100% 241 114.4KB/s 00:00 bootstrap.kubeconfig 100% 2169 1.4MB/s 00:00 kube-proxy.kubeconfig 100% 6271 3.0MB/s 00:00 kubelet 100% 413 206.2KB/s 00:00 kubelet.config 100% 269 74.5KB/s 00:00 kubelet.kubeconfig 100% 2298 1.2MB/s 00:00 kube-proxy 100% 191 98.9KB/s 00:00 kubelet.crt 100% 2197 412.9KB/s 00:00 kubelet.key 100% 1679 865.2KB/s 00:00 kubelet-client-2019-04-18-17-44-09.pem 100% 1277 167.2KB/s 00:00 kubelet-client-current.pem 100% 1277 290.6KB/s 00:00 kubelet.k8s-node01.root.log.INFO.20190418-172109.23949 100% 234KB 23.8MB/s 00:00 kubelet.INFO 100% 234KB 22.9MB/s 00:00 kubelet.k8s-node01.root.log.WARNING.20190418-174409.23949 100% 1164 270.6KB/s 00:00 kubelet.WARNING 100% 1164 738.6KB/s 00:00 kubelet.k8s-node01.root.log.ERROR.20190418-174409.23949 100% 465 213.4KB/s 00:00 kubelet.ERROR 100% 465 351.2KB/s 00:00 [root@k8s-node01 ~]# scp /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.186.142:/usr/lib/systemd/system/ root@192.168.186.142's password: kubelet.service 100% 264 185.1KB/s 00:00 kube-proxy.service 100% 231 75.3KB/s 00:00 [root@k8s-node01 ~]# 在node2操作 [root@k8s-node02 cfg]# cd ../ssl [root@k8s-node02 ssl]# pwd /opt/kubernetes/ssl [root@k8s-node02 ssl]# ls kubelet-client-2019-04-18-17-44-09.pem kubelet-client-current.pem kubelet.crt kubelet.key [root@k8s-node02 ssl]# rm -f * [root@k8s-node02 ssl]# ls [root@k8s-node02 ssl]# cd .. [root@k8s-node02 kubernetes]# ls bin cfg logs ssl [root@k8s-node02 kubernetes]# cd cfg/ [root@k8s-node02 cfg]# ls bootstrap.kubeconfig flanneld kubelet kubelet.config kubelet.kubeconfig kube-proxy kube-proxy.kubeconfig [root@k8s-node02 cfg]# sed -i 's@192.168.186.141@192.168.186.142@g' kubelet [root@k8s-node02 cfg]# cat kubelet KUBELET_OPTS="--logtostderr=false \ --log-dir=/opt/kubernetes/logs \ --v=4 \ --hostname-override=192.168.186.142 \ --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \ --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \ --config=/opt/kubernetes/cfg/kubelet.config \ --cert-dir=/opt/kubernetes/ssl \ --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0" [root@k8s-node02 cfg]# sed -i 's@192.168.186.141@192.168.186.142@g' kubelet.config [root@k8s-node02 cfg]# cat kubelet.config kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 address: 192.168.186.142 port: 10250 readOnlyPort: 10255 cgroupDriver: cgroupfs clusterDNS: - 10.0.0.2 clusterDomain: cluster.local. failSwapOn: false authentication: anonymous: enabled: true [root@k8s-node02 cfg]# sed -i 's@192.168.186.141@192.168.186.142@g' kube-proxy [root@k8s-node02 cfg]# cat kube-proxy KUBE_PROXY_OPTS="--logtostderr=true \ --v=4 \ --hostname-override=192.168.186.142 \ --cluster-cidr=10.0.0.0/24 \ --proxy-mode=ipvs \ --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig" [root@k8s-node02 cfg]# systemctl start kube-proxy [root@k8s-node02 cfg]# systemctl status kube-proxy ● kube-proxy.service - Kubernetes Proxy Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; disabled; vendor preset: disabled) Active: active (running) since Thu 2019-04-18 18:02:27 CST; 6s ago Main PID: 57457 (kube-proxy) Tasks: 0 Memory: 11.0M CGroup: /system.slice/kube-proxy.service ‣ 57457 /opt/kubernetes/bin/kube-proxy --logtostderr=true --v=4 --hostname-override=192.168.186.142 --cluster-cidr=1... Apr 18 18:02:28 k8s-node02 kube-proxy[57457]: -A KUBE-FORWARD -s 10.0.0.0/24 -m comment --comment "kubernetes forwarding ...ACCEPT Apr 18 18:02:28 k8s-node02 kube-proxy[57457]: -A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack pod d...ACCEPT Apr 18 18:02:28 k8s-node02 kube-proxy[57457]: COMMIT Apr 18 18:02:28 k8s-node02 kube-proxy[57457]: I0418 18:02:28.234622 57457 proxier.go:719] syncProxyRules took 176.666464ms Apr 18 18:02:29 k8s-node02 kube-proxy[57457]: I0418 18:02:29.442751 57457 config.go:141] Calling handler.OnEndpointsUpdate Apr 18 18:02:29 k8s-node02 kube-proxy[57457]: I0418 18:02:29.458787 57457 config.go:141] Calling handler.OnEndpointsUpdate Apr 18 18:02:31 k8s-node02 kube-proxy[57457]: I0418 18:02:31.450647 57457 config.go:141] Calling handler.OnEndpointsUpdate Apr 18 18:02:31 k8s-node02 kube-proxy[57457]: I0418 18:02:31.470753 57457 config.go:141] Calling handler.OnEndpointsUpdate Apr 18 18:02:33 k8s-node02 kube-proxy[57457]: I0418 18:02:33.459719 57457 config.go:141] Calling handler.OnEndpointsUpdate Apr 18 18:02:33 k8s-node02 kube-proxy[57457]: I0418 18:02:33.477608 57457 config.go:141] Calling handler.OnEndpointsUpdate Hint: Some lines were ellipsized, use -l to show in full. [root@k8s-node02 cfg]# ps axf|grep proxy 57707 pts/1 S+ 0:00 \_ grep --color=auto proxy 57457 ? Ssl 0:00 /opt/kubernetes/bin/kube-proxy --logtostderr=true --v=4 --hostname-override=192.168.186.142 --cluster- cidr=10.0.0.0/24 --proxy-mode=ipvs --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig [root@k8s-node02 logs]# systemctl start kubelet.service [root@k8s-node02 logs]# systemctl status kubelet.service ● kubelet.service - Kubernetes Kubelet Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled; vendor preset: disabled) Active: active (running) since Thu 2019-04-18 18:05:06 CST; 1s ago Main PID: 58060 (kubelet) Tasks: 9 Memory: 17.3M CGroup: /system.slice/kubelet.service └─58060 /opt/kubernetes/bin/kubelet --logtostderr=false --log-dir=/opt/kubernetes/logs --v=4 --hostname-override=192.168.186.142 --kubeconfig=/opt/kubernetes/cfg/kubelet.kubec... Apr 18 18:05:06 k8s-node02 systemd[1]: Started Kubernetes Kubelet. Apr 18 18:05:06 k8s-node02 kubelet[58060]: E0418 18:05:06.935679 58060 bootstrap.go:184] Unable to read existing bootstrap client config: invalid configuration: [unable to read client-... Hint: Some lines were ellipsized, use -l to show in full.
[root@k8s-master01 bin]# kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-AAISS6f1ueC4zdGHiKj9KOiZsXwcCicG8oNXJuLRKmw 116s kubelet-bootstrap Pending node-csr-ve2dlsj5vwoAxLhYmkait9Cc013EV6TuMc3mFmPlrMY 45m kubelet-bootstrap Approved,Issued [root@k8s-master01 bin]# kubectl certificate approve node-csr-AAISS6f1ueC4zdGHiKj9KOiZsXwcCicG8oNXJuLRKmw certificatesigningrequest.certificates.k8s.io/node-csr-AAISS6f1ueC4zdGHiKj9KOiZsXwcCicG8oNXJuLRKmw approved [root@k8s-master01 bin]# kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-AAISS6f1ueC4zdGHiKj9KOiZsXwcCicG8oNXJuLRKmw 3m4s kubelet-bootstrap Approved,Issued node-csr-ve2dlsj5vwoAxLhYmkait9Cc013EV6TuMc3mFmPlrMY 47m kubelet-bootstrap Approved,Issued [root@k8s-master01 bin]# kubectl get node NAME STATUS ROLES AGE VERSION 192.168.186.141 Ready <none> 24m v1.13.4 192.168.186.142 Ready <none> 33s v1.13.4
3. 部署测试示例¶
kubectl create deployment nginx --image=nginx # kubectl create deployment nginx --image=nginx --replicas=3 kubectl get pod kubectl expose deployment nginx --port=88 --target-port=80 --type=NodePort kubectl get svc nginx kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
root@k8s-master01 bin]# kubectl create deployment nginx --image=nginx deployment.apps/nginx created [root@k8s-master01 bin]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-5c7588df-nlj5l 0/1 ContainerCreating 0 9s 过一会在查看 [root@k8s-master01 bin]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-5c7588df-nlj5l 1/1 Running 0 26s 已经运行了。 接下来扩容为3个 [root@k8s-master01 bin]# kubectl scale deployment nginx --replicas=3 deployment.extensions/nginx scaled [root@k8s-master01 bin]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-5c7588df-c58ql 0/1 ContainerCreating 0 3s nginx-5c7588df-gh6l9 1/1 Running 0 3s nginx-5c7588df-nlj5l 1/1 Running 0 84s 过一会在查看 [root@k8s-master01 bin]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-5c7588df-c58ql 1/1 Running 0 28s nginx-5c7588df-gh6l9 1/1 Running 0 28s nginx-5c7588df-nlj5l 1/1 Running 0 109s 可以查看pod在哪个节点上 [root@k8s-master01 bin]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-5c7588df-c58ql 1/1 Running 0 55s 172.17.8.2 192.168.186.142 <none> <none> nginx-5c7588df-gh6l9 1/1 Running 0 55s 172.17.66.3 192.168.186.141 <none> <none> nginx-5c7588df-nlj5l 1/1 Running 0 2m16s 172.17.66.2 192.168.186.141 <none> 为新部署的nginx容器暴露端口 [root@k8s-master01 bin]# kubectl expose deployment nginx --port=88 --target-port=80 --type=NodePort service/nginx exposed [root@k8s-master01 bin]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 144m nginx NodePort 10.0.0.88 <none> 88:30232/TCP 74s 容器内是ip是10.0.0.88[在随便哪个node上都可以访问],端口是88,30232端口是宿主机的对应IP,是随机的,也可以指定 在node上测试 [root@k8s-node01 ~]# curl 10.0.0.88:88 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html> [root@k8s-master01 bin]# curl 192.168.186.141:30232 -I HTTP/1.1 200 OK Server: nginx/1.15.12 Date: Thu, 18 Apr 2019 10:18:53 GMT Content-Type: text/html Content-Length: 612 Last-Modified: Tue, 16 Apr 2019 13:08:19 GMT Connection: keep-alive ETag: "5cb5d3c3-264" Accept-Ranges: bytes 查看pod日志 [root@k8s-master01 bin]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-5c7588df-c58ql 1/1 Running 0 6m40s 172.17.8.2 192.168.186.142 <none> <none> nginx-5c7588df-gh6l9 1/1 Running 0 6m40s 172.17.66.3 192.168.186.141 <none> <none> nginx-5c7588df-nlj5l 1/1 Running 0 8m1s 172.17.66.2 192.168.186.141 <none> <none> [root@k8s-master01 bin]# kubectl logs nginx-5c7588df-c58ql Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-5c7588df-c58ql) 出现这个错误,需要先授权。 [root@k8s-master01 bin]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created [root@k8s-master01 bin]# kubectl logs nginx-5c7588df-c58ql 172.17.66.0 - - [18/Apr/2019:10:17:42 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-" 172.17.66.0 - - [18/Apr/2019:10:18:50 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"