lWiJY2yVCuuRomG5HSN5Kubelet stopped posting node status
Kubelet stopped posting node status
Returns nodes where kubelet had stopped posting status updates.
inputs
outputs
context.skip_sub_tasks = True
no_status_nodes = []
for node in problem_nodes:
nodename = node['nodename']
cmd = f"kubectl describe node {nodename}"
op = _exe(master_ip, cmd)
if "kubelet stopped posting node status" in op.lower():
context.skip_sub_tasks = False
no_status_nodes.append(node)
print(no_status_nodes)
copied- 1R6CHVIIkBPQtZfWnDAdBKubelet Dead
1
Identifies nodes where the Kubelet is in an inactive or dead state.
inputsoutputsdead_kubelet_nodes = [] for node in no_status_nodes: instance_id = node['instance_id'] cmd = "sudo systemctl status kubelet | grep 'Active:'" op = _exe(instance_id, cmd) if "inactive (dead)" in op: dead_kubelet_nodes.append(node) context.skip_sub_tasks = len(dead_kubelet_nodes) == 0 print(dead_kubelet_nodes)copied1 - 2IbAsoCsZhM2ewyCX7GaVKubelet exited
2
Identifies kubelets that are in active but exited state.
inputsoutputsexited_kubelet_nodes = [] for node in no_status_nodes: instance_id = node['instance_id'] cmd = "sudo systemctl status kubelet | grep 'Active:'" op = _exe(instance_id, cmd) if "active (exited)" in op: exited_kubelet_nodes.append(node) context.skip_sub_tasks = len(exited_kubelet_nodes) == 0 print(exited_kubelet_nodes)copied2 - 3NgnZ2L0OqiMQIpG5LCfWWorker kubelet unable to update master
3
Gets the master instance IP addresses in the cluster and uses it to perform connectivity checks.
inputsoutputsfor node in no_status_nodes: nodeip = node['nodeip'] cmd = "sudo systemctl status kubelet" op = _exe(nodeip, cmd) #_problem = False if "Unable to register node with API server" in op: pass master_ip_addr = _get_ip_addr(master_ip) cmd1 = f'aws ec2 describe-instances --filters "Name=ip-address,Values={master_ip_addr}" --query "Reservations[*].Instances[*].InstanceId" --output text' master_instance_ids = [_exe(None, cmd1).strip()] print(master_instance_ids)copied3- 3.1xSYmJ2ECr0PZ9k9aTbFvSecurity group blocked the API request
3.1
Looks at the security groups of the master instances (by their IDs) to check if there are possible port configuration mismatches preventing connectivity.
inputsoutputsimport json for instance_id in master_instance_ids: _problem = True cmd = f"aws ec2 describe-instances --instance-ids {instance_id} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --output=text" sg_ids1 = _exe(None, cmd) print(sg_ids1) sg_ids = re.split('\s',sg_ids1.strip()) if sg_ids: for sg_id in sg_ids: if not sg_id: continue cmd1 = 'aws ec2 describe-security-groups --filter Name=group-id,Values=' cmd1+= sg_id cmd1+= ' --query SecurityGroups[*].IpPermissions[*]' op = _exe(None, cmd1) json_op = json.loads(op) for sg in json_op: for rule in sg: if 'FromPort' in rule: port_lo = int(rule['FromPort']) port_hi = port_lo if 'ToPort' in rule: port_hi = int(rule['ToPort']) if port >= port_lo and port <= port_hi: _problem = False else: continue if _problem: break if _problem: context.log("ERROR", "Found problem")copied3.1