Troubleshooting error: kubectl Cannot Reach the API

Purpose Diagnose and recover a single-node kURL / kubeadm cluster where kubectl fails, and the control plane is down because:

The etcd static-pod manifest is gone

apiserver.crt and apiserver.key are gone

controller-manager.conf and scheduler.conf are empty

Note: The etcd data directory and cluster CA remain intact on disk.

Scope

Apply this procedure only when all of the following conditions are met:

The node is a self-managed kURL or kubeadm control plane (not EKS, GKE, or AKS).

kubectl fails with localhost:8080 connection refused or API port 6443 connection refused.

kubelet is active, but ss shows no listener on 6443 or 2379.

/etc/kubernetes/manifests/etcd.yaml is missing.

/etc/kubernetes/pki/apiserver.crt or apiserver.key is missing.

/var/lib/etcd/member contains snap/ and wal/.

/etc/kubernetes/pki/ca.crt and /etc/kubernetes/pki/etcd/ exist.

Out of Scope: Do not use this procedure if etcd data or the cluster CA is lost, or on multi-node etcd clusters that have lost quorum. Those scenarios require snapshot recovery or member rebuilds.

Impact and Overview

Customer-facing Issue

The Swimlane / Turbine UI and APIs are unavailable because the Kubernetes API is not listening.

kubectl commands fail immediately.

KOTS Admin Console and Swimlane UI do not load.

Workloads are stopped.

Node remains reachable over SSH; kubelet is active.

Impact Assessment

No Kubernetes API dependent components (scheduling, CoreDNS, ingress, pods) can run. Application data (etcd, Postgres, Mongo PVCs) remains intact on disk, but is unreachable until control plane services are restored.

Environment Details

Item	Value
Node	swimlane
Address(Example)	10.20.x.x
OS	Rocky Linux 9.4
Kubernetes	v1.36.5
Runtime	containerd 2.3.6
Installer	kURL (config: /opt/replicated/kubeadm.conf)
API server	https://10.20.x.x:6443
HAProxy front door	0.0.0.0:6444 -> 10.20.x.x:6443
etcd data	/var/lib/etcd
etcd image	docker.io/kurlsh/etcd:v3.6.15
Admin kubeconfig	/etc/kubernetes/admin.conf

Note: the IP here is the lab IP, and it should be different in the customer environment.

Detailed Findings

1. kubectl with no kubeconfig defaults to localhost:8080

Non-login SSH sessions do not evaluate /etc/profile, leaving KUBECONFIG unset and causing kubectl to default to http://localhost:8080.

The connection to the server localhost:8080 was refused - did you specify the right host or port?
Client Version: v1.36.5

Setting the kubeconfig explicitly reveals the underlying API server failure.

2. API Port is Closed

KUBECONFIG=/etc/kubernetes/admin.conf kubectl get nodes
# Output:
dial tcp 10.20.x.x:6443: connect: connection refused

The admin kubeconfig file /etc/kubernetes/admin.conf is intact, but the API process is not listening.

3. kubelet Status & Container State

kubelet is running (listening on port 10250), but control plane containers (kube-apiserver, kube-controller-manager, kube-scheduler, haproxy) are in Exited status. No etcd container is present.

4. API Server Certificate Missing

Logs show kube-apiserver crashing due to missing TLS certificate files:

"command failed" err="open /etc/kubernetes/pki/apiserver.crt: no such file or directory"

5. Empty Controller Manager & Scheduler Kubeconfigs

Both controller-manager.conf and scheduler.conf are 0 bytes, causing authentication errors on startup.

6. Missing etcd Static-Pod Manifest

The manifest /etc/kubernetes/manifests/etcd.yaml is missing. However, etcd data (/var/lib/etcd/member) and PKI certificates under /etc/kubernetes/pki/etcd/ remain intact.

Root Cause

The control plane failed because critical files were accidentally deleted or truncated while processes were running. Following a kubelet restart, missing manifests and key certificates prevented control plane components from launching.

Safety Rules & Guardrails

DO NOT run kubeadm reset.

DO NOT delete or move /var/lib/etcd.

DO NOT regenerate the CA (e.g. kubeadm init phase certs ca).

DO NOT overwrite admin.conf unless confirmed corrupt.

ALWAYS back up PKI, manifests, and configs before applying fixes.

Generate only missing objects: API server cert, missing kubeconfigs, and etcd manifest.

Diagnosis Steps

Execute the following commands as root on the control plane node:

1. Verify API Access

export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl get nodes

2. Check System Resources & Listeners

systemctl is-active kubelet
ss -lnt | awk '/:6443|:6444|:2379|:2380|:10250/'
df -h /
free -h

3. Inspect Containers & Files

crictl ps -a | grep -E 'etcd|kube-apiserver|kube-controller-manager|kube-scheduler|haproxy'
ls -la /etc/kubernetes/manifests
wc -c /etc/kubernetes/*.conf

4. Inspect Container Logs

API=$(crictl ps -a --name kube-apiserver -q | head -1)
crictl logs --tail 30 "$API"
CM=$(crictl ps -a --name kube-controller-manager -q | head -1)
crictl logs --tail 20 "$CM"

5. Verify etcd & PKI Integrity

ls -lh /var/lib/etcd/member/snap /var/lib/etcd/member/wal
find /etc/kubernetes/pki -printf '%p %s\n' | sort
openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -subject -ext subjectAltName

Resolution Procedure

Step 1: Backup Existing Files

BK=/root/cp-repair-backup-$(date +%Y%m%d%H%M%S)
mkdir -p "$BK"
cp -a /etc/kubernetes/pki "$BK/pki"
cp -a /etc/kubernetes/manifests "$BK/manifests"
cp -a /etc/kubernetes/*.conf "$BK/"
echo "Backup completed: $BK"

Step 2: Clean Up Truncated Configurations

rm -f /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf

Step 3: Re-issue API Server Certificate

kubeadm init phase certs apiserver --config /opt/replicated/kubeadm.conf
# Verify certs
ls -l /etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/apiserver.key

Step 4: Regenerate Kubeconfigs

kubeadm init phase kubeconfig controller-manager --config /opt/replicated/kubeadm.conf
kubeadm init phase kubeconfig scheduler --config /opt/replicated/kubeadm.conf

Step 5: Recreate etcd Manifest & Patch Image

kubeadm init phase etcd local --config /opt/replicated/kubeadm.conf

from pathlib import Path
p = Path("/etc/kubernetes/manifests/etcd.yaml")
text = p.read_text()
old = "registry.k8s.io/etcd:3.6.8-0"
new = "docker.io/kurlsh/etcd:v3.6.15"
if old not in text:
    raise SystemExit("expected kubeadm etcd image was not in the manifest")
p.write_text(text.replace(old, new))
print("patched etcd image to", new)

Step 6: Ensure Environment Profile Configuration

grep -q 'KUBECONFIG=/etc/kubernetes/admin.conf' /root/.bashrc \
  || printf '\nexport KUBECONFIG=/etc/kubernetes/admin.conf\n' >> /root/.bashrc

Step 7: Restart Kubelet & Verify etcd Health

systemctl restart kubelet
# Check etcd container status
crictl ps --name etcd

# Verify endpoint health
ETCD=$(crictl ps --name etcd -q | head -1)
crictl exec "$ETCD" etcdctl endpoint health \
  --endpoints=https://127.0.x.x:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key

Verification

export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl get nodes -o wide
kubectl get --raw='/readyz'
kubectl get pods -n kube-system -o wide
kubectl get pods -A

Troubleshooting Matrix - etcd data under `/var/lib/etcd` - cluster CA, etcd certificates, front-proxy certificates, and service-account keys - `admin.conf` and `kubelet.conf` - application PVCs - HAProxy configuration - the existing static-pod manifests for the API server, controller manager, scheduler, and HAProxy, other than kubelet recreating their containers If a later step fails

Symptom	Cause	Action
etcd exits on data-dir or WAL	Cannot open existing database	Do not delete data directory. Collect crictl logs and restore from snapshot.
etcd health check fails TLS	Certificates mismatch	Compare SANs with openssl on server.crt. Do not regenerate CA.
API server cannot open apiserver.crt	Cert not in mounted host path	Verify /etc/kubernetes/pki/apiserver.crt exists on host.
API CrashLoopBackOff after etcd healthy	kubelet container backoff	Run systemctl restart kubelet after etcd is confirmed healthy.
Pods stuck on CreatePodSandbox	Missing /run/flannel/subnet.env	Ensure API is healthy and flannel pod finishes setup.
kubectl defaults to localhost:8080	Unset KUBECONFIG env variable	Export KUBECONFIG=/etc/kubernetes/admin.conf or use login shell.