---
title: Troubleshooting error: kubectl Cannot Reach the API
slug: troubleshotting/troubleshooting-error-kubectl-cannot-reach-the-api
createdAt: 2026-11-10T13:33:50.759Z
---

**Purpose**
Diagnose and recover a single-node kURL / kubeadm cluster where kubectl fails, and the control plane is down because:

- The etcd static-pod manifest is gone
- apiserver.crt and apiserver.key are gone
- controller-manager.conf and scheduler.conf are empty

Note: The etcd data directory and cluster CA remain intact on disk.

# Scope

Apply this procedure **only** when all of the following conditions are met:

- The node is a self-managed kURL or kubeadm control plane (not EKS, GKE, or AKS).
- kubectl fails with localhost:8080 connection refused or API port 6443 connection refused.
- kubelet is active, but ss shows no listener on 6443 or 2379.
- /etc/kubernetes/manifests/etcd.yaml is missing.
- /etc/kubernetes/pki/apiserver.crt or apiserver.key is missing.
- /var/lib/etcd/member contains snap/ and wal/.
- /etc/kubernetes/pki/ca.crt and /etc/kubernetes/pki/etcd/ exist.

**Out of Scope:** Do not use this procedure if etcd data or the cluster CA is lost, or on multi-node etcd clusters that have lost quorum. Those scenarios require snapshot recovery or member rebuilds.

# Impact and Overview

## Customer-facing Issue

The Swimlane / Turbine UI and APIs are unavailable because the Kubernetes API is not listening.

- kubectl commands fail immediately.
- KOTS Admin Console and Swimlane UI do not load.
- Workloads are stopped.
- Node remains reachable over SSH; kubelet is active.

## Impact Assessment

No Kubernetes API dependent components (scheduling, CoreDNS, ingress, pods) can run. Application data (etcd, Postgres, Mongo PVCs) remains intact on disk, but is unreachable until control plane services are restored.

## Environment Details

| Item               | Value                                       |
| ------------------ | ------------------------------------------- |
| Node               | swimlane                                    |
| Address(Example)   | 10.20.x.x                                   |
| OS                 | Rocky Linux 9.4                             |
| Kubernetes         | v1.36.5                                     |
| Runtime            | containerd 2.3.6                            |
| Installer          | kURL (config: /opt/replicated/kubeadm.conf) |
| API server         | https\://10.20.x.x:6443                     |
| HAProxy front door | 0.0.0.0:6444 -> 10.20.x.x:6443              |
| etcd data          | /var/lib/etcd                               |
| etcd image         | docker.io/kurlsh/etcd\:v3.6.15              |
| Admin kubeconfig   | /etc/kubernetes/admin.conf                  |

**Note:** the IP here is the lab IP, and it should be different in the customer environment.

# Detailed Findings

## 1. kubectl with no kubeconfig defaults to localhost:8080

Non-login SSH sessions do not evaluate /etc/profile, leaving KUBECONFIG unset and causing kubectl to default to http\://localhost:8080.

```shell
The connection to the server localhost:8080 was refused - did you specify the right host or port?
Client Version: v1.36.5
```

Setting the kubeconfig explicitly reveals the underlying API server failure.

## 2. API Port is Closed

```shell
KUBECONFIG=/etc/kubernetes/admin.conf kubectl get nodes
# Output:
dial tcp 10.20.x.x:6443: connect: connection refused
```

The admin kubeconfig file /etc/kubernetes/admin.conf is intact, but the API process is not listening.

## 3. kubelet Status & Container State

kubelet is running (listening on port 10250), but control plane containers (kube-apiserver, kube-controller-manager, kube-scheduler, haproxy) are in Exited status. No etcd container is present.

## 4. API Server Certificate Missing

Logs show kube-apiserver crashing due to missing TLS certificate files:

```shell
"command failed" err="open /etc/kubernetes/pki/apiserver.crt: no such file or directory"
```

## 5. Empty Controller Manager & Scheduler Kubeconfigs

Both controller-manager.conf and scheduler.conf are 0 bytes, causing authentication errors on startup.

## 6. Missing etcd Static-Pod Manifest

The manifest /etc/kubernetes/manifests/etcd.yaml is missing. However, etcd data (/var/lib/etcd/member) and PKI certificates under /etc/kubernetes/pki/etcd/ remain intact.

# Root Cause

The control plane failed because critical files were accidentally deleted or truncated while processes were running. Following a kubelet restart, missing manifests and key certificates prevented control plane components from launching.

# Safety Rules & Guardrails

- **DO NOT** run kubeadm reset.
- **DO NOT** delete or move /var/lib/etcd.
- **DO NOT** regenerate the CA (e.g. kubeadm init phase certs ca).
- **DO NOT** overwrite admin.conf unless confirmed corrupt.
- **ALWAYS** back up PKI, manifests, and configs before applying fixes.
- Generate **only** missing objects: API server cert, missing kubeconfigs, and etcd manifest.

# Diagnosis Steps

Execute the following commands as root on the control plane node:

## 1. Verify API Access

```shell
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl get nodes
```

## 2. Check System Resources & Listeners

```shell
systemctl is-active kubelet
ss -lnt | awk '/:6443|:6444|:2379|:2380|:10250/'
df -h /
free -h
```

## 3. Inspect Containers & Files

```shell
crictl ps -a | grep -E 'etcd|kube-apiserver|kube-controller-manager|kube-scheduler|haproxy'
ls -la /etc/kubernetes/manifests
wc -c /etc/kubernetes/*.conf
```

## 4. Inspect Container Logs

```shell
API=$(crictl ps -a --name kube-apiserver -q | head -1)
crictl logs --tail 30 "$API"
CM=$(crictl ps -a --name kube-controller-manager -q | head -1)
crictl logs --tail 20 "$CM"
```

## 5. Verify etcd & PKI Integrity

```shell
ls -lh /var/lib/etcd/member/snap /var/lib/etcd/member/wal
find /etc/kubernetes/pki -printf '%p %s\n' | sort
openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -subject -ext subjectAltName
```

# Resolution Procedure

## Step 1: Backup Existing Files

```shell
BK=/root/cp-repair-backup-$(date +%Y%m%d%H%M%S)
mkdir -p "$BK"
cp -a /etc/kubernetes/pki "$BK/pki"
cp -a /etc/kubernetes/manifests "$BK/manifests"
cp -a /etc/kubernetes/*.conf "$BK/"
echo "Backup completed: $BK"
```

## Step 2: Clean Up Truncated Configurations

```shell
rm -f /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
```

## Step 3: Re-issue API Server Certificate

```shell
kubeadm init phase certs apiserver --config /opt/replicated/kubeadm.conf
# Verify certs
ls -l /etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/apiserver.key
```

## Step 4: Regenerate Kubeconfigs

```shell
kubeadm init phase kubeconfig controller-manager --config /opt/replicated/kubeadm.conf
kubeadm init phase kubeconfig scheduler --config /opt/replicated/kubeadm.conf
```

## Step 5: Recreate etcd Manifest & Patch Image

```shell
kubeadm init phase etcd local --config /opt/replicated/kubeadm.conf
```

```python
from pathlib import Path
p = Path("/etc/kubernetes/manifests/etcd.yaml")
text = p.read_text()
old = "registry.k8s.io/etcd:3.6.8-0"
new = "docker.io/kurlsh/etcd:v3.6.15"
if old not in text:
    raise SystemExit("expected kubeadm etcd image was not in the manifest")
p.write_text(text.replace(old, new))
print("patched etcd image to", new)
```

## Step 6: Ensure Environment Profile Configuration

```shell
grep -q 'KUBECONFIG=/etc/kubernetes/admin.conf' /root/.bashrc \
  || printf '\nexport KUBECONFIG=/etc/kubernetes/admin.conf\n' >> /root/.bashrc
```

## Step 7: Restart Kubelet & Verify etcd Health

```shell
systemctl restart kubelet
# Check etcd container status
crictl ps --name etcd

# Verify endpoint health
ETCD=$(crictl ps --name etcd -q | head -1)
crictl exec "$ETCD" etcdctl endpoint health \
  --endpoints=https://127.0.x.x:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key
```

# Verification

```shell
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl get nodes -o wide
kubectl get --raw='/readyz'
kubectl get pods -n kube-system -o wide
kubectl get pods -A
```

Troubleshooting Matrix
\- etcd data under \`/var/lib/etcd\`
\- cluster CA, etcd certificates, front-proxy certificates, and service-account keys
\- \`admin.conf\` and \`kubelet.conf\`
\- application PVCs
\- HAProxy configuration
\- the existing static-pod manifests for the API server, controller manager, scheduler, and HAProxy, other than kubelet recreating their containers
If a later step fails

| Symptom                                 | Cause                           | Action                                                                       |
| --------------------------------------- | ------------------------------- | ---------------------------------------------------------------------------- |
| etcd exits on data-dir or WAL           | Cannot open existing database   | Do not delete data directory. Collect crictl logs and restore from snapshot. |
| etcd health check fails TLS             | Certificates mismatch           | Compare SANs with openssl on server.crt. Do not regenerate CA.               |
| API server cannot open apiserver.crt    | Cert not in mounted host path   | Verify /etc/kubernetes/pki/apiserver.crt exists on host.                     |
| API CrashLoopBackOff after etcd healthy | kubelet container backoff       | Run systemctl restart kubelet after etcd is confirmed healthy.               |
| Pods stuck on CreatePodSandbox          | Missing /run/flannel/subnet.env | Ensure API is healthy and flannel pod finishes setup.                        |
| kubectl defaults to localhost:8080      | Unset KUBECONFIG env variable   | Export KUBECONFIG=/etc/kubernetes/admin.conf or use login shell.             |