> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chainstack.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Diagnose and resolve common Chainstack Self-Hosted issues including pod failures, storage problems, network connectivity, and node synchronization errors.

This guide helps you diagnose and resolve common issues with Chainstack Self-Hosted.

## Installation troubleshooting

### Prerequisites check fails

Installer fails to verify the requirements.

#### kubectl not found

```bash theme={"system"}
# Verify kubectl is installed
which kubectl

# For k3s, ensure KUBECONFIG is set
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
```

#### Helm version too old

```bash theme={"system"}
# Check version (v3+ required)
helm version

# Update Helm if needed
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
```

#### yq not found or wrong version

```bash theme={"system"}
# Must be mikefarah/yq v4+
yq --version

# Install correct version
wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/local/bin/yq
chmod +x /usr/local/bin/yq
```

### Installation times out

Helm install hangs or times out after 15 minutes.

```bash theme={"system"}
# Check pod status
kubectl get pods -n control-panel

# Look for pods stuck in Pending or CrashLoopBackOff
kubectl describe pod <pod-name> -n control-panel
```

#### PersistentVolumeClaim pending

```bash theme={"system"}
kubectl get pvc -n control-panel
kubectl describe pvc <pvc-name> -n control-panel
```

Ensure storage class exists and is default, and verify sufficient disk space.

#### Image pull errors

```bash theme={"system"}
kubectl describe pod <pod-name> -n control-panel | grep Events
```

Check network connectivity to container registry and verify no firewall is blocking image pulls.

#### Insufficient resources

```bash theme={"system"}
kubectl describe node | grep -A 10 "Allocated resources"
```

Ensure node has enough CPU and memory.

### Pods crash looping

Pods repeatedly restart (CrashLoopBackOff status).

```bash theme={"system"}
# Check pod logs
kubectl logs <pod-name> -n control-panel

# Check previous container logs if crashing
kubectl logs <pod-name> -n control-panel --previous

# Describe pod for events
kubectl describe pod <pod-name> -n control-panel
```

#### Database connection issues

Check PostgreSQL pod is running and verify database credentials match.

```bash theme={"system"}
kubectl logs cp-pg-postgresql-0 -n control-panel
```

#### Configuration errors

Review the values file for typos and ensure all required environment variables are set.

## Access troubleshooting

### Cannot access web interface

Browser cannot reach the Control Panel.

#### Verify UI service exists

```bash theme={"system"}
kubectl get svc cp-cp-ui -n control-panel
```

#### Check UI pod is running

```bash theme={"system"}
kubectl get pods -l app.kubernetes.io/name=cp-ui -n control-panel
```

#### Verify external service

```bash theme={"system"}
kubectl get svc -n control-panel | grep -E "(LoadBalancer|NodePort)"
```

#### Test with port forward

```bash theme={"system"}
kubectl port-forward svc/cp-cp-ui 8080:80 -n control-panel --address 0.0.0.0
```

Then access `http://<server-ip>:8080`

#### Check firewall

Ensure the port is open on your server and check cloud provider security groups if applicable.

### Login fails with valid credentials

The login page loads and accepts credentials, but shows "Login failed".

#### Backend URL points to in-cluster address

This is the most common cause. The UI is configured to reach the deployments API at an in-cluster URL that your browser cannot resolve.

1. Check the configured backend URL:

   ```bash theme={"system"}
   kubectl get configmap cp-cp-ui-config -n control-panel -o jsonpath='{.data.config\.json}'
   ```

   If the output shows `{"apiBaseUrl":"http://cp-cp-deployments-api"}`, the UI is using the in-cluster URL.

2. Patch the configmap to use your server's external address:

   ```bash theme={"system"}
   kubectl patch configmap cp-cp-ui-config -n control-panel \
     --type merge \
     -p '{"data":{"config.json":"{\"apiBaseUrl\":\"http://<SERVER-IP>:8081\"}"}}'
   ```

3. Restart the UI pod to pick up the change:

   ```bash theme={"system"}
   kubectl rollout restart deployment cp-cp-ui -n control-panel
   ```

4. Expose the deployments API on port 8081:

   ```bash theme={"system"}
   kubectl port-forward svc/cp-cp-deployments-api 8081:8080 -n control-panel --address 0.0.0.0 &
   ```

5. Hard-refresh the login page and try again.

<Note>
  This happens when the installer runs with `--yes` or when you press Enter at the backend URL prompt, accepting the in-cluster default. The in-cluster URL works for pod-to-pod communication but is not reachable from a browser outside the cluster.
</Note>

#### Check auth service

```bash theme={"system"}
kubectl get pods -l app.kubernetes.io/name=cp-auth -n control-panel
kubectl logs -l app.kubernetes.io/name=cp-auth -n control-panel
```

#### Check Keycloak

```bash theme={"system"}
kubectl get pods -l app.kubernetes.io/name=keycloak -n control-panel
kubectl logs cp-keycloak-0 -n control-panel
```

#### Verify credentials from values file

```bash theme={"system"}
grep BOOTSTRAP ~/.config/cp-suite/values/cp-*.yaml
```

## Node deployment troubleshooting

### Node stuck in deploying state

Node deployment does not progress.

#### Check workflows service

```bash theme={"system"}
kubectl logs -l app.kubernetes.io/name=cp-workflows -n control-panel
```

#### Check Temporal

```bash theme={"system"}
kubectl get pods -n control-panel | grep temporal
```

#### Verify resource availability

```bash theme={"system"}
kubectl describe nodes | grep -A10 "Allocated resources"
```

#### Check deployments API logs

```bash theme={"system"}
kubectl logs -l app.kubernetes.io/name=cp-deployments-api -n control-panel
```

### Node pod errors

Blockchain node pods run in the `control-panel-deployments` namespace. Use the node ID from **Chainstack Self-Hosted > Nodes > Node details** to find the relevant pods.

#### Check node pod status

```bash theme={"system"}
kubectl get pods -n control-panel-deployments | grep <node-id>
```

#### Check node pod logs

```bash theme={"system"}
# Execution client (reth)
kubectl logs -f <pod-name> -n control-panel-deployments

# Previous container logs if crashing
kubectl logs <pod-name> -n control-panel-deployments --previous
```

#### Check node PVC status

```bash theme={"system"}
kubectl get pvc -n control-panel-deployments | grep <node-id>
```

### Transient `PopulatorDataSourceNotFound` warning

During the first \~10 seconds of a node deployment, `kubectl get events` may show a warning on the reth PVC:

```text theme={"system"}
Warning  PopulatorDataSourceNotFound  ...  Data source control-panel-deployments/cp-<node-id>-reth-snapshot not found
```

This is a brief race condition between PVC creation and the snapshot data source being created by the Control Panel workflows. It resolves automatically once `cp-workflows` finishes the post-apply hook (typically within 10–15 seconds). No action is required.

If the warning persists for more than a minute, check `cp-workflows` and `cp-bolt` logs:

```bash theme={"system"}
kubectl logs -n control-panel -l app.kubernetes.io/name=cp-workflows --tail=50
kubectl logs -n control-panel -l app.kubernetes.io/name=cp-bolt --tail=50
```

## System health checks

### Check all components

Run this comprehensive health check:

```bash theme={"system"}
#!/bin/bash
echo "=== Checking Control Panel Health ==="

echo -e "\n--- Pod Status (Control Panel) ---"
kubectl get pods -n control-panel

echo -e "\n--- Pod Status (Node deployments) ---"
kubectl get pods -n control-panel-deployments

echo -e "\n--- Services ---"
kubectl get svc -n control-panel

echo -e "\n--- PVC Status (Control Panel) ---"
kubectl get pvc -n control-panel

echo -e "\n--- PVC Status (Node deployments) ---"
kubectl get pvc -n control-panel-deployments

echo -e "\n--- Recent Events ---"
kubectl get events -n control-panel --sort-by='.lastTimestamp' | tail -20
kubectl get events -n control-panel-deployments --sort-by='.lastTimestamp' | tail -20
```

### Check resource usage

```bash theme={"system"}
# Node resources
kubectl top nodes

# Pod resources (requires metrics-server)
kubectl top pods -n control-panel
```

### Check logs for errors

```bash theme={"system"}
# All pods, last hour
timestamp=$(date +%s)
touch debug-$timestamp.txt
debug=debug-$timestamp.txt
for pod in $(kubectl get pods -n control-panel -o name); do
  echo "=== $pod ===" >> $debug
  kubectl logs $pod -n control-panel --since=1h 2>/dev/null >> $debug
done
for pod in $(kubectl get pods -n control-panel-deployments -o name); do
  echo "=== $pod ===" >> $debug
  kubectl logs $pod -n control-panel-deployments --since=1h 2>/dev/null >> $debug
done
```

## Recovery procedures

### Restart all services

```bash theme={"system"}
# Restart all deployments
kubectl rollout restart deployment -n control-panel

# Watch pods restart
kubectl get pods -n control-panel -w
```

### Reset to clean state

<Warning>
  This destroys all data. Only use as last resort.
</Warning>

```bash theme={"system"}
# Uninstall completely
cpctl uninstall

# Reinstall
cpctl install
```

### Recover from database issues

If PostgreSQL has issues:

```bash theme={"system"}
# Check PostgreSQL logs
kubectl logs cp-pg-postgresql-0 -n control-panel

# Check PVC status
kubectl get pvc -n control-panel | grep postgresql
```

## Getting help

If you cannot resolve an issue, check the [FAQ](/docs/self-hosted/faq) for known issues.

### Gather diagnostic information

```bash theme={"system"}
cpctl status > diagnostic.txt
kubectl get events -n control-panel >> diagnostic.txt
kubectl get events -n control-panel-deployments >> diagnostic.txt
timestamp=$(date +%s)
touch debug-$timestamp.txt
debug=debug-$timestamp.txt
for pod in $(kubectl get pods -n control-panel -o name); do
  echo "=== $pod ===" >> $debug
  kubectl logs $pod -n control-panel --since=1h 2>/dev/null >> $debug
done
for pod in $(kubectl get pods -n control-panel-deployments -o name); do
  echo "=== $pod ===" >> $debug
  kubectl logs $pod -n control-panel-deployments --since=1h 2>/dev/null >> $debug
done
```

### Contact support

* [Chainstack Support Center](https://support.chainstack.com/hc/en-us)
* [Chainstack Feedback](https://ideas.chainstack.com/en?b=6968ecf76c93cb94db0422c8)

When contacting support, include:

* Version of Chainstack Self-Hosted
* Kubernetes distribution and version
* `diagnostic.txt` file
* `debug-$timestamp.txt` file
* Other relevant error messages and logs
* Steps to reproduce the issue