[StackOverflow/kubernetes] Airflow Kubernetes Pods Exception ERROR - (404) Reason: Not Found

### ROOT CAUSE The 404 error indicates that the Kubernetes API server cannot find the requested resource (likely a Pod). This could be caused by: 1. **Incorrect resource reference**: The client (e.g., Airflow) is querying a resource that no longer exists or was never properly created. 2. **Namespace mismatch**: The pod exists but in a different namespace than the one being queried. 3. **Pod deletion**: The pod was created but immediately terminated (e.g., due to resource constraints or startup failure). 4. **API server misconfiguration**: Authentication/authorization issues or incorrect endpoint configuration. ### CODE FIX To resolve this, ensure the following: 1. **Verify Pod Creation**: ```python # Example: Check if pod exists before accessing from kubernetes import client, config from kubernetes.client.exceptions import ApiException config.load_kube_config() v1 = client.CoreV1Api() try: ret = v1.read_namespaced_pod(pod_name="your-pod-name", namespace="your-namespace") print("Pod exists") except ApiException as e: print(f"Pod not found: {e}") ``` 2. **Check Permissions**: Ensure the service account used by the client has the `get` permission for pods in the target namespace: ```yaml # Example RBAC rule apiVersion: rbac.authorization.kubernetes.io/v1 kind: ClusterRole rules: - apiGroups: [""] resources: ["pods"] verbs: ["get"] ``` 3. **Namespace Validation**: Confirm the pod exists in the correct namespace: ```bash kubectl get pods -n your-namespace ``` 4. **Handle Transient States**: If pods are created and deleted quickly, implement retries with exponential backoff: ```python import time from kubernetes.stream import stream def get_pod_logs(pod_name, namespace): for attempt in range(5): try: logs = stream(v1, "get pod/{}/log".format(pod_name), namespace=namespace) return logs except ApiException as e: if e.status == 404: time.sleep(2**attempt) # Exponential backoff else: raise ``` 5. **Debugging**: Use `kubectl describe pod` to check pod events: ```bash kubectl describe pod your-pod-name -n your-namespace ``` **Note**: The specific fix depends on the context (e.g., Airflow configuration, pod creation logic). If the issue persists, investigate the pod's lifecycle events and resource constraints.