Summary: In this wiki page I'll show you how to recover from the message 'AttachVolume.Attach failed for volume “pvc-xxx” : PersistentVolume “pvc-xxx” is marked for deletion' when your pods won't start.
Date: 24 April 2026

One morning we got the call that one of our applications was done. Some investigation showed the cluster had automatically rebooted that night after an automatic upgrade and now some of the pods wouldn't come up again with the following error: 'AttachVolume.Attach failed for volume “pvc-xxx” : PersistentVolume “pvc-xxx” is marked for deletion'. I won't go into the root cause why that happened, but I will show you how to recover from that. Note that you can't undo the deletion. Kubernetes has marked the PVs for deletion and will do so. But we can save the data and recreate the PersistentVolumes and point them to the original disks.

The first step is to make sure the physical disks do not get deleted. Each PersistentVolume has a Reclaim Policy, which defines what to do when a PersistentVolume gets deleted. To save the disks from deletion, we need to set the Reclaim Policy to 'Retain'. First, find the affected PVs:

kubectl get pv -A
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM               STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-f5398340-6c03-4d01-badd-fecd5ce0f98b   64Gi       RWO            Delete           Terminating   ops/app-1           sbs-default    <unset>                          329d
pvc-83fe29bd-6f89-427b-8f23-3c34c82af6aa   10Gi       RWO            Delete           Bound         app/app-3           sbs-default    <unset>                          6d23h
pvc-9fff8f2a-cd22-43c1-9439-b244ea09ffb3   256Gi      RWO            Delete           Terminating   ops/app-2           scw-bssd       <unset>                          32m
pvc-7742a553-fab3-4a65-a9de-3e11f223c990   20Gi       RWO            Delete           Terminating   ops/app-1-db        scw-bssd       <unset>                          2y324d

Note that you can see the affected PVs by checking the status. The terminating status indicated that the pv is marked for deletion. So now we need to set the Reclaim Policy for the affected PersistentVolumes to 'Retain':

kubectl patch pv pvc-f5398340-6c03-4d01-badd-fecd5ce0f98b -p "{\"spec\":{\"persistentVolumeReclaimPolicy\":\"Retain\"}}"
kubectl patch pv pvc-9fff8f2a-cd22-43c1-9439-b244ea09ffb3 -p "{\"spec\":{\"persistentVolumeReclaimPolicy\":\"Retain\"}}"
kubectl patch pv pvc-7742a553-fab3-4a65-a9de-3e11f223c990 -p "{\"spec\":{\"persistentVolumeReclaimPolicy\":\"Retain\"}}"

Now that we've patched the PersistentVolumes, we can again look at their status:

kubectl get pv -A
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM               STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-f5398340-6c03-4d01-badd-fecd5ce0f98b   64Gi       RWO            Retain           Terminating   ops/app-1           sbs-default    <unset>                          329d
pvc-83fe29bd-6f89-427b-8f23-3c34c82af6aa   10Gi       RWO            Delete           Bound         app/app-3           sbs-default    <unset>                          6d23h
pvc-9fff8f2a-cd22-43c1-9439-b244ea09ffb3   256Gi      RWO            Retain           Terminating   ops/app-2           scw-bssd       <unset>                          32m
pvc-7742a553-fab3-4a65-a9de-3e11f223c990   20Gi       RWO            Retain           Terminating   ops/app-1-db        scw-bssd       <unset>                          2y324d

Now that the underlying data is secure, we also need to make sure we can recreate the PVs using the underlying disks. To do so, make backups of the PV's manifest, and edit it to create new ones:

kubectl get pv pvc-f5398340-6c03-4d01-badd-fecd5ce0f98b -o yaml > app-1.yaml
kubectl get pv pvc-9fff8f2a-cd22-43c1-9439-b244ea09ffb3 -o yaml > app-2.yaml
kubectl get pv pvc-7742a553-fab3-4a65-a9de-3e11f223c990 -o yaml > app-1-db.yaml

Open the yaml files in your favorite editor and remove the following fields:

• metadata.annotations.kubectl.kubernetes.io/last-applied-configuration
• metadata.creationTimestamp
• metdata.deletionGracePeriodSeconds
• metadata.deletionTimestamp
• metdata.resourceVersion
• metdata.uid
• claimRef.resourceVersion
• status

Depending on your specific situation there might be adittional fields that you might need to remove. For every field you remove, think about what the field says. It is crucial to identify the claim and the disk, or is it set by kubernetes? If it is the latter, the field can probably be removed.

This is an example on one of the manifests:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: csi.scaleway.com
  finalizers:
    - kubernetes.io/pv-protection
  name: pvc-9fff8f2a-cd22-43c1-9439-b244ea09ffb3
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 256Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: app-2
    namespace: ops
    uid: 9fff8f2a-cd22-43c1-9439-b244ea09ffb3
  csi:
    driver: csi.scaleway.com
    fsType: ext4
    volumeAttributes:
      encrypted: "false"
      storage.kubernetes.io/csiProvisionerIdentity: 1661413450120-2026-csi.scaleway.com
    volumeHandle: nl-xxx-n/f9a3b2d7-9eac-48d4-a32c-b1f69c80c13c
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.csi.scaleway.com/zone
              operator: In
              values:
                - nl-xxx-n
  persistentVolumeReclaimPolicy: Retain
  storageClassName: scw-bssd
  volumeMode: Filesystem

As yuo can see, inside the csi: section there is a reference to the block storage device being used at the cloud provider (scaleway in this case)

Now that we've made sure the data is safe and that we can recreate the PVs, we can actually remove the PVs. To do so, start with scaling down the statefulsets:

kubectl scale statefulset app-1 --replicas=0
kubectl scale statefulset app-2 --replicas=0
kubectl scale statefulset app-1-db --replicas=0

This will remove the pods that are failing. Now to actually remove the PVs, edit the existing manifests and remove the finalizer that is preventing the PVs from getting deleted. You can read all about finalizers here, so there are two ways. I'll use the patch method here:

kubectl patch pv pvc-f5398340-6c03-4d01-badd-fecd5ce0f98b -p '{\"metadata\": {\"finalizers\": null}}' --type merge
kubectl patch pv pvc-9fff8f2a-cd22-43c1-9439-b244ea09ffb3 -p '{\"metadata\": {\"finalizers\": null}}' --type merge
kubectl patch pv pvc-7742a553-fab3-4a65-a9de-3e11f223c990 -p '{\"metadata\": {\"finalizers\": null}}' --type merge

Now, the PVs will be removed. It's time to recreate them. In the previous step we created manifests which we can now use to recreate the PVs:

kubectl apply -f app-1.yaml
kubectl apply -f app-2.yaml
kubectl apply -f app-1-db.yaml

Now the PVs are recreated again. To start using them, simply scale up the statefulsets again:

kubectl scale statefulset app-1 --replicas=1
kubectl scale statefulset app-2 --replicas=1
kubectl scale statefulset app-1-db --replicas=1

Now the pods will start up again, with all the data still in place.

This wiki has been made possible by: