# Undelete PVs on Kubernetes
**Summary**: In this wiki page I'll show you how to recover from the message 'AttachVolume.Attach failed for volume "pvc-xxx" : PersistentVolume "pvc-xxx" is marked for deletion' when your pods won't start. \\
**Date**: 24 April 2026 \\
{{tag>kubernetes fix}}
One morning we got the call that one of our applications was done. Some investigation showed the cluster had automatically rebooted that night after an automatic upgrade and now some of the pods wouldn't come up again with the following error: 'AttachVolume.Attach failed for volume "pvc-xxx" : PersistentVolume "pvc-xxx" is marked for deletion'. I won't go into the root cause why that happened, but I will show you how to recover from that. Note that you can't undo the deletion. Kubernetes has marked the PVs for deletion and will do so. But we can save the data and recreate the PersistentVolumes and point them to the original disks.
## Find and Retain PersistentVolumes
The first step is to make sure the physical disks do not get deleted. Each PersistentVolume has a Reclaim Policy, which defines what to do when a PersistentVolume gets deleted. To save the disks from deletion, we need to set the Reclaim Policy to 'Retain'. First, find the affected PVs:
kubectl get pv -A
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-f5398340-6c03-4d01-badd-fecd5ce0f98b 64Gi RWO Delete Terminating ops/app-1 sbs-default 329d
pvc-83fe29bd-6f89-427b-8f23-3c34c82af6aa 10Gi RWO Delete Bound app/app-3 sbs-default 6d23h
pvc-9fff8f2a-cd22-43c1-9439-b244ea09ffb3 256Gi RWO Delete Terminating ops/app-2 scw-bssd 32m
pvc-7742a553-fab3-4a65-a9de-3e11f223c990 20Gi RWO Delete Terminating ops/app-1-db scw-bssd 2y324d
Note that you can see the affected PVs by checking the status. The terminating status indicated that the pv is marked for deletion. So now we need to set the Reclaim Policy for the affected PersistentVolumes to 'Retain':
kubectl patch pv pvc-f5398340-6c03-4d01-badd-fecd5ce0f98b -p "{\"spec\":{\"persistentVolumeReclaimPolicy\":\"Retain\"}}"
kubectl patch pv pvc-9fff8f2a-cd22-43c1-9439-b244ea09ffb3 -p "{\"spec\":{\"persistentVolumeReclaimPolicy\":\"Retain\"}}"
kubectl patch pv pvc-7742a553-fab3-4a65-a9de-3e11f223c990 -p "{\"spec\":{\"persistentVolumeReclaimPolicy\":\"Retain\"}}"
Now that we've patched the PersistentVolumes, we can again look at their status:
kubectl get pv -A
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-f5398340-6c03-4d01-badd-fecd5ce0f98b 64Gi RWO Retain Terminating ops/app-1 sbs-default 329d
pvc-83fe29bd-6f89-427b-8f23-3c34c82af6aa 10Gi RWO Delete Bound app/app-3 sbs-default 6d23h
pvc-9fff8f2a-cd22-43c1-9439-b244ea09ffb3 256Gi RWO Retain Terminating ops/app-2 scw-bssd 32m
pvc-7742a553-fab3-4a65-a9de-3e11f223c990 20Gi RWO Retain Terminating ops/app-1-db scw-bssd 2y324d
## Make New Manifests for the PersistentVolumes
Now that the underlying data is secure, we also need to make sure we can recreate the PVs using the underlying disks. To do so, make backups of the PV's manifest, and edit it to create new ones:
kubectl get pv pvc-f5398340-6c03-4d01-badd-fecd5ce0f98b -o yaml > app-1.yaml
kubectl get pv pvc-9fff8f2a-cd22-43c1-9439-b244ea09ffb3 -o yaml > app-2.yaml
kubectl get pv pvc-7742a553-fab3-4a65-a9de-3e11f223c990 -o yaml > app-1-db.yaml
Open the yaml files in your favorite editor and remove the following fields:
* metadata.annotations.kubectl.kubernetes.io/last-applied-configuration
* metadata.creationTimestamp
* metdata.deletionGracePeriodSeconds
* metadata.deletionTimestamp
* metdata.resourceVersion
* metdata.uid
* claimRef.resourceVersion
* status
Depending on your specific situation there might be adittional fields that you might need to remove. For every field you remove, think about what the field says. It is crucial to identify the claim and the disk, or is it set by kubernetes? If it is the latter, the field can probably be removed.
This is an example on one of the manifests:
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: csi.scaleway.com
finalizers:
- kubernetes.io/pv-protection
name: pvc-9fff8f2a-cd22-43c1-9439-b244ea09ffb3
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 256Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: app-2
namespace: ops
uid: 9fff8f2a-cd22-43c1-9439-b244ea09ffb3
csi:
driver: csi.scaleway.com
fsType: ext4
volumeAttributes:
encrypted: "false"
storage.kubernetes.io/csiProvisionerIdentity: 1661413450120-2026-csi.scaleway.com
volumeHandle: nl-xxx-n/f9a3b2d7-9eac-48d4-a32c-b1f69c80c13c
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: topology.csi.scaleway.com/zone
operator: In
values:
- nl-xxx-n
persistentVolumeReclaimPolicy: Retain
storageClassName: scw-bssd
volumeMode: Filesystem
As yuo can see, inside the `csi:` section there is a reference to the block storage device being used at the cloud provider (scaleway in this case)
## Remove and Recreate the PVs
Now that we've made sure the data is safe and that we can recreate the PVs, we can actually remove the PVs. To do so, start with scaling down the statefulsets:
kubectl scale statefulset app-1 --replicas=0
kubectl scale statefulset app-2 --replicas=0
kubectl scale statefulset app-1-db --replicas=0
This will remove the pods that are failing. Now to actually remove the PVs, edit the existing manifests and remove the finalizer that is preventing the PVs from getting deleted. You can read all about finalizers [[k8sforcedelete|here]], so there are two ways. I'll use the patch method here:
kubectl patch pv pvc-f5398340-6c03-4d01-badd-fecd5ce0f98b -p '{\"metadata\": {\"finalizers\": null}}' --type merge
kubectl patch pv pvc-9fff8f2a-cd22-43c1-9439-b244ea09ffb3 -p '{\"metadata\": {\"finalizers\": null}}' --type merge
kubectl patch pv pvc-7742a553-fab3-4a65-a9de-3e11f223c990 -p '{\"metadata\": {\"finalizers\": null}}' --type merge
Now, the PVs will be removed. It's time to recreate them. In the previous step we created manifests which we can now use to recreate the PVs:
kubectl apply -f app-1.yaml
kubectl apply -f app-2.yaml
kubectl apply -f app-1-db.yaml
Now the PVs are recreated again. To start using them, simply scale up the statefulsets again:
kubectl scale statefulset app-1 --replicas=1
kubectl scale statefulset app-2 --replicas=1
kubectl scale statefulset app-1-db --replicas=1
Now the pods will start up again, with all the data still in place.
//This wiki has been made possible by://