Deleting an object whose OCFL root is on NFS failed partway through the purge:
WARN (FileUtil) Failed to delete file: <root>/db6/.../v2/content/.fcrepo
java.nio.file.DirectoryNotEmptyException: <root>/db6/.../v2/content/.fcrepo
at io.ocfl.core.util.FileUtil.lambda$deleteDirectory$4(FileUtil.java:329)
at io.ocfl.core.util.FileUtil.deleteDirectory(FileUtil.java:327)
at io.ocfl.core.storage.filesystem.FileSystemStorage.deleteDirectory(FileSystemStorage.java:296)
at io.ocfl.core.storage.DefaultOcflStorage.purgeObject(DefaultOcflStorage.java:333)
Caused by: io.ocfl.api.exception.CorruptObjectException: Failed to purge object ... The object may need to be deleted manually.
at io.ocfl.core.storage.DefaultOcflStorage.purgeObject(DefaultOcflStorage.java:336)
No WARN was logged for any file inside .fcrepo, so every Files.delete succeeded and the directory was still non-empty at rmdir time. We suspect this was due to an NFS silly-rename: a file held open elsewhere is renamed to .nfsXXXX instead of unlinked. The entry cleared on its own shortly afterwards.
Delete order is incidental.** The order comes from Comparator.reverseOrder(), not from a decision. Here it happened to remove v2/ before inventory.json. Failing one step later, while deleting v1/, would have left a root inventory advertising versions that no longer exist
It might be worth a retry loop with a backoff in deleteDirectory when a DirectoryNotEmptyException is thrown.
In purgeObject, delete the root inventory, its sidecar and the namaste file first, then the rest. This mirrors the write path's inventory-last commit point: after that, a partial failure leaves identifiable garbage rather than a half-object. Works on S3 as well as filesystem.
What is not clear, is what process was holding the file open for NFS to silly-rewrite. It might have been a client application, it might have been dcfl-java via Fedora. Either way, the retry would likely resolve the issue.
Deleting an object whose OCFL root is on NFS failed partway through the purge:
No WARN was logged for any file inside
.fcrepo, so everyFiles.deletesucceeded and the directory was still non-empty atrmdirtime. We suspect this was due to an NFS silly-rename: a file held open elsewhere is renamed to.nfsXXXXinstead of unlinked. The entry cleared on its own shortly afterwards.Delete order is incidental.** The order comes from
Comparator.reverseOrder(), not from a decision. Here it happened to removev2/beforeinventory.json. Failing one step later, while deletingv1/, would have left a root inventory advertising versions that no longer existIt might be worth a retry loop with a backoff in deleteDirectory when a DirectoryNotEmptyException is thrown.
In
purgeObject, delete the root inventory, its sidecar and the namaste file first, then the rest. This mirrors the write path's inventory-last commit point: after that, a partial failure leaves identifiable garbage rather than a half-object. Works on S3 as well as filesystem.What is not clear, is what process was holding the file open for NFS to silly-rewrite. It might have been a client application, it might have been dcfl-java via Fedora. Either way, the retry would likely resolve the issue.