[HDDS-9543] Snapshot diff is failing with Dir object Ids required but not found in bucket

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Snapshot
• Assignee: Swaminathan Balachandran
• Reporter: Varsha Ravi
• Created: 26/Oct/23 05:40
• Updated: 01/Nov/23 17:29

Description

Snapshot diff failed with the below error

2023-10-25 05:10:03,296 INFO [IPC Server handler 45 on 9862]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager: Submitting snap diff report generation request for volume: vol304, bucket: bucket651, fromSnapshot: cm-1546356022-1698230795068-7 and toSnapshot: cm-1546356022-1698230795068-8
2023-10-25 05:10:03,296 INFO [snapshot-diff-job-thread-id-5]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager: Started snap diff report generation for volume: 'vol304', bucket: 'bucket651', fromSnapshot: 'cm-1546356022-1698230795068-7', toSnapshot: 'cm-1546356022-1698230795068-8'
2023-10-25 05:10:03,297 INFO [snapshot-diff-job-thread-id-5]-org.apache.hadoop.ozone.om.snapshot.SnapshotCache: Loading snapshot. Table key: /vol304/bucket651/cm-1546356022-1698230795068-7
2023-10-25 05:10:03,297 INFO [snapshot-diff-job-thread-id-5]-org.apache.hadoop.ozone.om.snapshot.SnapshotCache: Loading snapshot. Table key: /vol304/bucket651/cm-1546356022-1698230795068-8
2023-10-25 05:10:03,535 ERROR [snapshot-diff-job-thread-id-5]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager: Caught unchecked exception during diff report generation for volume: vol304 bucket: bucket651, fromSnapshot: cm-1546356022-1698230795068-7 and toSnapshot: cm-1546356022-1698230795068-8
java.lang.IllegalArgumentException: Dir object Ids required but not found in bucket: [-9223372036824369663]
        at org.apache.hadoop.ozone.om.snapshot.FSODirectoryPathResolver.getAbsolutePathForObjectIDs(FSODirectoryPathResolver.java:104)
        at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$5(SnapshotDiffManager.java:982)
        at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.generateSnapshotDiffReport(SnapshotDiffManager.java:1015)
        at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$2(SnapshotDiffManager.java:742)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Steps to reproduce:

• Create the below directory structure in cluster.

• ozone fs -ls -R ofs://ozone1/vol1/buck1
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:23 ofs://ozone1/vol1/buck1/dir1
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:23 ofs://ozone1/vol1/buck1/dir2
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:23 ofs://ozone1/vol1/buck1/dir3
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:23 ofs://ozone1/vol1/buck1/dir4
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:24 ofs://ozone1/vol1/buck1/dir5
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:24 ofs://ozone1/vol1/buck1/dir6
  -rw-rw-rw-   3 hrt_qa hrt_qa       3476 2023-10-25 15:25 ofs://ozone1/vol1/buck1/dir6/file6
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:24 ofs://ozone1/vol1/buck1/dir7
  -rw-rw-rw-   3 hrt_qa hrt_qa       3476 2023-10-25 15:26 ofs://ozone1/vol1/buck1/dir7/file7
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:24 ofs://ozone1/vol1/buck1/dir8
  -rw-rw-rw-   3 hrt_qa hrt_qa       3476 2023-10-25 15:26 ofs://ozone1/vol1/buck1/dir8/file8
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:24 ofs://ozone1/vol1/buck1/dir9
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:24 ofs://ozone1/vol1/buck1/dir9/dir10
  -rw-rw-rw-   3 hrt_qa hrt_qa       3476 2023-10-25 15:26 ofs://ozone1/vol1/buck1/dir9/file9

• Create snapshot snap1
• Change the directory to below structure by deleting, moving files and folders.

• ozone fs -ls -R ofs://ozone1/vol1/buck1
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:23 ofs://ozone1/vol1/buck1/dir1
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:31 ofs://ozone1/vol1/buck1/dir4
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:24 ofs://ozone1/vol1/buck1/dir4/dir5
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:33 ofs://ozone1/vol1/buck1/dir6
  -rw-rw-rw-   3 hrt_qa hrt_qa       3476 2023-10-25 15:25 ofs://ozone1/vol1/buck1/dir6/file6-new
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:24 ofs://ozone1/vol1/buck1/dir7
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:24 ofs://ozone1/vol1/buck1/dir8
  -rw-rw-rw-   3 hrt_qa hrt_qa       3476 2023-10-25 15:26 ofs://ozone1/vol1/buck1/dir8/file8
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:34 ofs://ozone1/vol1/buck1/dir9
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:24 ofs://ozone1/vol1/buck1/dir9/dir10
  drwxrwxrwx   - hrt_qa hrt_qa          0 2023-10-25 15:23 ofs://ozone1/vol1/buck1/dir9/dir3
  -rw-rw-rw-   3 hrt_qa hrt_qa       3476 2023-10-25 15:26 ofs://ozone1/vol1/buck1/dir9/file9-new

• Create snapshot snap2
• Run snapshot diff between snap1 and snap2