[HDDS-9113] [snapshot] Snapshot path contents listing is failing intermittently

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: Ozone Manager
• Assignee: Unassigned
• Reporter: Arun Sarin
• Created: 02/Aug/23 20:59
• Updated: 03/Aug/23 17:10
• Resolved: 03/Aug/23 17:10

Description

Snapshot path contents listing is failing intermittently 

2023-08-01 19:36:50,541|INFO|MainThread|machine.py:188 - run()||GUID=e574a938-a16e-4a1e-9c42-5786f286e727|RUNNING: /opt/cloudera/parcels/CDH/bin/ozone sh snapshot create o3://ozone1/vol-fiisp/buck-6j174 snap-2yg7s
2023-08-01 19:36:54,676|INFO|MainThread|machine.py:230 - run()||GUID=e574a938-a16e-4a1e-9c42-5786f286e727|Exit Code: 0
2023-08-01 19:36:54,679|INFO|MainThread|machine.py:188 - run()||GUID=7dd9a1cf-a3ca-4260-9eb2-13cdafeb2519|RUNNING: klist -k -t /home/hrt_qa/hadoopqa/keytabs/hrt_qa.headless.keytab | grep -v HTTP
2023-08-01 19:36:54,698|INFO|MainThread|machine.py:230 - run()||GUID=7dd9a1cf-a3ca-4260-9eb2-13cdafeb2519|Exit Code: 0
2023-08-01 19:36:54,699|INFO|MainThread|machine.py:2131 - get_principal_from_user()|--- user principal is [e-mail]
2023-08-01 19:36:54,700|INFO|MainThread|machine.py:188 - run()||GUID=d0dd3259-5244-416c-afc0-adf6185698ad|RUNNING: /opt/cloudera/parcels/CDH/bin/ozone fs -ls -R ofs://ozone1/vol-fiisp/buck-6j174/.snapshot/snap-2yg7s
2023-08-01 19:37:02,974|INFO|MainThread|machine.py:203 - run()||GUID=d0dd3259-5244-416c-afc0-adf6185698ad|ls: `ofs://ozone1/vol-fiisp/buck-6j174/.snapshot/snap-2yg7s': No such file or directory
2023-08-01 19:37:03,023|INFO|MainThread|machine.py:232 - run()||GUID=d0dd3259-5244-416c-afc0-adf6185698ad|Exit Code: 1
2023-08-01 19:37:03,023|INFO|MainThread|machine.py:238 - run()|Command /opt/cloudera/parcels/CDH/bin/ozone fs -ls -R ofs://ozone1/vol-fiisp/buck-6j174/.snapshot/snap-2yg7s failed after 0 retries 
2023-08-01 19:37:03,037|INFO|MainThread|conftest.py:272 - pytest_runtest_makereport()|call: <CallInfo when='call' exception: Snapshot path contents list failed
assert 1 == 0>

Snapshot creation was successful [validated in om-audit.log]

2023-08-01 19:36:54,627 | INFO  | OMAudit | user=[e-mail] | ip=10.64.62.46 | op=CREATE_SNAPSHOT {volume=vol-fiisp, bucket=buck-6j174, snapshotName=snap-2yg7s} | ret=SUCCESS | 

The volume and bucket has space and namespace quota set -

Volume quota for current test SpaceQuota: 8053063680,NameSpace quota: 19
Bucket quota for current test SpaceQuota: 8053063680,NameSpace quota: 19 

ozone-om.log error stack-trace :

2023-08-01 19:36:50,236 INFO [SstFilteringService#0]-org.apache.hadoop.ozone.om.snapshot.SnapshotCache: Loading snapshot. Table key: /vol-dsjsj/buck-h5ed5/snap-4w2ww
2023-08-01 19:36:50,237 ERROR [SstFilteringService#0]-org.apache.hadoop.ozone.om.SstFilteringService: Error during Snapshot sst filtering 
FILE_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Unable to load snapshot. Snapshot with table key '/vol-dsjsj/buck-h5ed5/snap-4w2ww' is no longer active
    at org.apache.hadoop.ozone.om.snapshot.SnapshotCache.get(SnapshotCache.java:205)
    at org.apache.hadoop.ozone.om.snapshot.SnapshotCache.get(SnapshotCache.java:151)
    at org.apache.hadoop.ozone.om.SstFilteringService$SstFilteringTask.call(SstFilteringService.java:178)
    at org.apache.hadoop.hdds.utils.BackgroundService$PeriodicalTask.lambda$run$0(BackgroundService.java:121)
    at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834) 

Issue Links

• duplicates: HDDS-9112 Make snapshot dir wait poll max timeout configurable