[HDDS-8072] [snapshot] OM error due to 'Error during Snapshot sst filtering'

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Resolved
• Component/s: Ozone Manager
• Labels: ozone-snapshot
• Assignee: Dave Teng
• Reporter: Jyotirmoy Sinha
• Created: 03/Mar/23 04:16
• Updated: 29/Mar/23 20:54
• Resolved: 29/Mar/23 20:54

Description

After creating 13525 snapshots on a cluster across various volume/buckets there is an OM error due to 'Error during Snapshot sst filtering'

ozone-om.log stacktrace -

2023-03-02 20:24:13,850 ERROR org.apache.hadoop.ozone.om.SstFilteringService: Error during Snapshot sst filtering
java.io.IOException: Failed init RocksDB, db path : /var/lib/hadoop-ozone/om/data/db.snapshots/om.db-efd0f99b-3689-4ed6-a671-d2cb4d5fa982, exception :org.rocksdb.RocksDBException While open a file for random read: /var/lib/hadoop-ozone/om/data/db.snapshots/om.db-efd0f99b-3689-4ed6-a671-d2cb4d5fa982/000152.sst: Too many open files
        at org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:180)
        at org.apache.hadoop.hdds.utils.db.DBStoreBuilder.build(DBStoreBuilder.java:219)
        at org.apache.hadoop.ozone.om.OmMetadataManagerImpl.loadDB(OmMetadataManagerImpl.java:481)
        at org.apache.hadoop.ozone.om.SstFilteringService$SstFilteringTask.call(SstFilteringService.java:147)
        at org.apache.hadoop.hdds.utils.BackgroundService$PeriodicalTask.lambda$run$0(BackgroundService.java:121)
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: class org.apache.hadoop.hdds.utils.db.RocksDatabase: Failed to open /var/lib/hadoop-ozone/om/data/db.snapshots/om.db-efd0f99b-3689-4ed6-a671-d2cb4d5fa982; status : IOError; message : While open a file for random read: /var/lib/hadoop-ozone/om/data/db.snapshots/om.db-efd0f99b-3689-4ed6-a671-d2cb4d5fa982/000152.sst: Too many open files
        at org.apache.hadoop.hdds.utils.HddsServerUtil.toIOException(HddsServerUtil.java:576)
        at org.apache.hadoop.hdds.utils.db.RocksDatabase.toIOException(RocksDatabase.java:85)
        at org.apache.hadoop.hdds.utils.db.RocksDatabase.open(RocksDatabase.java:162)
        at org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:116)
        ... 12 more
Caused by: org.rocksdb.RocksDBException: While open a file for random read: /var/lib/hadoop-ozone/om/data/db.snapshots/om.db-efd0f99b-3689-4ed6-a671-d2cb4d5fa982/000152.sst: Too many open files
        at org.rocksdb.RocksDB.openROnly(Native Method)
        at org.rocksdb.RocksDB.openReadOnly(RocksDB.java:488)
        at org.rocksdb.RocksDB.openReadOnly(RocksDB.java:443)
        at org.apache.hadoop.hdds.utils.db.managed.ManagedRocksDB.openReadOnly(ManagedRocksDB.java:45)
        at org.apache.hadoop.hdds.utils.db.RocksDatabase.open(RocksDatabase.java:145)
        ... 13 more 

 

Issue Links

• is fixed by: HDDS-8120 Close rocksDB connection after SstFilteringTask finishes