[HDFS-12337] Ozone: Concurrent RocksDB open calls fail because of "No locks available"

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: HDFS-7240
• Fix Version/s: HDFS-7240
• Component/s: ozone
• Target Version/s: HDFS-7240
• Hadoop Flags: Reviewed
• Assignee: Mukul Kumar Singh
• Reporter: Mukul Kumar Singh
• Created: 22/Aug/17 12:49
• Updated: 26/Apr/18 21:53
• Resolved: 23/Aug/17 17:25

Description

HDFS-12216 fixes the issue with the static container cache and re-using the same container port on datanode restart. However TestKeys still fails after HDFS-12216 is fixed.

The test is now failing because Concurrent RocksDB open calls fail. In the current code BlockDeleting service and Dispatcher tries to open the db concurrently.

This jira will also fix the keepPort property for Ratis Container port and set the reuse address property for XceiverServerRatis correctly.

2017-08-22 16:51:34,453 [BlockDeletingService#1] INFO  utils.RocksDBStore (RocksDBStore.java:<init>(64)) - opening db file /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db
2017-08-22 16:51:34,460 [nioEventLoopGroup-9-1] INFO  logging.LoggingHandler (Slf4JLogger.java:info(101)) - [id: 0x8822cd3d, /0.0.0.0:57044] RECEIVED: [id: 0x61367e6f, /127.0.0.1:57173 => /127.0.0
.1:57044]
2017-08-22 16:51:34,461 [nioEventLoopGroup-10-1] INFO  utils.RocksDBStore (RocksDBStore.java:<init>(64)) - opening db file /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db
2017-08-22 16:51:34,465 [nioEventLoopGroup-10-1] INFO  utils.RocksDBStore (RocksDBStore.java:<init>(67)) - Failed init RocksDB, db path : /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.dbexception org.rocksdb.RocksDBException: lock /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db/LOCK: No locks available
2017-08-22 16:51:34,465 [BlockDeletingService#1] INFO  background.BlockDeletingService (BlockDeletingService.java:call(203))      - The elapsed time of task@70a576ee for deleting blocks: 12ms.
2017-08-22 16:51:34,474 [nioEventLoopGroup-10-1] INFO  impl.Dispatcher (ContainerUtils.java:logAndReturnError(129))      - Operation: GetKey : Trace ID: 73f19131-f63b-459a-8f09-9a3db893a296 : Message: 621a3b15-b9fc-4d49-a6e6-29d4c40cc91f : Result: UNABLE_TO_READ_METADATA_DB
2017-08-22 16:51:34,475 [Thread-382] INFO  exceptions.OzoneExceptionMapper (OzoneExceptionMapper.java:toResponse(39)) ozone  c2a23759-c76f-49ea-b574-f0802a4e5b75/c0df3a48-f75b-4b5e-b1bd-c189ce698056/13b3d486-3d7a-49e4-bc9d-1ef63e674548 hdfs 73f19131-f63b-459a-8f09-9a3db893a296 - Returning exception. ex: {"httpCode":500,"shortMessage":"internalServerError","resource":"hdfs","message":"621a3b15-b9fc-4d49-a6e6-29d4c40cc91f","requestID":"73f19131-f63b-459a-8f09-9a3db893a296","hostName":"hw13605.local"}

Issue Links

• duplicates: HDFS-12343 TestKeys#testPutAndGetKeyWithDnRestart failed