[HBASE-18379] SnapshotManager#checkSnapshotSupport() should better handle malfunctioning hdfs snapshot

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: snapshot
• Assignee: Unassigned
• Reporter: Ted Yu
• Created: 13/Jul/17 20:43
• Updated: 26/Apr/18 18:01

Description

The following was observed by a customer which prevented master from coming up:

2017-07-13 13:25:07,898 FATAL [xyz:16000.activeMasterManager] master.HMaster: Failed to become active master
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: Daily_Snapshot_Apps_2017-xx
at org.apache.hadoop.fs.Path.initialize(Path.java:205)
at org.apache.hadoop.fs.Path.<init>(Path.java:171)
at org.apache.hadoop.fs.Path.<init>(Path.java:93)
at org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:230)
at org.apache.hadoop.hdfs.protocol.HdfsFileStatus.makeQualified(HdfsFileStatus.java:263)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:911)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1534)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1574)
at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.getCompletedSnapshots(SnapshotManager.java:206)
at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.checkSnapshotSupport(SnapshotManager.java:1011)
at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.initialize(SnapshotManager.java:1070)
at org.apache.hadoop.hbase.procedure.MasterProcedureManagerHost.initialize(MasterProcedureManagerHost.java:50)
at org.apache.hadoop.hbase.master.HMaster.initializeZKBasedSystemTrackers(HMaster.java:667)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:732)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:213)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1863)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: Daily_Snapshot_Apps_2017-xx
at java.net.URI.checkPath(URI.java:1823)
at java.net.URI.<init>(URI.java:745)
at org.apache.hadoop.fs.Path.initialize(Path.java:202)

Turns out the exception can be reproduced using hdfs command line accessing .snapshot directory.

SnapshotManager#checkSnapshotSupport() should better handle malfunctioning hdfs snapshot so that master starts up.