HBase
  1. HBase
  2. HBASE-8614

Handle possible NullPointerException in FSUtils#create(FileSystem fs, Path path, FsPermission perm, InetSocketAddress[] favoredNodes)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Running test suite on hadoop 2.0, I saw the following test failure:

      testFavoredNodes(org.apache.hadoop.hbase.regionserver.TestRegionFavoredNodes)  Time elapsed: 0.106 sec  <<< ERROR!
      org.apache.hadoop.hbase.exceptions.DroppedSnapshotException: region: table,rrr,1369355298031.1fdb1b446b02b497f0869a08adad7745.
              at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1568)
              at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1429)
              at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1347)
              at org.apache.hadoop.hbase.MiniHBaseCluster.flushcache(MiniHBaseCluster.java:531)
              at org.apache.hadoop.hbase.HBaseTestingUtility.flush(HBaseTestingUtility.java:961)
              at org.apache.hadoop.hbase.regionserver.TestRegionFavoredNodes.testFavoredNodes(TestRegionFavoredNodes.java:132)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
              at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
              at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
              at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
              at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
              at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
              at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
              at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
              at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
              at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
              at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
              at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
              at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
              at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
              at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
              at org.junit.runners.Suite.runChild(Suite.java:127)
              at org.junit.runners.Suite.runChild(Suite.java:26)
              at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
              at java.util.concurrent.FutureTask.run(FutureTask.java:138)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:662)
      Caused by: java.io.IOException: java.lang.NullPointerException
              at org.apache.hadoop.hbase.util.FSUtils.create(FSUtils.java:293)
              at org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.createOutputStream(AbstractHFileWriter.java:268)
              at org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:427)
              at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.<init>(StoreFile.java:791)
              at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.<init>(StoreFile.java:733)
              at org.apache.hadoop.hbase.regionserver.StoreFile$WriterBuilder.build(StoreFile.java:671)
              at org.apache.hadoop.hbase.regionserver.HStore.createWriterInTmp(HStore.java:799)
              at org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
              at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:704)
              at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:1813)
              at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1543)
              ... 33 more
      Caused by: java.lang.NullPointerException
              at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1306)
              at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:283)
      

      DistributedFileSystem#create() which supports favoredNodes parameter threw NullPointerException.
      We should handle the NullPointerException and fall back to conventional DistributedFileSystem#create()

      1. 8614-v2.txt
        1 kB
        Ted Yu
      2. 8614-v1.txt
        0.7 kB
        Ted Yu

        Issue Links

          Activity

          Hide
          Matteo Bertozzi added a comment -
          +} catch (NullPointerException npe) {
          +  LOG.debug("Ignoring (most likely Reflection related exception) " + npe);       
          

          does this means that if the create() overload doesn't exists, we don't create the file and we fail silently?

          Show
          Matteo Bertozzi added a comment - +} catch (NullPointerException npe) { + LOG.debug( "Ignoring (most likely Reflection related exception) " + npe); does this means that if the create() overload doesn't exists, we don't create the file and we fail silently?
          Hide
          Ted Yu added a comment -

          if the create() overload doesn't exists, we don't create the file and we fail silently?

          No.
          We fall back to this call:

              return create(fs, path, perm, true);
          
          Show
          Ted Yu added a comment - if the create() overload doesn't exists, we don't create the file and we fail silently? No. We fall back to this call: return create(fs, path, perm, true );
          Hide
          Himanshu Vashishtha added a comment -

          Hmm, isn't this the IOException which is being thrown at https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java#L293?
          ITE is thrown in case underlying method throws an exception, and we re-throw an IOE from there.

          Show
          Himanshu Vashishtha added a comment - Hmm, isn't this the IOException which is being thrown at https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java#L293? ITE is thrown in case underlying method throws an exception, and we re-throw an IOE from there.
          Hide
          Ted Yu added a comment -

          No InvocationTargetException was thrown - FSUtils#create(FileSystem fs, Path path, FsPermission perm, InetSocketAddress[] favoredNodes) exists in hadoop 2.0 but its execution resulted in NullPointerException.

          Show
          Ted Yu added a comment - No InvocationTargetException was thrown - FSUtils#create(FileSystem fs, Path path, FsPermission perm, InetSocketAddress[] favoredNodes) exists in hadoop 2.0 but its execution resulted in NullPointerException.
          Hide
          Himanshu Vashishtha added a comment -

          But the stack trace you posted has

          Caused by: java.io.IOException: java.lang.NullPointerException
                  at org.apache.hadoop.hbase.util.FSUtils.create(FSUtils.java:293)
                  at org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.createOutputStream(AbstractHFileWriter.java:268)
          
          
          Show
          Himanshu Vashishtha added a comment - But the stack trace you posted has Caused by: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hbase.util.FSUtils.create(FSUtils.java:293) at org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.createOutputStream(AbstractHFileWriter.java:268)
          Hide
          Ted Yu added a comment -

          NullPointerException occurred in DFSClient.java and surfaced through FSUtils#create().

          Show
          Ted Yu added a comment - NullPointerException occurred in DFSClient.java and surfaced through FSUtils#create().
          Hide
          Himanshu Vashishtha added a comment -

          Yes, and ITE is thrown in case underlying method which is being invoked throws some exception (in this case NPE). So, I don't think catching NPE like this helps here. You could look getCause() and do a check, for example.

          Show
          Himanshu Vashishtha added a comment - Yes, and ITE is thrown in case underlying method which is being invoked throws some exception (in this case NPE). So, I don't think catching NPE like this helps here. You could look getCause() and do a check, for example.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12584718/8614-v1.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12584718/8614-v1.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5820//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          Patch v2 addresses Himanshu's comment

          Show
          Ted Yu added a comment - Patch v2 addresses Himanshu's comment
          Hide
          Himanshu Vashishtha added a comment -

          Looks good; but why it throws a NPE? This sounds like a bug in the implementation at HDFS level.

          Show
          Himanshu Vashishtha added a comment - Looks good; but why it throws a NPE? This sounds like a bug in the implementation at HDFS level.
          Hide
          Ted Yu added a comment -

          This sounds like a bug in the implementation at HDFS level.

          Agreed.

          Will look into DFSClient and file HDFS JIRA accordingly.

          Show
          Ted Yu added a comment - This sounds like a bug in the implementation at HDFS level. Agreed. Will look into DFSClient and file HDFS JIRA accordingly.
          Hide
          Himanshu Vashishtha added a comment -

          +1. Yeah, handling npe like this looks ugly. You could at least doc it when committing. Thanks.

          Show
          Himanshu Vashishtha added a comment - +1. Yeah, handling npe like this looks ugly. You could at least doc it when committing. Thanks.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12584755/8614-v2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12584755/8614-v2.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5824//console This message is automatically generated.

            People

            • Assignee:
              Ted Yu
              Reporter:
              Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development