HBase
  1. HBase
  2. HBASE-8944

Document workaround for MiniDFSCluster exceptions when running tests

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.6, 0.94.6.1
    • Fix Version/s: 0.98.0
    • Component/s: None
    • Labels:
      None

      Description

      The HBaseTestingUtility appears to have an unhandled NullPointerException in certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not in certain Linux environments such as CentOS.

      In Java:

      HBaseTestingUtility testingUtility = new HBaseTestingUtility();
      testingUtility.startMiniCluster(1);  // blows up on CentOS Linux but not Mac OSX
      

      Creates this exception:

      ERROR in (create-table) (MiniDFSCluster.java:426)
      Uncaught exception, not in assertion.
      expected: nil
        actual: java.lang.NullPointerException: null
       at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes (MiniDFSCluster.java:426)
          org.apache.hadoop.hdfs.MiniDFSCluster.<init> (MiniDFSCluster.java:284)
          org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster (HBaseTestingUtility.java:444)
          org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:612)
          org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:568)
          org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:555)
      

      I also created a stack overflow ticket here, with more info:

      http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments

      Steps to reproduce:

      1. Download a copy of CentOS
      2. Install Java
      3. Fire up a minicluster as part of the HBaseTestingUtility

      1. 8944.patch
        1 kB
        Andrew Purtell
      2. 8944.patch
        2 kB
        Andrew Purtell

        Activity

        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #617 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/617/)
        HBASE-8944. Document workaround for MiniDFSCluster exceptions when running tests (apurtell: rev 1503429)

        • /hbase/trunk/src/main/docbkx/troubleshooting.xml
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #617 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/617/ ) HBASE-8944 . Document workaround for MiniDFSCluster exceptions when running tests (apurtell: rev 1503429) /hbase/trunk/src/main/docbkx/troubleshooting.xml
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #4255 (See https://builds.apache.org/job/HBase-TRUNK/4255/)
        HBASE-8944. Document workaround for MiniDFSCluster exceptions when running tests (apurtell: rev 1503429)

        • /hbase/trunk/src/main/docbkx/troubleshooting.xml
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4255 (See https://builds.apache.org/job/HBase-TRUNK/4255/ ) HBASE-8944 . Document workaround for MiniDFSCluster exceptions when running tests (apurtell: rev 1503429) /hbase/trunk/src/main/docbkx/troubleshooting.xml
        Hide
        Andrew Purtell added a comment -

        What I committed to trunk. Adds a new section to troubleshooting section for unit and integration tests. Eyeballed output html locally with Firefox.

        Show
        Andrew Purtell added a comment - What I committed to trunk. Adds a new section to troubleshooting section for unit and integration tests. Eyeballed output html locally with Firefox.
        Hide
        stack added a comment -

        +1 on patch Andrew Purtell

        Show
        stack added a comment - +1 on patch Andrew Purtell
        Hide
        David Williams added a comment -

        Hi Andrew, yes, absolutely. And thanks for taking the time to check this out.

        Show
        David Williams added a comment - Hi Andrew, yes, absolutely. And thanks for taking the time to check this out.
        Hide
        Andrew Purtell added a comment -

        Thanks for taking the time to check David Williams. As this is HDFS-2556, would you consider the attached update to our troubleshooting manual section sufficient here?

        Show
        Andrew Purtell added a comment - Thanks for taking the time to check David Williams . As this is HDFS-2556 , would you consider the attached update to our troubleshooting manual section sufficient here?
        Hide
        David Williams added a comment -

        Also for all version combinations above.

        Show
        David Williams added a comment - Also for all version combinations above.
        Hide
        David Williams added a comment -

        Update, based on Andrew Purtell's input, running

        $ umask 022
        

        Fixes this issue on Linux

        Show
        David Williams added a comment - Update, based on Andrew Purtell's input, running $ umask 022 Fixes this issue on Linux
        Hide
        David Williams added a comment -

        Great, sounds good. I testes 0.94.6.1 against Hadoop 1.1.2 and 1.0.4. Using 1.1.2 the tests pass on Mac OSX but not CentOS. Using 1.0.4 the tests fail as well on Mac OSX with this error:

        ERROR in (create-table) (MiniHBaseCluster.java:224)
        Uncaught exception, not in assertion.
        expected: nil
          actual: java.io.IOException: Shutting down
         at org.apache.hadoop.hbase.MiniHBaseCluster.init (MiniHBaseCluster.java:224)
            org.apache.hadoop.hbase.MiniHBaseCluster.<init> (MiniHBaseCluster.java:87)
            org.apache.hadoop.hbase.MiniHBaseCluster.<init> (MiniHBaseCluster.java:78)
            org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster (HBaseTestingUtility.java:657)
            org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:627)
            org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:575)
            org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:562)
            hbase.table_test$test_config.doInvoke (table_test.clj:10)
            clojure.lang.RestFn.invoke (RestFn.java:397)
            hbase.table_test/fn (table_test.clj:19)
            clojure.test$test_var$fn__7145.invoke (test.clj:701)
        

        So so far the results look something like this:

        hbase version hadoop version Mac OSX CentOS
        0.94.9 1.2.0 pass fail
        0.94.9 1.1.2 pass fail
        0.94.6.1 1.2.0 pass fail
        0.94.6.1 1.1.2 pass fail
        0.94.6.1 1.0.4 fail fail
        Show
        David Williams added a comment - Great, sounds good. I testes 0.94.6.1 against Hadoop 1.1.2 and 1.0.4. Using 1.1.2 the tests pass on Mac OSX but not CentOS. Using 1.0.4 the tests fail as well on Mac OSX with this error: ERROR in (create-table) (MiniHBaseCluster.java:224) Uncaught exception, not in assertion. expected: nil actual: java.io.IOException: Shutting down at org.apache.hadoop.hbase.MiniHBaseCluster.init (MiniHBaseCluster.java:224) org.apache.hadoop.hbase.MiniHBaseCluster.<init> (MiniHBaseCluster.java:87) org.apache.hadoop.hbase.MiniHBaseCluster.<init> (MiniHBaseCluster.java:78) org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster (HBaseTestingUtility.java:657) org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:627) org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:575) org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:562) hbase.table_test$test_config.doInvoke (table_test.clj:10) clojure.lang.RestFn.invoke (RestFn.java:397) hbase.table_test/fn (table_test.clj:19) clojure.test$test_var$fn__7145.invoke (test.clj:701) So so far the results look something like this: hbase version hadoop version Mac OSX CentOS 0.94.9 1.2.0 pass fail 0.94.9 1.1.2 pass fail 0.94.6.1 1.2.0 pass fail 0.94.6.1 1.1.2 pass fail 0.94.6.1 1.0.4 fail fail
        Hide
        Andrew Purtell added a comment -

        Isn't this HDFS-2556? Try umask 022 before running tests to confirm.

        Show
        Andrew Purtell added a comment - Isn't this HDFS-2556 ? Try umask 022 before running tests to confirm.
        Hide
        stack added a comment -

        David Williams So a 0.94.6.1 against an hadoop 1.2? If so, you are into new territory so you very well may have uncovered an incompatibility. Lets figure it. Thanks.

        Show
        stack added a comment - David Williams So a 0.94.6.1 against an hadoop 1.2? If so, you are into new territory so you very well may have uncovered an incompatibility. Lets figure it. Thanks.
        Hide
        David Williams added a comment -

        Hi Stack, I'm checking out the source tree now. The sources aren't included on Maven Central. Should know more when its down checking out. I am going to need to identify which branch the 1.2.0 build is based on.

        Show
        David Williams added a comment - Hi Stack, I'm checking out the source tree now. The sources aren't included on Maven Central. Should know more when its down checking out. I am going to need to identify which branch the 1.2.0 build is based on.
        Hide
        stack added a comment -

        David Williams What hdfs/hadoop are you running? Looking at the hdfs src, cut and paste what is at #426 in org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes if you don't mind.

        Show
        stack added a comment - David Williams What hdfs/hadoop are you running? Looking at the hdfs src, cut and paste what is at #426 in org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes if you don't mind.

          People

          • Assignee:
            Andrew Purtell
            Reporter:
            David Williams
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development