Bigtop
  1. Bigtop
  2. BIGTOP-483

Smoke test of Hadoop fails with clitest missing testConf.xml file

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.6.0
    • Component/s: Tests
    • Labels:
      None
    • Environment:

      Found on ubuntu, but applies to all systems.

      Description

      12/03/21 23:10:05 INFO cli.TestCLI: File: /home/ubuntu/bigtop/bigtop-tests/test-execution/smokes/hadoop/target/clitest_data/testConf.xml not found
      Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.923 sec <<< FAILURE!

      Results :

      Failed tests:
      testAll(org.apache.bigtop.itest.hadooptests.TestTestCLI)

      Tests run: 10, Failures: 1, Errors: 0, Skipped: 0

      More details:

      -------------------------------------------------------------------------------
      Test set: org.apache.bigtop.itest.hadooptests.TestTestCLI
      -------------------------------------------------------------------------------
      Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.923 sec <<< FAILURE!
      testAll(org.apache.bigtop.itest.hadooptests.TestTestCLI) Time elapsed: 0.096 sec <<< FAILURE!
      junit.framework.AssertionFailedError: Error reading test config file
      at junit.framework.Assert.fail(Assert.java:47)
      at junit.framework.Assert.assertTrue(Assert.java:20)
      at org.apache.hadoop.cli.TestCLI.readTestConfigFile(TestCLI.java:105)
      at org.apache.hadoop.cli.TestCLI.setUp(TestCLI.java:114)

        Activity

        Hide
        David Liu added a comment -

        Also seen by others. Comments from Lei Zou:

        Do a find on testConf.xml under ./bigtop, then copy it to the target place, you can get pass this poiny

        ~/bigtop$ cp ./build/hadoop/deb/hadoop-1.0.1/src/test/org/apache/hadoop/cli/testConf.xml /home/ubuntu/bigtop/bigtop-tests/test-execution/smokes/hadoop/target/clitest_data/testConf.xml

        Show
        David Liu added a comment - Also seen by others. Comments from Lei Zou: Do a find on testConf.xml under ./bigtop, then copy it to the target place, you can get pass this poiny ~/bigtop$ cp ./build/hadoop/deb/hadoop-1.0.1/src/test/org/apache/hadoop/cli/testConf.xml /home/ubuntu/bigtop/bigtop-tests/test-execution/smokes/hadoop/target/clitest_data/testConf.xml
        Hide
        David Liu added a comment -

        Adding the missing file from /build/hadoop/deb/hadoop-1.0.1/src/test/org/apache/hadoop/cli/testConf.xml

        Show
        David Liu added a comment - Adding the missing file from /build/hadoop/deb/hadoop-1.0.1/src/test/org/apache/hadoop/cli/testConf.xml
        Hide
        David Liu added a comment -

        Please review the patch attached.

        Show
        David Liu added a comment - Please review the patch attached.
        Hide
        Roman Shaposhnik added a comment -

        David, sorry to mention it before, but is there any chance you can try to take a look at how to reuse the original artifact from the Hadoop itself? At this point, lets not worry about Hadoop 1.X code line and focus more on the Hadoop 0.23 (AKA Hadoop 2.X) codeline.

        Here's what I have in mind:
        https://github.com/apache/hadoop-common/blob/branch-0.22/common/src/test/core/org/apache/hadoop/cli/CLITestHelper.java#L85

        Note however that this is Hadoop branch-0.22. It would be very nice if you could port this to the Hadoop upstream branch-2

        I'd be willing to help.

        Show
        Roman Shaposhnik added a comment - David, sorry to mention it before, but is there any chance you can try to take a look at how to reuse the original artifact from the Hadoop itself? At this point, lets not worry about Hadoop 1.X code line and focus more on the Hadoop 0.23 (AKA Hadoop 2.X) codeline. Here's what I have in mind: https://github.com/apache/hadoop-common/blob/branch-0.22/common/src/test/core/org/apache/hadoop/cli/CLITestHelper.java#L85 Note however that this is Hadoop branch-0.22. It would be very nice if you could port this to the Hadoop upstream branch-2 I'd be willing to help.
        Hide
        doug chang added a comment -

        Hey David, how about I work on this with you? Find me at the sqoop meetup today and we can chat. This is a good learning experience, using system properties to get rid of the equivalent of #defines in Java. What ever they are called, static final constants.

        dc

        Show
        doug chang added a comment - Hey David, how about I work on this with you? Find me at the sqoop meetup today and we can chat. This is a good learning experience, using system properties to get rid of the equivalent of #defines in Java. What ever they are called, static final constants. dc
        Hide
        Konstantin Boudnik added a comment -

        There's that file in the source tree
        bigtop-tests/test-artifacts/hadoop/src/main/resources/testConfCluster.xml
        Perhaps this is something you were looking for?

        Show
        Konstantin Boudnik added a comment - There's that file in the source tree bigtop-tests/test-artifacts/hadoop/src/main/resources/testConfCluster.xml Perhaps this is something you were looking for?
        Hide
        Konstantin Boudnik added a comment -

        Also, the question is: why all of a sudden that test doesn't pass anymore? It was there since the beginning and our development platform was Ubuntu. That makes one think...

        Show
        Konstantin Boudnik added a comment - Also, the question is: why all of a sudden that test doesn't pass anymore? It was there since the beginning and our development platform was Ubuntu. That makes one think...
        Hide
        David Liu added a comment -

        We can change the code to use the testConfCluster.xml which is almost same as testConf.xml.
        But I cannot figure out the lineage of of these similar files. Does this file originate from Hadoop project, or it is specific to bigtop? Also what's the lineage of these paths?
        https://github.com/apache/hadoop-common/blob/branch-0.22/common/src/test/core/org/apache/hadoop/cli/TestCLI.java
        https://github.com/apache/hadoop/blob/trunk/src/test/core/org/apache/hadoop/cli/TestCLI.java
        They are quite different.

        Show
        David Liu added a comment - We can change the code to use the testConfCluster.xml which is almost same as testConf.xml. But I cannot figure out the lineage of of these similar files. Does this file originate from Hadoop project, or it is specific to bigtop? Also what's the lineage of these paths? https://github.com/apache/hadoop-common/blob/branch-0.22/common/src/test/core/org/apache/hadoop/cli/TestCLI.java https://github.com/apache/hadoop/blob/trunk/src/test/core/org/apache/hadoop/cli/TestCLI.java They are quite different.
        Hide
        Konstantin Boudnik added a comment - - edited

        I think the second URL is a very old repo. The last update there came in back in Jun 18, 2009

        Here's the corrent trunk code for this file

        Show
        Konstantin Boudnik added a comment - - edited I think the second URL is a very old repo. The last update there came in back in Jun 18, 2009 Here's the corrent trunk code for this file
        Hide
        David Liu added a comment -

        Thanks. So seems we need to
        a. use https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/hadoop-common/src/test/resources/testConf.xml, and
        b. put it where the testConfCluster.xml is, and
        c. remove any reference to testConfCluster.xml (as they duplicates), and
        d. update the TestCLI.java and other source files to match the latest hadoop ones.

        Does this sound like a reasonable plan?

        Show
        David Liu added a comment - Thanks. So seems we need to a. use https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/hadoop-common/src/test/resources/testConf.xml , and b. put it where the testConfCluster.xml is, and c. remove any reference to testConfCluster.xml (as they duplicates), and d. update the TestCLI.java and other source files to match the latest hadoop ones. Does this sound like a reasonable plan?
        Hide
        Roman Shaposhnik added a comment -

        David, first of all thanks for being persistent on this issue. Some of this dates back to the most ancient time of iTest ever so please don't take it as a final truth written in stone. Essentially, the fact that we've always had it that way doesn't mean we shouldn't change it.

        That said, the issue really boils down to whether we:

        1. try to essentially fork the testConf.xml from upstream and maintain it in Bigtop
        2. try to work with upstream to make testConf.xml not only applicable to the true unit tests, but also in a more generic system test context (what Bigtop does)

        Of course number 2 has an advantage of HDFS developers maintaining it for us according to whatever changes they make to the source code of HDFS, but we have to make sure that there's nothing in there that is miniDFS specific (IOW hard coded assumptions that are going to break on a real cluster).

        Show
        Roman Shaposhnik added a comment - David, first of all thanks for being persistent on this issue. Some of this dates back to the most ancient time of iTest ever so please don't take it as a final truth written in stone. Essentially, the fact that we've always had it that way doesn't mean we shouldn't change it. That said, the issue really boils down to whether we: try to essentially fork the testConf.xml from upstream and maintain it in Bigtop try to work with upstream to make testConf.xml not only applicable to the true unit tests, but also in a more generic system test context (what Bigtop does) Of course number 2 has an advantage of HDFS developers maintaining it for us according to whatever changes they make to the source code of HDFS, but we have to make sure that there's nothing in there that is miniDFS specific (IOW hard coded assumptions that are going to break on a real cluster).
        Hide
        David Liu added a comment -

        Digging further, it seems the real issue is iTest's TestCLI.groovy tries to use System.properties.setProperty() to change the xml file name to testConfCluster.xml, but that doesn't work, as the base class is using bigtop/build/hadoop/deb/hadoop-1.0.1/src/test/org/apache/hadoop/cli/TestCLI.java which has
        static String testConfigFile = "testConf.xml";
        This code as is should work well if tests are from https://github.com/apache/hadoop-common/blob/branch-0.22/common/src/test/core/org/apache/hadoop/cli/CLITestHelper.java#L85

        But I am not clear, how to attach branch-0.22's test code to bigtop, or whether that is possible.

        I am not versed in the history of the test setup, and too new to comment on the right approach. But I sure hope bigtop has its own set of minimal set of testing for each sub project that has no upstream dependencies just to verify the basic health of the build and setup.

        Show
        David Liu added a comment - Digging further, it seems the real issue is iTest's TestCLI.groovy tries to use System.properties.setProperty() to change the xml file name to testConfCluster.xml, but that doesn't work, as the base class is using bigtop/build/hadoop/deb/hadoop-1.0.1/src/test/org/apache/hadoop/cli/TestCLI.java which has static String testConfigFile = "testConf.xml"; This code as is should work well if tests are from https://github.com/apache/hadoop-common/blob/branch-0.22/common/src/test/core/org/apache/hadoop/cli/CLITestHelper.java#L85 But I am not clear, how to attach branch-0.22's test code to bigtop, or whether that is possible. I am not versed in the history of the test setup, and too new to comment on the right approach. But I sure hope bigtop has its own set of minimal set of testing for each sub project that has no upstream dependencies just to verify the basic health of the build and setup.
        Hide
        Roman Shaposhnik added a comment -

        David, would you be interested in working with upstream Hadoop on the following JIRA: HADOOP-7730
        Basically, at this point it would be very nice if you could look at the patches that are attached to that JIRA and create yet another patch that would apply to Hadoop's trunk (in its current form).

        Does this sound like something you might want to give a try?

        Show
        Roman Shaposhnik added a comment - David, would you be interested in working with upstream Hadoop on the following JIRA: HADOOP-7730 Basically, at this point it would be very nice if you could look at the patches that are attached to that JIRA and create yet another patch that would apply to Hadoop's trunk (in its current form). Does this sound like something you might want to give a try?
        Hide
        Roman Shaposhnik added a comment -

        Canceling a Bigtop patch since it appears that the consensus is to make it happen upstream.

        Show
        Roman Shaposhnik added a comment - Canceling a Bigtop patch since it appears that the consensus is to make it happen upstream.
        Hide
        Roman Shaposhnik added a comment -

        DUP of BIGTOP-895

        Show
        Roman Shaposhnik added a comment - DUP of BIGTOP-895

          People

          • Assignee:
            Roman Shaposhnik
            Reporter:
            David Liu
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development