Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: libhdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The libhdfs unit tests (ant test-c++-libhdfs -Dislibhdfs=1) do not run yet because the scripts are in the common subproject,

      1. log
        15 kB
        Konstantin Boudnik
      2. hdfs-756-2.patch
        3 kB
        Eli Collins
      3. hdfs-756.patch
        7 kB
        Eli Collins

        Issue Links

          Activity

          dhruba borthakur created issue -
          dhruba borthakur made changes -
          Field Original Value New Value
          Link This issue is related to HDFS-712 [ HDFS-712 ]
          Eli Collins made changes -
          Assignee Eli Collins [ eli ]
          dhruba borthakur made changes -
          Link This issue is duplicated by HDFS-769 [ HDFS-769 ]
          Hide
          Konstantin Boudnik added a comment -

          I'd suggest to raise the priority on this, because it makes full build ant test to fail all the time.

          Show
          Konstantin Boudnik added a comment - I'd suggest to raise the priority on this, because it makes full build ant test to fail all the time.
          Hide
          Eli Collins added a comment -

          Agreed, I'll have a patch out soon.

          Show
          Eli Collins added a comment - Agreed, I'll have a patch out soon.
          Eli Collins made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          Hide
          Eli Collins added a comment -

          Patch attached. The project split means the libhdfs test needs to access both the hdfs and common respos. The test lives in and is run out of the hdfs repo, however it runs an instance of hdfs, and therefore needs access to the common repo's bin directory. test-libhdfs.sh runs the hdfs instance out of the common repo (build/test/libhdfs and sub-directories get created there) since hadoop-daemon.sh makes doing otherwise a pain.

          This means the test now requires setting HADOOP_CORE_HOME. Once HDFS-621 is checked in it would be nice to convert this test to run a MiniDFS cluster and no longer depend on a common repo. It doesn't seem like running one-daemon per process using the traditional startup scripts adds much additional coverage. Reasonable?

          To run the test:
          export HADOOP_CORE_HOME=<common repo dir>
          ant -Dcompile.c++=true -Dlibhdfs=true test-c++-libhdfs

          You may need to ant clean your common directory to remove old hdfs jar files in the root directory, build/ivy or the lib dirs.

          Show
          Eli Collins added a comment - Patch attached. The project split means the libhdfs test needs to access both the hdfs and common respos. The test lives in and is run out of the hdfs repo, however it runs an instance of hdfs, and therefore needs access to the common repo's bin directory. test-libhdfs.sh runs the hdfs instance out of the common repo (build/test/libhdfs and sub-directories get created there) since hadoop-daemon.sh makes doing otherwise a pain. This means the test now requires setting HADOOP_CORE_HOME. Once HDFS-621 is checked in it would be nice to convert this test to run a MiniDFS cluster and no longer depend on a common repo. It doesn't seem like running one-daemon per process using the traditional startup scripts adds much additional coverage. Reasonable? To run the test: export HADOOP_CORE_HOME=<common repo dir> ant -Dcompile.c++=true -Dlibhdfs=true test-c++-libhdfs You may need to ant clean your common directory to remove old hdfs jar files in the root directory, build/ivy or the lib dirs.
          Eli Collins made changes -
          Attachment hdfs-756.patch [ 12425177 ]
          Eli Collins made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Konstantin Boudnik added a comment -

          I don't think a dependency on a Common workspace is the very plausible idea. For once, it will affect Hudson's configurations and will introduce an inter-build dependencies there.

          I believe a better way is to rely on the content of hadoop-core SNAPSHOT file which includes all bin/ scripts in it packed into bin.tgz file. What has to be done is to create a temp directory to unpack said bin.tgz file and then use it for the purpose of the test.

          This approach will allow to make the test self-sufficient and avoid having potentially harmful dependencies.

          Show
          Konstantin Boudnik added a comment - I don't think a dependency on a Common workspace is the very plausible idea. For once, it will affect Hudson's configurations and will introduce an inter-build dependencies there. I believe a better way is to rely on the content of hadoop-core SNAPSHOT file which includes all bin/ scripts in it packed into bin.tgz file. What has to be done is to create a temp directory to unpack said bin.tgz file and then use it for the purpose of the test. This approach will allow to make the test self-sufficient and avoid having potentially harmful dependencies.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12425177/hdfs-756.patch
          against trunk revision 881017.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 13 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/116/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/116/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/116/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/116/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12425177/hdfs-756.patch against trunk revision 881017. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 13 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/116/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/116/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/116/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/116/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          Slightly different modification of the proposal above is to have all needed scripts checked in along with the test.

          Also, I feel kind of uncomfortable with such a 'end-to-end' test to be executed as a part of what is essentially unit/functional validation. Shall we move it to a integration test suite or something?

          Show
          Konstantin Boudnik added a comment - Slightly different modification of the proposal above is to have all needed scripts checked in along with the test. Also, I feel kind of uncomfortable with such a 'end-to-end' test to be executed as a part of what is essentially unit/functional validation. Shall we move it to a integration test suite or something?
          Hide
          Eli Collins added a comment -

          Hey Konstantin – Thanks for the feedback. Using the snapshot jar should work and removes the dependency on a common repo. Since the snapshot jar is updated from maven this approach will still work if you've also got changes in a common repo that should be used when running the test. If people are OK with that I'll update the patch. Per the above though I think this patch should be temporary, what do you think about converting this test to run a MiniDFS cluster (once HDFS-621 is in) and have a separate end-to-end test?

          Show
          Eli Collins added a comment - Hey Konstantin – Thanks for the feedback. Using the snapshot jar should work and removes the dependency on a common repo. Since the snapshot jar is updated from maven this approach will still work if you've also got changes in a common repo that should be used when running the test. If people are OK with that I'll update the patch. Per the above though I think this patch should be temporary, what do you think about converting this test to run a MiniDFS cluster (once HDFS-621 is in) and have a separate end-to-end test?
          Hide
          Eli Collins added a comment -

          I'd prefer using the snapshot jar instead of duplicating the scripts, we're going to require the snapshot jar anyway to execute.

          Show
          Eli Collins added a comment - I'd prefer using the snapshot jar instead of duplicating the scripts, we're going to require the snapshot jar anyway to execute.
          Hide
          dhruba borthakur added a comment -

          Let's use the snapshot jar for now.

          Show
          dhruba borthakur added a comment - Let's use the snapshot jar for now.
          dhruba borthakur made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Konstantin Boudnik added a comment -

          I think this patch should be temporary, what do you think about converting this test to run a MiniDFS cluster (once HDFS-621 is in) and have a separate end-to-end test?

          I think this is a very good idea!

          Show
          Konstantin Boudnik added a comment - I think this patch should be temporary, what do you think about converting this test to run a MiniDFS cluster (once HDFS-621 is in) and have a separate end-to-end test? I think this is a very good idea!
          Hide
          Eli Collins added a comment -

          Here's an updated patch that creates the bin directory from the core jar. The config scripts in the bin directory assume the bin directory is named "bin" and that it lives in the hadoop home directory, so that's where the test scripts extracts it. The test script does not clobber a bin directory if it already exists, which I tested by creating an empty bin directory and checking that the tests fail because they can find the scripts to start the daemons. The tests pass after applying the patch from HDFS-727.

          Show
          Eli Collins added a comment - Here's an updated patch that creates the bin directory from the core jar. The config scripts in the bin directory assume the bin directory is named "bin" and that it lives in the hadoop home directory, so that's where the test scripts extracts it. The test script does not clobber a bin directory if it already exists, which I tested by creating an empty bin directory and checking that the tests fail because they can find the scripts to start the daemons. The tests pass after applying the patch from HDFS-727 .
          Eli Collins made changes -
          Attachment hdfs-756-2.patch [ 12425498 ]
          Hide
          Eli Collins added a comment -

          That last part should have read "the tests fails because it can not find the scripts to start the daemons". Sorry for the noise.

          Show
          Eli Collins added a comment - That last part should have read "the tests fails because it can not find the scripts to start the daemons". Sorry for the noise.
          Eli Collins made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12425498/hdfs-756-2.patch
          against trunk revision 881695.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/119/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/119/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/119/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/119/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12425498/hdfs-756-2.patch against trunk revision 881695. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/119/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/119/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/119/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/119/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          The patch seems Ok. However, test is failing on my linux box (see the log file)

          Show
          Konstantin Boudnik added a comment - The patch seems Ok. However, test is failing on my linux box (see the log file)
          Konstantin Boudnik made changes -
          Attachment log [ 12425541 ]
          Hide
          Eli Collins added a comment -

          You need to apply the patch from HDFS-727 first. That's patch is ready to go just need a comitter to commit it.

          Show
          Eli Collins added a comment - You need to apply the patch from HDFS-727 first. That's patch is ready to go just need a comitter to commit it.
          Eli Collins made changes -
          Link This issue is blocked by HDFS-727 [ HDFS-727 ]
          Hide
          Konstantin Boudnik added a comment -

          You need to apply the patch from HDFS-727 first

          yup, my bad. everything seems to be working.

          +1 patch looks good

          Show
          Konstantin Boudnik added a comment - You need to apply the patch from HDFS-727 first yup, my bad. everything seems to be working. +1 patch looks good
          Hide
          Konstantin Boudnik added a comment -

          I've just committed this. Thanks Eli.

          Show
          Konstantin Boudnik added a comment - I've just committed this. Thanks Eli.
          Konstantin Boudnik made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Konstantin Boudnik made changes -
          Affects Version/s 0.22.0 [ 12314241 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #118 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/118/)
          . libhdfs unit tests do not run. (Eli Collins via cos).

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #118 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/118/ ) . libhdfs unit tests do not run. (Eli Collins via cos).
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #121 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/121/)
          . libhdfs unit tests do not run. (Eli Collins via cos).

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #121 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/121/ ) . libhdfs unit tests do not run. (Eli Collins via cos).
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #150 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/150/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #150 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/150/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #81 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/81/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #81 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/81/ )
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12314046 ]
          Fix Version/s 0.22.0 [ 12314241 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Eli Collins
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development