Flume
  1. Flume
  2. FLUME-1920

Test case TestFileChannel fails when flume is built from paths containing the string "hadoop"

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: v1.3.0, v1.4.0, v1.3.1, v1.5.0, v1.6.0
    • Fix Version/s: v1.6.0
    • Component/s: Test
    • Labels:
      None

      Description

      When flume is built from a path that contains the string "hadoop" (ie: /home/user/hadoop_components/flume), the test case TestFileChannel fails.

      The reason it fails is because it can't find the correct location of the hadoop jar file - the function findHadoopJar in TestFileChannel.java
      looks for the hadoop jar path by searching for the string "hadoop" in the property java.class.path, and returns the first match of the search.
      This is not necessarily the correct location of the hadoop jar.

      The right thing to do is to look for the name of the hadoop jar in the classpath (ie: hadoop-core-1.1.1.jar), instead of looking for the string "hadoop".

      1. FLUME-1920-2.patch
        4 kB
        li xiang
      2. FLUME-1920-1.patch
        2 kB
        li xiang
      3. FLUME-1920.patch
        2 kB
        Aline Guedes Pinto

        Issue Links

          Activity

          Hide
          Aline Guedes Pinto added a comment -

          Patch to fix test case on 1.3.0 and 1.3.1.

          Show
          Aline Guedes Pinto added a comment - Patch to fix test case on 1.3.0 and 1.3.1.
          Hide
          Hari Shreedharan added a comment -

          Aline Guedes Pinto - The patch causes the build to fail with:

          [INFO] Scanning for projects...
          [ERROR] The build could not read 1 project -> [Help 1]
          [ERROR]   
          [ERROR]   The project  (/Users/hshreedharan/work/flume-latest/flume/flume-ng-tests/pom.xml) has 1 error
          [ERROR]     Non-parseable POM /Users/hshreedharan/work/flume-latest/flume/flume-ng-tests/pom.xml: Duplicated tag: 'build' (position: START_TAG seen ...</dependencies>\n\n  <build>... @86:10)  @ line 86, column 10 -> [Help 2]
          [ERROR] 
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR] 
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
          [ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/ModelParseException
          

          You should probably have only one <build> tag in flume-ng-tests/pom.xml (There is one at the end - you should just add the contents of this patch in that one.

          Show
          Hari Shreedharan added a comment - Aline Guedes Pinto - The patch causes the build to fail with: [INFO] Scanning for projects... [ERROR] The build could not read 1 project -> [Help 1] [ERROR] [ERROR] The project (/Users/hshreedharan/work/flume-latest/flume/flume-ng-tests/pom.xml) has 1 error [ERROR] Non-parseable POM /Users/hshreedharan/work/flume-latest/flume/flume-ng-tests/pom.xml: Duplicated tag: 'build' (position: START_TAG seen ...</dependencies>\n\n <build>... @86:10) @ line 86, column 10 -> [Help 2] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch . [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http: //cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException [ERROR] [Help 2] http: //cwiki.apache.org/confluence/display/MAVEN/ModelParseException You should probably have only one <build> tag in flume-ng-tests/pom.xml (There is one at the end - you should just add the contents of this patch in that one.
          Hide
          Mike Percy added a comment - - edited

          Thanks for the patch Aline! Would be great if you could update the patch per Hari's comments above. Cancelling patch available for the moment

          Show
          Mike Percy added a comment - - edited Thanks for the patch Aline! Would be great if you could update the patch per Hari's comments above. Cancelling patch available for the moment
          Hide
          li xiang added a comment - - edited

          I found Flume 1.5.0 and 1.6.0-SNAPSHOT needs this patch as well

          Show
          li xiang added a comment - - edited I found Flume 1.5.0 and 1.6.0-SNAPSHOT needs this patch as well
          Hide
          li xiang added a comment -

          I studied Aline's patch and Hari's comments, re-compose flume-ng-tests/pom.xml. Pls review, thanks !

          Show
          li xiang added a comment - I studied Aline's patch and Hari's comments, re-compose flume-ng-tests/pom.xml. Pls review, thanks !
          Hide
          li xiang added a comment -

          Update flume-ng-tests/pom.xml according to Hari's comments

          Show
          li xiang added a comment - Update flume-ng-tests/pom.xml according to Hari's comments
          Hide
          Mike Percy added a comment -

          Hi Li Xiang,
          Pretty sure the hadoop jar related code is no longer needed here. I removed it locally and the test passes for me. I believe this is because the file channel no longer depends on Hadoop. However I am not able to reproduce the original problem so I am not sure if I am getting a false positive due to something else in my environment perhaps. Can you try removing the hadoop jar from this test completely and see if the tests still pass for you?

          Thanks, Mike

          Show
          Mike Percy added a comment - Hi Li Xiang, Pretty sure the hadoop jar related code is no longer needed here. I removed it locally and the test passes for me. I believe this is because the file channel no longer depends on Hadoop. However I am not able to reproduce the original problem so I am not sure if I am getting a false positive due to something else in my environment perhaps. Can you try removing the hadoop jar from this test completely and see if the tests still pass for you? Thanks, Mike
          Hide
          li xiang added a comment -

          Hi Mike, sorry for my late response.

          1. I failed to run this test case only: flume\flume-ng-tests\src\test\java\org\apache\flume\test\agent\TestFileChannel.java.
          Firstly I tried to modify the pom.xml to add "<include> **/TestFileChannel.java</include>" but other TestFileChannel.java in flume-ng-channel\flume-file-channel is tested, not the one in flume-ng-tests as expected.
          Also tried with "mvn test -Dtest=org.apache.flume.test.agent.TestFileChannel -DfailIfNoTests=false", to specify the package name, but no test is performed.

          How can I run this test case solo ?

          2. What do you mean by "removing the hadoop jar from this test" ?
          Remove the file hadoop.xxx.jar or remove haoop-jar related code ? The code in StagedInstall.java is to add "--classpath xxxx" when starting flume, xxxx is a directory in java class path which has hadoop jar. I can not understand the logic here, why a path containing hadoop jar is needed for flume to start.
          Really appreciate if you have any comments

          Show
          li xiang added a comment - Hi Mike, sorry for my late response. 1. I failed to run this test case only: flume\flume-ng-tests\src\test\java\org\apache\flume\test\agent\TestFileChannel.java. Firstly I tried to modify the pom.xml to add "<include> **/TestFileChannel.java</include>" but other TestFileChannel.java in flume-ng-channel\flume-file-channel is tested, not the one in flume-ng-tests as expected. Also tried with "mvn test -Dtest=org.apache.flume.test.agent.TestFileChannel -DfailIfNoTests=false", to specify the package name, but no test is performed. How can I run this test case solo ? 2. What do you mean by "removing the hadoop jar from this test" ? Remove the file hadoop.xxx.jar or remove haoop-jar related code ? The code in StagedInstall.java is to add "--classpath xxxx" when starting flume, xxxx is a directory in java class path which has hadoop jar. I can not understand the logic here, why a path containing hadoop jar is needed for flume to start. Really appreciate if you have any comments
          Hide
          li xiang added a comment - - edited

          Some analysis:

          1. In flume-ng-tests/src/test/java/org/apache/flume/test/agent/TestFileChannel.java
          findHadoopJar() is to locate the path of hadoop jar. Aline's fix is to correct an error in this function
          This path is used by class StagedInstall, to add "--classpath hadoop_jar_path" when run "flume-ng"

          2. In flume-ng-tests/src/test/java/org/apache/flume/test/util/StagedInstall.java
          A private member "agentClasspath" is set by setAgentClasspath().
          And then in startAgent(), if agentClasspath is not null, "--classpath hadoop_jar_path" will be appended when run "flume-ng"

          Show
          li xiang added a comment - - edited Some analysis: 1. In flume-ng-tests/src/test/java/org/apache/flume/test/agent/TestFileChannel.java findHadoopJar() is to locate the path of hadoop jar. Aline's fix is to correct an error in this function This path is used by class StagedInstall, to add "--classpath hadoop_jar_path" when run "flume-ng" 2. In flume-ng-tests/src/test/java/org/apache/flume/test/util/StagedInstall.java A private member "agentClasspath" is set by setAgentClasspath(). And then in startAgent(), if agentClasspath is not null, "--classpath hadoop_jar_path" will be appended when run "flume-ng"
          Hide
          li xiang added a comment -

          According to the comments left by the originator in flume-ng-tests/src/test/java/org/apache/flume/test/agent/TestFileChannel.java
          for findHadoopJar():

          • We search the class path of the current JVM process to grab the same
          • hadoop jar that's depended on by the file channel.

          It seems at that time, file channel depended on hadoop jar.

          Show
          li xiang added a comment - According to the comments left by the originator in flume-ng-tests/src/test/java/org/apache/flume/test/agent/TestFileChannel.java for findHadoopJar(): We search the class path of the current JVM process to grab the same hadoop jar that's depended on by the file channel. It seems at that time, file channel depended on hadoop jar.
          Hide
          Mike Percy added a comment -

          Yes, the dependency on Hadoop was removed from file channel.

          Show
          Mike Percy added a comment - Yes, the dependency on Hadoop was removed from file channel.
          Hide
          li xiang added a comment -

          Hi Mike, I found a way to run TestFileChannel: after entering flume-ng-tests folder, I issued "mvn package". Sorry that I am not good at maven.
          I failed to trigger the test by "mvn test" but "mvn package", is this because "<phase>package</phase>" is specified within <build> and </build> in the pom.xml ?

          the test case failed afterwards:
          Running org.apache.flume.test.agent.TestFileChannel
          Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.307 sec <<< FAILURE!
          testInOut(org.apache.flume.test.agent.TestFileChannel) Time elapsed: 0.085 sec <<< ERROR!
          java.lang.Exception: Failed to locate tar-ball distribution. Please specify explicitly via system property: flume.dist.tarball
          at org.apache.flume.test.util.StagedInstall.<init>(StagedInstall.java:229)
          at org.apache.flume.test.util.StagedInstall.getInstance(StagedInstall.java:79)
          at org.apache.flume.test.agent.TestFileChannel.testInOut(TestFileChannel.java:130)

          Do you know how to solve it ? Thanks !

          Show
          li xiang added a comment - Hi Mike, I found a way to run TestFileChannel: after entering flume-ng-tests folder, I issued "mvn package". Sorry that I am not good at maven. I failed to trigger the test by "mvn test" but "mvn package", is this because "<phase>package</phase>" is specified within <build> and </build> in the pom.xml ? the test case failed afterwards: Running org.apache.flume.test.agent.TestFileChannel Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.307 sec <<< FAILURE! testInOut(org.apache.flume.test.agent.TestFileChannel) Time elapsed: 0.085 sec <<< ERROR! java.lang.Exception: Failed to locate tar-ball distribution. Please specify explicitly via system property: flume.dist.tarball at org.apache.flume.test.util.StagedInstall.<init>(StagedInstall.java:229) at org.apache.flume.test.util.StagedInstall.getInstance(StagedInstall.java:79) at org.apache.flume.test.agent.TestFileChannel.testInOut(TestFileChannel.java:130) Do you know how to solve it ? Thanks !
          Hide
          Mike Percy added a comment -

          It will run if you do mvn install from the top level. But it takes a long time. Try mvn install -DskipTests from the top level, after that completes try mvn install from the integration test dir. There is a way to use the failsafe mvn plugin to run the integration test but I don't remember the exact command anymore.

          Show
          Mike Percy added a comment - It will run if you do mvn install from the top level. But it takes a long time. Try mvn install -DskipTests from the top level, after that completes try mvn install from the integration test dir. There is a way to use the failsafe mvn plugin to run the integration test but I don't remember the exact command anymore.
          Hide
          li xiang added a comment -

          Remove hadoop jar related code from both TestFileChannel.java and StagedInstall.java

          Show
          li xiang added a comment - Remove hadoop jar related code from both TestFileChannel.java and StagedInstall.java
          Hide
          li xiang added a comment -

          Hi Mike, thanks for the guide 8-).
          I uploaded the new patch "FLUME-1920-2.patch" to remove hadoop jar related code from both TestFileChannel.java and StagedInstall.java.

          The patch is for trunk.

          UT passed on trunk and jdk 1.7. Pls review if you have time on https://reviews.apache.org/r/24089/

          Show
          li xiang added a comment - Hi Mike, thanks for the guide 8-). I uploaded the new patch " FLUME-1920 -2.patch" to remove hadoop jar related code from both TestFileChannel.java and StagedInstall.java. The patch is for trunk. UT passed on trunk and jdk 1.7. Pls review if you have time on https://reviews.apache.org/r/24089/
          Hide
          Mike Percy added a comment -

          +1 looks good. Thanks for your persistence Li Xiang!

          Show
          Mike Percy added a comment - +1 looks good. Thanks for your persistence Li Xiang!
          Hide
          Mike Percy added a comment -

          Committed to trunk and flume-1.6 branches. Thanks for the patch!

          Show
          Mike Percy added a comment - Committed to trunk and flume-1.6 branches. Thanks for the patch!
          Hide
          ASF subversion and git services added a comment -

          Commit 0c5b87a5dfd53bcdcf8e2d8c570735573cd274ad in flume's branch refs/heads/trunk from Mike Percy
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=0c5b87a ]

          FLUME-1920. Remove Hadoop dependency from integration tests

          The file channel no longer actually depends on Hadoop, and this test was
          exhibiting failures when Flume is built from paths containing the string
          "hadoop". Since hadoop is not required for any of these tests, the
          straightforward thing to do was to remove the now unnecessary Hadoop JAR
          detection code completely.

          (Li Xiang via Mike Percy)

          Show
          ASF subversion and git services added a comment - Commit 0c5b87a5dfd53bcdcf8e2d8c570735573cd274ad in flume's branch refs/heads/trunk from Mike Percy [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=0c5b87a ] FLUME-1920 . Remove Hadoop dependency from integration tests The file channel no longer actually depends on Hadoop, and this test was exhibiting failures when Flume is built from paths containing the string "hadoop". Since hadoop is not required for any of these tests, the straightforward thing to do was to remove the now unnecessary Hadoop JAR detection code completely. (Li Xiang via Mike Percy)
          Hide
          ASF subversion and git services added a comment -

          Commit bb380a5f7695054216da94a72db1a3f8769668da in flume's branch refs/heads/flume-1.6 from Mike Percy
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=bb380a5 ]

          FLUME-1920. Remove Hadoop dependency from integration tests

          The file channel no longer actually depends on Hadoop, and this test was
          exhibiting failures when Flume is built from paths containing the string
          "hadoop". Since hadoop is not required for any of these tests, the
          straightforward thing to do was to remove the now unnecessary Hadoop JAR
          detection code completely.

          (Li Xiang via Mike Percy)

          Show
          ASF subversion and git services added a comment - Commit bb380a5f7695054216da94a72db1a3f8769668da in flume's branch refs/heads/flume-1.6 from Mike Percy [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=bb380a5 ] FLUME-1920 . Remove Hadoop dependency from integration tests The file channel no longer actually depends on Hadoop, and this test was exhibiting failures when Flume is built from paths containing the string "hadoop". Since hadoop is not required for any of these tests, the straightforward thing to do was to remove the now unnecessary Hadoop JAR detection code completely. (Li Xiang via Mike Percy)
          Hide
          Hudson added a comment -

          UNSTABLE: Integrated in Flume-trunk-hbase-98 #10 (See https://builds.apache.org/job/Flume-trunk-hbase-98/10/)
          FLUME-1920. Remove Hadoop dependency from integration tests (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=0c5b87a5dfd53bcdcf8e2d8c570735573cd274ad)

          • flume-ng-tests/src/test/java/org/apache/flume/test/util/StagedInstall.java
          • flume-ng-tests/src/test/java/org/apache/flume/test/agent/TestFileChannel.java
          Show
          Hudson added a comment - UNSTABLE: Integrated in Flume-trunk-hbase-98 #10 (See https://builds.apache.org/job/Flume-trunk-hbase-98/10/ ) FLUME-1920 . Remove Hadoop dependency from integration tests (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=0c5b87a5dfd53bcdcf8e2d8c570735573cd274ad ) flume-ng-tests/src/test/java/org/apache/flume/test/util/StagedInstall.java flume-ng-tests/src/test/java/org/apache/flume/test/agent/TestFileChannel.java
          Hide
          Hudson added a comment -

          UNSTABLE: Integrated in flume-trunk #650 (See https://builds.apache.org/job/flume-trunk/650/)
          FLUME-1920. Remove Hadoop dependency from integration tests (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=0c5b87a5dfd53bcdcf8e2d8c570735573cd274ad)

          • flume-ng-tests/src/test/java/org/apache/flume/test/agent/TestFileChannel.java
          • flume-ng-tests/src/test/java/org/apache/flume/test/util/StagedInstall.java
          Show
          Hudson added a comment - UNSTABLE: Integrated in flume-trunk #650 (See https://builds.apache.org/job/flume-trunk/650/ ) FLUME-1920 . Remove Hadoop dependency from integration tests (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=0c5b87a5dfd53bcdcf8e2d8c570735573cd274ad ) flume-ng-tests/src/test/java/org/apache/flume/test/agent/TestFileChannel.java flume-ng-tests/src/test/java/org/apache/flume/test/util/StagedInstall.java

            People

            • Assignee:
              li xiang
              Reporter:
              Aline Guedes Pinto
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development