Flume
  1. Flume
  2. FLUME-1920

Test case TestFileChannel fails when flume is built from paths containing the string "hadoop"

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: v1.3.0, v1.4.0, v1.3.1, v1.5.0, v1.6.0
    • Fix Version/s: None
    • Component/s: Test
    • Labels:
      None

      Description

      When flume is built from a path that contains the string "hadoop" (ie: /home/user/hadoop_components/flume), the test case TestFileChannel fails.

      The reason it fails is because it can't find the correct location of the hadoop jar file - the function findHadoopJar in TestFileChannel.java
      looks for the hadoop jar path by searching for the string "hadoop" in the property java.class.path, and returns the first match of the search.
      This is not necessarily the correct location of the hadoop jar.

      The right thing to do is to look for the name of the hadoop jar in the classpath (ie: hadoop-core-1.1.1.jar), instead of looking for the string "hadoop".

      1. FLUME-1920-1.patch
        2 kB
        li xiang
      2. FLUME-1920.patch
        2 kB
        Aline Guedes Pinto

        Activity

        Hide
        Mike Percy added a comment -

        Hi Li Xiang,
        Pretty sure the hadoop jar related code is no longer needed here. I removed it locally and the test passes for me. I believe this is because the file channel no longer depends on Hadoop. However I am not able to reproduce the original problem so I am not sure if I am getting a false positive due to something else in my environment perhaps. Can you try removing the hadoop jar from this test completely and see if the tests still pass for you?

        Thanks, Mike

        Show
        Mike Percy added a comment - Hi Li Xiang, Pretty sure the hadoop jar related code is no longer needed here. I removed it locally and the test passes for me. I believe this is because the file channel no longer depends on Hadoop. However I am not able to reproduce the original problem so I am not sure if I am getting a false positive due to something else in my environment perhaps. Can you try removing the hadoop jar from this test completely and see if the tests still pass for you? Thanks, Mike
        li xiang made changes -
        Attachment FLUME-1920-1.patch [ 12650987 ]
        Hide
        li xiang added a comment -

        Update flume-ng-tests/pom.xml according to Hari's comments

        Show
        li xiang added a comment - Update flume-ng-tests/pom.xml according to Hari's comments
        li xiang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s v1.5.0 [ 12324642 ]
        Affects Version/s v1.6.0 [ 12327047 ]
        Hide
        li xiang added a comment -

        I studied Aline's patch and Hari's comments, re-compose flume-ng-tests/pom.xml. Pls review, thanks !

        Show
        li xiang added a comment - I studied Aline's patch and Hari's comments, re-compose flume-ng-tests/pom.xml. Pls review, thanks !
        li xiang made changes -
        Comment [ Hi, Flume 1.6.0-SNAPSHOT also needs this patch ]
        Hide
        li xiang added a comment - - edited

        I found Flume 1.5.0 and 1.6.0-SNAPSHOT needs this patch as well

        Show
        li xiang added a comment - - edited I found Flume 1.5.0 and 1.6.0-SNAPSHOT needs this patch as well
        Mike Percy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Mike Percy added a comment - - edited

        Thanks for the patch Aline! Would be great if you could update the patch per Hari's comments above. Cancelling patch available for the moment

        Show
        Mike Percy added a comment - - edited Thanks for the patch Aline! Would be great if you could update the patch per Hari's comments above. Cancelling patch available for the moment
        Mike Percy made changes -
        Fix Version/s v1.3.0 [ 12322140 ]
        Fix Version/s v1.3.1 [ 12323879 ]
        Hide
        Hari Shreedharan added a comment -

        Aline Guedes Pinto - The patch causes the build to fail with:

        [INFO] Scanning for projects...
        [ERROR] The build could not read 1 project -> [Help 1]
        [ERROR]   
        [ERROR]   The project  (/Users/hshreedharan/work/flume-latest/flume/flume-ng-tests/pom.xml) has 1 error
        [ERROR]     Non-parseable POM /Users/hshreedharan/work/flume-latest/flume/flume-ng-tests/pom.xml: Duplicated tag: 'build' (position: START_TAG seen ...</dependencies>\n\n  <build>... @86:10)  @ line 86, column 10 -> [Help 2]
        [ERROR] 
        [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
        [ERROR] Re-run Maven using the -X switch to enable full debug logging.
        [ERROR] 
        [ERROR] For more information about the errors and possible solutions, please read the following articles:
        [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
        [ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/ModelParseException
        

        You should probably have only one <build> tag in flume-ng-tests/pom.xml (There is one at the end - you should just add the contents of this patch in that one.

        Show
        Hari Shreedharan added a comment - Aline Guedes Pinto - The patch causes the build to fail with: [INFO] Scanning for projects... [ERROR] The build could not read 1 project -> [Help 1] [ERROR] [ERROR] The project (/Users/hshreedharan/work/flume-latest/flume/flume-ng-tests/pom.xml) has 1 error [ERROR] Non-parseable POM /Users/hshreedharan/work/flume-latest/flume/flume-ng-tests/pom.xml: Duplicated tag: 'build' (position: START_TAG seen ...</dependencies>\n\n <build>... @86:10) @ line 86, column 10 -> [Help 2] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch . [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http: //cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException [ERROR] [Help 2] http: //cwiki.apache.org/confluence/display/MAVEN/ModelParseException You should probably have only one <build> tag in flume-ng-tests/pom.xml (There is one at the end - you should just add the contents of this patch in that one.
        Aline Guedes Pinto made changes -
        Attachment FLUME-1920.patch [ 12570149 ]
        Aline Guedes Pinto made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Fix Version/s v1.3.1 [ 12323879 ]
        Fix Version/s v1.3.0 [ 12322140 ]
        Hide
        Aline Guedes Pinto added a comment -

        Patch to fix test case on 1.3.0 and 1.3.1.

        Show
        Aline Guedes Pinto added a comment - Patch to fix test case on 1.3.0 and 1.3.1.
        Aline Guedes Pinto made changes -
        Description When flume is built from a path that contains the string "hadoop" (ie: /home/user/hadoop_components/flume), the test case TestFileChannel fails.

        The reason it fails is because it can't find the correct location of the hadoop jar file - the function findHadoopJar in TestFileChannel.java
        looks for the hadoop jar path by searching for the string "hadoop" in the property java.class.path, and returns the first match of the search.
        This is not necessarily the correct location of the hadoop jar.

        The right thing to do is to look for the name of the hadoop jar in the classpath (ie: hadoop-core-1.1.1.jar), instead of looking for the string "hadoop".
        Aline Guedes Pinto made changes -
        Field Original Value New Value
        Comment [ When flume is built from a path that contains the string "hadoop" (ie: /home/user/hadoop_components/flume), the test case TestFileChannel fails.

        The reason it fails is because it can't find the correct location of the hadoop jar file - the function findHadoopJar in TestFileChannel.java
        looks for the hadoop jar path by searching for the string "hadoop" in the property java.class.path, and returns the first match of the search.
        This is not necessarily the correct location of the hadoop jar.

        The right thing to do is to look for the name of the hadoop jar in the classpath (ie: hadoop-core-1.1.1.jar), instead of looking for the string "hadoop". ]
        Aline Guedes Pinto created issue -

          People

          • Assignee:
            Aline Guedes Pinto
            Reporter:
            Aline Guedes Pinto
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development