Hadoop Common
  1. Hadoop Common
  2. HADOOP-9489

Eclipse instructions in BUILDING.txt don't work

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.7.0
    • Fix Version/s: None
    • Component/s: build
    • Labels:
      None
    • Target Version/s:

      Description

      I have tried several times to import Hadoop trunk into Eclipse following the instructions in the BUILDING.txt file, but so far have not been able to get it to work.

      If I use a fresh install of Eclipse 4.2.2, Eclipse will complain about an undefined M2_REPO environment variable. I discovered that this is defined automatically by the M2Eclipse plugin, and think that the BUILDING.txt doc should be updated to explain this.

      After installing M2Eclipse I tried importing the code again, and now get over 2500 errors related to missing class dependencies. Many of these errors correspond to missing classes in the oah*.proto namespace, which makes me think that 'mvn eclipse:eclipse' is not triggering protoc.

      1. eclipse_hadoop_errors.txt
        451 kB
        Carl Steinbach
      2. error.log
        15 kB
        Tsuyoshi Ozawa
      3. HADOOP-9489.003.patch
        3 kB
        Chris Nauroth
      4. HADOOP-9489.004.patch
        2 kB
        Chris Nauroth
      5. HADOOP-9489.1.patch
        3 kB
        Chris Nauroth
      6. HADOOP-9489.2.patch
        3 kB
        Chris Nauroth

        Activity

        Carl Steinbach created issue -
        Hide
        Carl Steinbach added a comment -

        Attaching a file containing the list build failures flagged by Eclipse.

        Show
        Carl Steinbach added a comment - Attaching a file containing the list build failures flagged by Eclipse.
        Carl Steinbach made changes -
        Field Original Value New Value
        Attachment eclipse_hadoop_errors.txt [ 12579667 ]
        Chris Nauroth made changes -
        Assignee Chris Nauroth [ cnauroth ]
        Hide
        Chris Nauroth added a comment -

        Hi, Carl Steinbach. I think I understand what's going wrong with fresh imports of the project, and I'm attaching a patch that attempts to fix it. Would you mind trying out this patch in your environment to see if it works for you?

        Here is a summary of what I did:

        1. I changed BUILDING.txt to state that m2eclipse is required, that you should do an mvn install of the whole Hadoop project (not just hadoop-maven-plugins), and that you should import by clicking File -> Import -> Maven -> Existing Maven Projects. Once I did it this way, a fresh import showed ~100 errors.
        2. hadoop-common still had some errors for missing compiled protoc test sources. This was because the pom.xml was not specifying generated-test-sources/java as a source directory. I added a call to build-helper-maven-plugin to set that up correctly. This eliminated ~40 errors for me.
        3. m2eclipse didn't know how to handle the goals in our custom Maven plugin for calling protoc and generating version info. I updated the top-level pom.xml to run the protoc goal during an Eclipse build and ignore the version-info goal. This eliminated nearly all of the remaining ~60 errors.

        After all of that, a fresh import still gives me one error:

        Project 'hadoop-streaming' is missing required source folder: '/Users/chris/git/hadoop-common/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf'

        It appears that Eclipse doesn't like us to have a test resource pointing at a path outside of the module. I'm not yet sure how to fix this one (other than just copying the file we need out of hadoop-yarn-server-resourcemanager into hadoop-streaming, but then we'd have a double-maintenance issue). It doesn't appear to be fatal though.

        I'm curious to see if these fixes address all of the problems that you saw. Thanks!

        Show
        Chris Nauroth added a comment - Hi, Carl Steinbach . I think I understand what's going wrong with fresh imports of the project, and I'm attaching a patch that attempts to fix it. Would you mind trying out this patch in your environment to see if it works for you? Here is a summary of what I did: I changed BUILDING.txt to state that m2eclipse is required, that you should do an mvn install of the whole Hadoop project (not just hadoop-maven-plugins), and that you should import by clicking File -> Import -> Maven -> Existing Maven Projects. Once I did it this way, a fresh import showed ~100 errors. hadoop-common still had some errors for missing compiled protoc test sources. This was because the pom.xml was not specifying generated-test-sources/java as a source directory. I added a call to build-helper-maven-plugin to set that up correctly. This eliminated ~40 errors for me. m2eclipse didn't know how to handle the goals in our custom Maven plugin for calling protoc and generating version info. I updated the top-level pom.xml to run the protoc goal during an Eclipse build and ignore the version-info goal. This eliminated nearly all of the remaining ~60 errors. After all of that, a fresh import still gives me one error: Project 'hadoop-streaming' is missing required source folder: '/Users/chris/git/hadoop-common/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf' It appears that Eclipse doesn't like us to have a test resource pointing at a path outside of the module. I'm not yet sure how to fix this one (other than just copying the file we need out of hadoop-yarn-server-resourcemanager into hadoop-streaming, but then we'd have a double-maintenance issue). It doesn't appear to be fatal though. I'm curious to see if these fixes address all of the problems that you saw. Thanks!
        Chris Nauroth made changes -
        Attachment HADOOP-9489.1.patch [ 12579713 ]
        Chris Nauroth made changes -
        Affects Version/s 3.0.0 [ 12320357 ]
        Target Version/s 3.0.0 [ 12320357 ]
        Hide
        Robert Joseph Evans added a comment -

        I am not really sure that we want to require m2eclipse plug-in. It the past using mvn eclipse:eclipse and the m2plugin caused conflicts with one another, although recently there have not been as many issues with that.

        http://maven.apache.org/plugins/maven-eclipse-plugin/

        has instructions on how to use the maven-eclipse-plugin. This is what is run when you run eclipse:eclipse. You can set M2_REPO manually inside eclipse, or you can run mvn eclipse:configure-workspace to set the M2_REPO. Please refer to

        http://maven.apache.org/plugins/maven-eclipse-plugin/configure-workspace-mojo.html

        for more details about that command because you may need to define some other parameters on the command line to make it work properly with your environment.

        Show
        Robert Joseph Evans added a comment - I am not really sure that we want to require m2eclipse plug-in. It the past using mvn eclipse:eclipse and the m2plugin caused conflicts with one another, although recently there have not been as many issues with that. http://maven.apache.org/plugins/maven-eclipse-plugin/ has instructions on how to use the maven-eclipse-plugin. This is what is run when you run eclipse:eclipse. You can set M2_REPO manually inside eclipse, or you can run mvn eclipse:configure-workspace to set the M2_REPO. Please refer to http://maven.apache.org/plugins/maven-eclipse-plugin/configure-workspace-mojo.html for more details about that command because you may need to define some other parameters on the command line to make it work properly with your environment.
        Hide
        Lokesh Basu added a comment -

        Thanks Chris. The patch provided by you helped me build hadoop on my system. I have been trying to build this for two days. Thanks again

        Show
        Lokesh Basu added a comment - Thanks Chris. The patch provided by you helped me build hadoop on my system. I have been trying to build this for two days. Thanks again
        Hide
        Chris Nauroth added a comment -

        I just noticed that this ancient patch of mine never went in. The old patch still applied with some warnings, but here is a rebase just to make it cleaner.

        Is this patch still helpful? I'm not a regular Eclipse user, so I'm probably not the best judge.

        Show
        Chris Nauroth added a comment - I just noticed that this ancient patch of mine never went in. The old patch still applied with some warnings, but here is a rebase just to make it cleaner. Is this patch still helpful? I'm not a regular Eclipse user, so I'm probably not the best judge.
        Chris Nauroth made changes -
        Attachment HADOOP-9489.2.patch [ 12681630 ]
        Chris Nauroth made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Chris Nauroth made changes -
        Affects Version/s 2.7.0 [ 12327583 ]
        Affects Version/s 3.0.0 [ 12320357 ]
        Target Version/s 3.0.0 [ 12320357 ] 2.7.0 [ 12327583 ]
        Priority Major [ 3 ] Minor [ 4 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12681630/HADOOP-9489.2.patch
        against trunk revision 49c3889.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.security.ssl.TestReloadingX509TrustManager

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5086//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5086//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12681630/HADOOP-9489.2.patch against trunk revision 49c3889. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.security.ssl.TestReloadingX509TrustManager +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5086//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5086//console This message is automatically generated.
        Hide
        Chris Nauroth added a comment -

        The test failure is unrelated. This is a change in build scripts only.

        Show
        Chris Nauroth added a comment - The test failure is unrelated. This is a change in build scripts only.
        Hide
        Chengbing Liu added a comment -

        This is quite helpful to Eclipse users, I think we should have it in the next release. Thanks Chris Nauroth!

        Show
        Chengbing Liu added a comment - This is quite helpful to Eclipse users, I think we should have it in the next release. Thanks Chris Nauroth !
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12681630/HADOOP-9489.2.patch
        against trunk revision 8ca0d95.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5791//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5791//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12681630/HADOOP-9489.2.patch against trunk revision 8ca0d95. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5791//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5791//console This message is automatically generated.
        Hide
        Allen Wittenauer added a comment -

        +1

        Show
        Allen Wittenauer added a comment - +1
        Hide
        Tsuyoshi Ozawa added a comment -

        Carl Steinbach Thank you for taknig this JIRA!

        http://www.sonatype.org/m2eclipse/
        

        The link should be http://eclipse.org/m2e/ since http://www.sonatype.org/m2eclipse/ is redirected to the unrelated page.

        Show
        Tsuyoshi Ozawa added a comment - Carl Steinbach Thank you for taknig this JIRA! http: //www.sonatype.org/m2eclipse/ The link should be http://eclipse.org/m2e/ since http://www.sonatype.org/m2eclipse/ is redirected to the unrelated page.
        Hide
        Tsuyoshi Ozawa added a comment -

        I got some errors:

        An internal error occurred during: "Importing Maven projects".
        Missing parameter for pluginExecutionFilter. groupId, artifactId, versionRange and goals must be specificed, but found: groupId = 'org.apache.hadoop'
        artifactId = 'hadoop-maven-plugins'
        versionRange = 'null'
        goals = '[protoc]'

        Additionally, I also got errors about objects generated by probobuf.

        Show
        Tsuyoshi Ozawa added a comment - I got some errors: An internal error occurred during: "Importing Maven projects". Missing parameter for pluginExecutionFilter. groupId, artifactId, versionRange and goals must be specificed, but found: groupId = 'org.apache.hadoop' artifactId = 'hadoop-maven-plugins' versionRange = 'null' goals = ' [protoc] ' Additionally, I also got errors about objects generated by probobuf.
        Hide
        Tsuyoshi Ozawa added a comment -

        Attaching error log.

        Show
        Tsuyoshi Ozawa added a comment - Attaching error log.
        Tsuyoshi Ozawa made changes -
        Attachment error.log [ 12701727 ]
        Hide
        Chris Nauroth added a comment -

        Thanks for the reviews. I'm attaching patch v003 with the changed hyperlink as suggested by Tsuyoshi.

        Robert Joseph Evans, you had some earlier comments about this patch:

        https://issues.apache.org/jira/browse/HADOOP-9489?focusedCommentId=13638027&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13638027

        Do you object to committing this patch? It sounds like some people have found it useful to apply locally in their environments.

        I'm not a regular Eclipse user, so I'm going to defer to other community members' judgment on this.

        Show
        Chris Nauroth added a comment - Thanks for the reviews. I'm attaching patch v003 with the changed hyperlink as suggested by Tsuyoshi. Robert Joseph Evans , you had some earlier comments about this patch: https://issues.apache.org/jira/browse/HADOOP-9489?focusedCommentId=13638027&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13638027 Do you object to committing this patch? It sounds like some people have found it useful to apply locally in their environments. I'm not a regular Eclipse user, so I'm going to defer to other community members' judgment on this.
        Chris Nauroth made changes -
        Attachment HADOOP-9489.003.patch [ 12701728 ]
        Hide
        Chris Nauroth added a comment -

        Thanks, Tsuyoshi. If this isn't working consistently, then I'm inclined to abandon the patch. Is there anyone else who'd want to pick it up from here and try to work through the errors Tsuyoshi saw?

        Show
        Chris Nauroth added a comment - Thanks, Tsuyoshi. If this isn't working consistently, then I'm inclined to abandon the patch. Is there anyone else who'd want to pick it up from here and try to work through the errors Tsuyoshi saw?
        Chris Nauroth made changes -
        Assignee Chris Nauroth [ cnauroth ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12701728/HADOOP-9489.003.patch
        against trunk revision aa55fd3.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5802//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5802//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12701728/HADOOP-9489.003.patch against trunk revision aa55fd3. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5802//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5802//console This message is automatically generated.
        Hide
        Chengbing Liu added a comment -

        Chris Nauroth The following description

        m2eclipse didn't know how to handle the goals in our custom Maven plugin for calling protoc and generating version info. I updated the top-level pom.xml to run the protoc goal during an Eclipse build and ignore the version-info goal. This eliminated nearly all of the remaining ~60 errors.

        is not a problem any more. So I think we can just remove the change made to

        {base_dir}

        /pom.xml, keep the rest of it, and test again.

        Show
        Chengbing Liu added a comment - Chris Nauroth The following description m2eclipse didn't know how to handle the goals in our custom Maven plugin for calling protoc and generating version info. I updated the top-level pom.xml to run the protoc goal during an Eclipse build and ignore the version-info goal. This eliminated nearly all of the remaining ~60 errors. is not a problem any more. So I think we can just remove the change made to {base_dir} /pom.xml, keep the rest of it, and test again.
        Hide
        Chris Nauroth added a comment -

        Thanks, Chengbing Liu. Here is patch v004, without the change in the root pom.xml file.

        Show
        Chris Nauroth added a comment - Thanks, Chengbing Liu . Here is patch v004, without the change in the root pom.xml file.
        Chris Nauroth made changes -
        Attachment HADOOP-9489.004.patch [ 12701924 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12701924/HADOOP-9489.004.patch
        against trunk revision ca1c00b.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5814//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5814//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12701924/HADOOP-9489.004.patch against trunk revision ca1c00b. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5814//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5814//console This message is automatically generated.

          People

          • Assignee:
            Unassigned
            Reporter:
            Carl Steinbach
          • Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

            • Created:
              Updated:

              Development