Hadoop Common
  1. Hadoop Common
  2. HADOOP-9489

Eclipse instructions in BUILDING.txt don't work

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: build
    • Labels:
      None
    • Target Version/s:

      Description

      I have tried several times to import Hadoop trunk into Eclipse following the instructions in the BUILDING.txt file, but so far have not been able to get it to work.

      If I use a fresh install of Eclipse 4.2.2, Eclipse will complain about an undefined M2_REPO environment variable. I discovered that this is defined automatically by the M2Eclipse plugin, and think that the BUILDING.txt doc should be updated to explain this.

      After installing M2Eclipse I tried importing the code again, and now get over 2500 errors related to missing class dependencies. Many of these errors correspond to missing classes in the oah*.proto namespace, which makes me think that 'mvn eclipse:eclipse' is not triggering protoc.

      1. HADOOP-9489.1.patch
        3 kB
        Chris Nauroth
      2. eclipse_hadoop_errors.txt
        451 kB
        Carl Steinbach

        Activity

        Hide
        Lokesh Basu added a comment -

        Thanks Chris. The patch provided by you helped me build hadoop on my system. I have been trying to build this for two days. Thanks again

        Show
        Lokesh Basu added a comment - Thanks Chris. The patch provided by you helped me build hadoop on my system. I have been trying to build this for two days. Thanks again
        Hide
        Robert Joseph Evans added a comment -

        I am not really sure that we want to require m2eclipse plug-in. It the past using mvn eclipse:eclipse and the m2plugin caused conflicts with one another, although recently there have not been as many issues with that.

        http://maven.apache.org/plugins/maven-eclipse-plugin/

        has instructions on how to use the maven-eclipse-plugin. This is what is run when you run eclipse:eclipse. You can set M2_REPO manually inside eclipse, or you can run mvn eclipse:configure-workspace to set the M2_REPO. Please refer to

        http://maven.apache.org/plugins/maven-eclipse-plugin/configure-workspace-mojo.html

        for more details about that command because you may need to define some other parameters on the command line to make it work properly with your environment.

        Show
        Robert Joseph Evans added a comment - I am not really sure that we want to require m2eclipse plug-in. It the past using mvn eclipse:eclipse and the m2plugin caused conflicts with one another, although recently there have not been as many issues with that. http://maven.apache.org/plugins/maven-eclipse-plugin/ has instructions on how to use the maven-eclipse-plugin. This is what is run when you run eclipse:eclipse. You can set M2_REPO manually inside eclipse, or you can run mvn eclipse:configure-workspace to set the M2_REPO. Please refer to http://maven.apache.org/plugins/maven-eclipse-plugin/configure-workspace-mojo.html for more details about that command because you may need to define some other parameters on the command line to make it work properly with your environment.
        Chris Nauroth made changes -
        Affects Version/s 3.0.0 [ 12320357 ]
        Target Version/s 3.0.0 [ 12320357 ]
        Chris Nauroth made changes -
        Attachment HADOOP-9489.1.patch [ 12579713 ]
        Hide
        Chris Nauroth added a comment -

        Hi, Carl Steinbach. I think I understand what's going wrong with fresh imports of the project, and I'm attaching a patch that attempts to fix it. Would you mind trying out this patch in your environment to see if it works for you?

        Here is a summary of what I did:

        1. I changed BUILDING.txt to state that m2eclipse is required, that you should do an mvn install of the whole Hadoop project (not just hadoop-maven-plugins), and that you should import by clicking File -> Import -> Maven -> Existing Maven Projects. Once I did it this way, a fresh import showed ~100 errors.
        2. hadoop-common still had some errors for missing compiled protoc test sources. This was because the pom.xml was not specifying generated-test-sources/java as a source directory. I added a call to build-helper-maven-plugin to set that up correctly. This eliminated ~40 errors for me.
        3. m2eclipse didn't know how to handle the goals in our custom Maven plugin for calling protoc and generating version info. I updated the top-level pom.xml to run the protoc goal during an Eclipse build and ignore the version-info goal. This eliminated nearly all of the remaining ~60 errors.

        After all of that, a fresh import still gives me one error:

        Project 'hadoop-streaming' is missing required source folder: '/Users/chris/git/hadoop-common/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf'

        It appears that Eclipse doesn't like us to have a test resource pointing at a path outside of the module. I'm not yet sure how to fix this one (other than just copying the file we need out of hadoop-yarn-server-resourcemanager into hadoop-streaming, but then we'd have a double-maintenance issue). It doesn't appear to be fatal though.

        I'm curious to see if these fixes address all of the problems that you saw. Thanks!

        Show
        Chris Nauroth added a comment - Hi, Carl Steinbach . I think I understand what's going wrong with fresh imports of the project, and I'm attaching a patch that attempts to fix it. Would you mind trying out this patch in your environment to see if it works for you? Here is a summary of what I did: I changed BUILDING.txt to state that m2eclipse is required, that you should do an mvn install of the whole Hadoop project (not just hadoop-maven-plugins), and that you should import by clicking File -> Import -> Maven -> Existing Maven Projects. Once I did it this way, a fresh import showed ~100 errors. hadoop-common still had some errors for missing compiled protoc test sources. This was because the pom.xml was not specifying generated-test-sources/java as a source directory. I added a call to build-helper-maven-plugin to set that up correctly. This eliminated ~40 errors for me. m2eclipse didn't know how to handle the goals in our custom Maven plugin for calling protoc and generating version info. I updated the top-level pom.xml to run the protoc goal during an Eclipse build and ignore the version-info goal. This eliminated nearly all of the remaining ~60 errors. After all of that, a fresh import still gives me one error: Project 'hadoop-streaming' is missing required source folder: '/Users/chris/git/hadoop-common/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf' It appears that Eclipse doesn't like us to have a test resource pointing at a path outside of the module. I'm not yet sure how to fix this one (other than just copying the file we need out of hadoop-yarn-server-resourcemanager into hadoop-streaming, but then we'd have a double-maintenance issue). It doesn't appear to be fatal though. I'm curious to see if these fixes address all of the problems that you saw. Thanks!
        Chris Nauroth made changes -
        Assignee Chris Nauroth [ cnauroth ]
        Carl Steinbach made changes -
        Field Original Value New Value
        Attachment eclipse_hadoop_errors.txt [ 12579667 ]
        Hide
        Carl Steinbach added a comment -

        Attaching a file containing the list build failures flagged by Eclipse.

        Show
        Carl Steinbach added a comment - Attaching a file containing the list build failures flagged by Eclipse.
        Carl Steinbach created issue -

          People

          • Assignee:
            Chris Nauroth
            Reporter:
            Carl Steinbach
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:

              Development