Hadoop Common
  1. Hadoop Common
  2. HADOOP-5658

Eclipse templates fail out of the box; need updating

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: build
    • Labels:
      None
    • Environment:

      Eclipse Ganymede, both on Linux and OS X.

    • Hadoop Flags:
      Reviewed

      Description

      The Hadoop templates, when run "out of the box" have Eclipse build "problems". I've produced a small patch that fixes it.

      1. eclipse.zip
        2 kB
        Philip Zeyliger
      2. HADOOP-5658-v3.patch
        5 kB
        Philip Zeyliger
      3. HADOOP-5658-v2.patch
        5 kB
        Philip Zeyliger
      4. disable java builder.pdf
        68 kB
        Philip Zeyliger
      5. HADOOP-5658.patch
        4 kB
        Philip Zeyliger
      6. eclipse-problems.txt
        34 kB
        Philip Zeyliger

        Activity

        Philip Zeyliger created issue -
        Hide
        Philip Zeyliger added a comment -

        When doing a clean "eclipse-files" ant build, I saw four problems on my way to getting Eclipse happy.

        1. Initially the external builder would fail on "contrib/eclipse-plugin". I hacked around this.
        2. Then, there were still 237 build problems. (I've attached them in eclipse-problems.txt, though they're genuinely uninteresting.)
        3. When reproducing on a Mac, I found that Eclipse would try to run ant with Java 1.5, which doesn't work for Hadoop.
        4. When trying to use an svn tree that I'd already checked out, the external builder would fail to find build.xml.

        The patch attached attacks the four problems mentioned above. Specifically:

        • The "gridmix2" classpaths are updated so that Eclipse recognize the files to be in the right package. This solves the 237 build problems. There are also whitespace changes, which make it easier to diff against the XML files generated by Eclipse's GUI.
        • The "Hadoop_Ant_Builder" custom build task is updated to use Java 1.6, to use a remote ant process, and to reference $ {build_project:build.xml}

          instead of a subdirectory of the workspace. Using Java 1.6 for ant is a necessity on OS X, where Eclipse necessarily is running under Java 1.5. It also "accidentally" solves the problem for the eclipse-plugin, because its build rule is no longer triggered. The eclipse-plugin is broken for the current (Ganymede = 3.4) version of Eclipse, anyway. Finally, using a reference to the build_project directory makes the Eclipse integration work regardless of where you place your checkout.

        Hudson may complain that this patch contains no tests. I tested this patch manually on Eclipse on OS X and Linux.

        With this patch, there are several tests that "just work" if you right click and "Run as Junit". There are many others that don't, because of a dependency on test.build.data, and a few other properties. (See the target "test-core" in build.xml.) I'm hoping to get all the tests working in Eclipse, but that will be a separate JIRA.

        I've also tried to add more details to the wiki page at http://wiki.apache.org/hadoop/EclipseEnvironment.

        Show
        Philip Zeyliger added a comment - When doing a clean "eclipse-files" ant build, I saw four problems on my way to getting Eclipse happy. Initially the external builder would fail on "contrib/eclipse-plugin". I hacked around this. Then, there were still 237 build problems. (I've attached them in eclipse-problems.txt, though they're genuinely uninteresting.) When reproducing on a Mac, I found that Eclipse would try to run ant with Java 1.5, which doesn't work for Hadoop. When trying to use an svn tree that I'd already checked out, the external builder would fail to find build.xml. The patch attached attacks the four problems mentioned above. Specifically: The "gridmix2" classpaths are updated so that Eclipse recognize the files to be in the right package. This solves the 237 build problems. There are also whitespace changes, which make it easier to diff against the XML files generated by Eclipse's GUI. The "Hadoop_Ant_Builder" custom build task is updated to use Java 1.6, to use a remote ant process, and to reference $ {build_project:build.xml} instead of a subdirectory of the workspace. Using Java 1.6 for ant is a necessity on OS X, where Eclipse necessarily is running under Java 1.5. It also "accidentally" solves the problem for the eclipse-plugin, because its build rule is no longer triggered. The eclipse-plugin is broken for the current (Ganymede = 3.4) version of Eclipse, anyway. Finally, using a reference to the build_project directory makes the Eclipse integration work regardless of where you place your checkout. Hudson may complain that this patch contains no tests. I tested this patch manually on Eclipse on OS X and Linux. With this patch, there are several tests that "just work" if you right click and "Run as Junit". There are many others that don't, because of a dependency on test.build.data , and a few other properties. (See the target "test-core" in build.xml.) I'm hoping to get all the tests working in Eclipse, but that will be a separate JIRA. I've also tried to add more details to the wiki page at http://wiki.apache.org/hadoop/EclipseEnvironment .
        Philip Zeyliger made changes -
        Field Original Value New Value
        Attachment HADOOP-5658.patch [ 12405255 ]
        Attachment eclipse-problems.txt [ 12405256 ]
        Philip Zeyliger made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12405256/eclipse-problems.txt
        against trunk revision 764287.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 190 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/188/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12405256/eclipse-problems.txt against trunk revision 764287. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 190 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/188/console This message is automatically generated.
        Hide
        Philip Zeyliger added a comment -

        Trying to attach patch again; Hudson picked up the other attachment.

        Show
        Philip Zeyliger added a comment - Trying to attach patch again; Hudson picked up the other attachment.
        Philip Zeyliger made changes -
        Attachment HADOOP-5658.patch [ 12405335 ]
        Philip Zeyliger made changes -
        Attachment HADOOP-5658.patch [ 12405255 ]
        Philip Zeyliger made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Philip Zeyliger added a comment -

        Re-submitting identical patch; Hudson picked up the wrong file.

        Show
        Philip Zeyliger added a comment - Re-submitting identical patch; Hudson picked up the wrong file.
        Philip Zeyliger made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12405335/HADOOP-5658.patch
        against trunk revision 764605.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/193/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/193/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/193/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/193/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12405335/HADOOP-5658.patch against trunk revision 764605. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/193/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/193/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/193/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/193/console This message is automatically generated.
        Hide
        Konstantin Shvachko added a comment -
        1. I think current version is correct
          -<stringAttribute key="org.eclipse.ui.externaltools.ATTR_LOCATION" value="${workspace_loc:/@PROJECT@/build.xml}"/>
          +<stringAttribute key="org.eclipse.ui.externaltools.ATTR_LOCATION" value="${build_project:build.xml}"/>
          

          because after applying your patch I had to manually reselect Buildfile. Otherwise I could not change build targets etc.

        2. Also you should probably change build targets. This is what it should be
          ATTR_ANT_CLEAN_TARGETS = clean
          ATTR_ANT_MANUAL_TARGETS = compile-core-test
          ATTR_ANT_AFTER_CLEAN_TARGETS = compile-core-test
          

          instead of

          ATTR_ANT_AFTER_CLEAN_TARGETS = clean
          

        Other than that didn't see any problems.

        Show
        Konstantin Shvachko added a comment - I think current version is correct -<stringAttribute key= "org.eclipse.ui.externaltools.ATTR_LOCATION" value= "${workspace_loc:/@PROJECT@/build.xml}" /> +<stringAttribute key= "org.eclipse.ui.externaltools.ATTR_LOCATION" value= "${build_project:build.xml}" /> because after applying your patch I had to manually reselect Buildfile. Otherwise I could not change build targets etc. Also you should probably change build targets. This is what it should be ATTR_ANT_CLEAN_TARGETS = clean ATTR_ANT_MANUAL_TARGETS = compile-core-test ATTR_ANT_AFTER_CLEAN_TARGETS = compile-core-test instead of ATTR_ANT_AFTER_CLEAN_TARGETS = clean Other than that didn't see any problems.
        Hide
        Konstantin Shvachko added a comment -

        One more thing: is it possible to turn off the default "Java Builder"?

        Show
        Konstantin Shvachko added a comment - One more thing: is it possible to turn off the default "Java Builder"?
        Hide
        Philip Zeyliger added a comment -

        Thanks for taking a look!

        -<stringAttribute key="org.eclipse.ui.externaltools.ATTR_LOCATION" value="${workspace_loc:/@PROJECT@/build.xml}"/>
        +<stringAttribute key="org.eclipse.ui.externaltools.ATTR_LOCATION" value="${build_project:build.xml}"/>
        

        You're right, this change shouldn't have been necessary. I was initially concerned that this forced me to put my src checkout into workspace/project, but it should be resolving the project location correctly from this. The documentation (for those following along at home) is at http://help.eclipse.org/stable/index.jsp?topic=/org.eclipse.platform.doc.user/concepts/concepts-exttools.htm . I've reverted this bit back.

        is it possible to turn off the default "Java Builder"?

        This is an interesting idea. When I tried it, Eclipse tried to warn me away from it (screenshot attached in next comment). I also found that I could no longer do "[Right-click] Run...as Junit Test" on simple tests that used to work with that feature. I suspect that with a little bit more classpath massaging, turning off the Java Builder will work. I'd rather not do it in this pass.

        Show
        Philip Zeyliger added a comment - Thanks for taking a look! -<stringAttribute key= "org.eclipse.ui.externaltools.ATTR_LOCATION" value= "${workspace_loc:/@PROJECT@/build.xml}" /> +<stringAttribute key= "org.eclipse.ui.externaltools.ATTR_LOCATION" value= "${build_project:build.xml}" /> You're right, this change shouldn't have been necessary. I was initially concerned that this forced me to put my src checkout into workspace/project, but it should be resolving the project location correctly from this. The documentation (for those following along at home) is at http://help.eclipse.org/stable/index.jsp?topic=/org.eclipse.platform.doc.user/concepts/concepts-exttools.htm . I've reverted this bit back. is it possible to turn off the default "Java Builder"? This is an interesting idea. When I tried it, Eclipse tried to warn me away from it (screenshot attached in next comment). I also found that I could no longer do " [Right-click] Run...as Junit Test" on simple tests that used to work with that feature. I suspect that with a little bit more classpath massaging, turning off the Java Builder will work. I'd rather not do it in this pass.
        Hide
        Philip Zeyliger added a comment -

        Uploading new patch, and the Eclipse warning I'm talking about.

        Show
        Philip Zeyliger added a comment - Uploading new patch, and the Eclipse warning I'm talking about.
        Philip Zeyliger made changes -
        Attachment disable java builder.pdf [ 12405718 ]
        Attachment HADOOP-5658-v2.patch [ 12405719 ]
        Philip Zeyliger made changes -
        Status Patch Available [ 10002 ] In Progress [ 3 ]
        Hide
        Konstantin Shvachko added a comment -

        Thanks Philip for fixing it.
        Yes, lets skip "Java Builder" for now.
        We are testing it with Ravi. He is trying to setup his eclipse and we are comparing to my configurations, which I setup some time ago.
        In order to make it work out of the box .classpath file needs a couple of extra class folders, which are

        <classpathentry kind="lib" path="build/test/classes"/>
        <classpathentry kind="lib" path="build/test/webapps"/>
        <classpathentry kind="lib" path="build/classes"/>
        <classpathentry kind="lib" path="build/conf"/>
        
        Show
        Konstantin Shvachko added a comment - Thanks Philip for fixing it. Yes, lets skip "Java Builder" for now. We are testing it with Ravi. He is trying to setup his eclipse and we are comparing to my configurations, which I setup some time ago. In order to make it work out of the box .classpath file needs a couple of extra class folders, which are <classpathentry kind= "lib" path= "build/test/classes" /> <classpathentry kind= "lib" path= "build/test/webapps" /> <classpathentry kind= "lib" path= "build/classes" /> <classpathentry kind= "lib" path= "build/conf" />
        Hide
        Philip Zeyliger added a comment -

        {{ <classpathentry kind="lib" path="build/test/webapps"/> }}

        This directory doesn't exist in my build. Did you perhaps make that one manually at some point? For a while I had a "webapps.jar", but I managed to get rid of it.

        Looking at the "NameNode.launch" that I've got started over here (coming soon to a JIRA near you!), I ended up adding conf/, build/ (because that's where webapps/ gets placed), and build/classes. In the patch I'm about to upload, I've got conf/ and build/classes added. I'm having trouble with build/, because Eclipse refuses to add both: "Cannot nest 'hadoop-trunk/build/test/classes' inside library 'hadoop-trunk/build'".

        Show
        Philip Zeyliger added a comment - {{ <classpathentry kind="lib" path="build/test/webapps"/> }} This directory doesn't exist in my build. Did you perhaps make that one manually at some point? For a while I had a "webapps.jar", but I managed to get rid of it. Looking at the "NameNode.launch" that I've got started over here (coming soon to a JIRA near you!), I ended up adding conf/, build/ (because that's where webapps/ gets placed), and build/classes. In the patch I'm about to upload, I've got conf/ and build/classes added. I'm having trouble with build/, because Eclipse refuses to add both: "Cannot nest 'hadoop-trunk/build/test/classes' inside library 'hadoop-trunk/build'".
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I think we no longer have build/conf.

        Show
        Tsz Wo Nicholas Sze added a comment - I think we no longer have build/conf.
        Hide
        Philip Zeyliger added a comment -

        Here's a version with "build/classes" and "conf/" added. (Not build/conf, which doesn't exist.)

        Show
        Philip Zeyliger added a comment - Here's a version with "build/classes" and "conf/" added. (Not build/conf, which doesn't exist.)
        Philip Zeyliger made changes -
        Attachment HADOOP-5658-v3.patch [ 12405814 ]
        Hide
        Ravi Phulari added a comment - - edited

        Thanks Philip , I was trying to setup Hadoop dev environment from scratch with your patch

        Builder is failing complaining - Invalid External Tool Builder

        I followed all instructions given on - http://wiki.apache.org/hadoop/EclipseEnvironment
        Which includes
        1) Checkout Hadoop source using eclipse svn repository wizard adn Create new Hadoop project
        2) Apply patch HADOOP-5658v3
        3) Run ant <eclipse-files>
        4) Refresh the Eclipse project.
        Ensured that the Java version used matches the version of the project (currently 1.6 is used).

        Could you please verify at your end and update patch with instruction that you are following to setup Hadoop environment from scratch.

          • Eclipse Ganymede on Mac OS X.

        Thanks,

        Show
        Ravi Phulari added a comment - - edited Thanks Philip , I was trying to setup Hadoop dev environment from scratch with your patch Builder is failing complaining - Invalid External Tool Builder I followed all instructions given on - http://wiki.apache.org/hadoop/EclipseEnvironment Which includes 1) Checkout Hadoop source using eclipse svn repository wizard adn Create new Hadoop project 2) Apply patch HADOOP-5658 v3 3) Run ant <eclipse-files> 4) Refresh the Eclipse project. Ensured that the Java version used matches the version of the project (currently 1.6 is used). Could you please verify at your end and update patch with instruction that you are following to setup Hadoop environment from scratch. Eclipse Ganymede on Mac OS X. Thanks,
        Hide
        Philip Zeyliger added a comment -

        Eclipse on OS X is definitely whinier than on Linux, but I've gotten this to work on both. I'm running through it as I type this up:

        1. Turn off Build Automatically
        2. Make sure that Eclipse Preferences...Installed JREs contains "JVM 1.6"
        3. Make sure ANT_HOME is set in Classpath Variables in Eclipse settings.
        4. Checked out Hadoop using Subversive as a New Java Project; using Java 1.6; output path is build/eclipse-classfiles
        5. patch -p0 < /tmp/HADOOP-5658-v3.patch
        6. ant eclipse-files (When doing this for myself, I typically do compile-core-test at this point too, just to accelerate things later.)
        7. Right-click on project and hit refresh in Eclipse
        8. Right-click on project and hit Build Project in Eclipse
        9. Refresh again
        10. Build project again.
        11. Check Problems: no errors, just warnings. Good.

        So, it worked pretty much for me. I haven't managed to avoid the refresh/build twice problem, though I've gotten it to work better if I built compile-core-test on the command line ahead of time.

        What version of Eclipse are you running? I'm at 3.4.2 (M20090211-1700) (on OS X). To figure out what's going on, we'll also have to compare our .classpath, .externalToolBuilders/Hadoop_Ant_Builder.launch, and .project files. I'm attaching mine as eclipse.zip, for comparison. Can you send a screenshot of the error you're seeing?

        Show
        Philip Zeyliger added a comment - Eclipse on OS X is definitely whinier than on Linux, but I've gotten this to work on both. I'm running through it as I type this up: Turn off Build Automatically Make sure that Eclipse Preferences...Installed JREs contains "JVM 1.6" Make sure ANT_HOME is set in Classpath Variables in Eclipse settings. Checked out Hadoop using Subversive as a New Java Project; using Java 1.6; output path is build/eclipse-classfiles patch -p0 < /tmp/ HADOOP-5658 -v3.patch ant eclipse-files (When doing this for myself, I typically do compile-core-test at this point too, just to accelerate things later.) Right-click on project and hit refresh in Eclipse Right-click on project and hit Build Project in Eclipse Refresh again Build project again. Check Problems: no errors, just warnings. Good. So, it worked pretty much for me. I haven't managed to avoid the refresh/build twice problem, though I've gotten it to work better if I built compile-core-test on the command line ahead of time. What version of Eclipse are you running? I'm at 3.4.2 (M20090211-1700) (on OS X). To figure out what's going on, we'll also have to compare our .classpath, .externalToolBuilders/Hadoop_Ant_Builder.launch, and .project files. I'm attaching mine as eclipse.zip, for comparison. Can you send a screenshot of the error you're seeing?
        Hide
        Philip Zeyliger added a comment -

        For comparison's case, my (generated) .classpath, .externalToolBuilders, and .project.

        Show
        Philip Zeyliger added a comment - For comparison's case, my (generated) .classpath, .externalToolBuilders, and .project.
        Philip Zeyliger made changes -
        Attachment eclipse.zip [ 12405827 ]
        Hide
        Ravi Phulari added a comment -

        +1

        Thanks Philip , I followed all your steps and it builds perfectly .

        ~Eariler it failed because I was using different output path for Eclipse. ~

        Show
        Ravi Phulari added a comment - +1 Thanks Philip , I followed all your steps and it builds perfectly . ~Eariler it failed because I was using different output path for Eclipse. ~
        Hide
        Konstantin Shvachko added a comment -

        I just committed this. Thank you Philip.

        Show
        Konstantin Shvachko added a comment - I just committed this. Thank you Philip.
        Konstantin Shvachko made changes -
        Status In Progress [ 3 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Assignee Philip Zeyliger [ philip ]
        Fix Version/s 0.21.0 [ 12313563 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #811 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/811/)
        . Fix Eclipse templates. Contributed by Philip Zeyliger.

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #811 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/811/ ) . Fix Eclipse templates. Contributed by Philip Zeyliger.
        Hide
        Philip Zeyliger added a comment -

        Thanks Konstantin and Ravi for testing this out.

        I put together a screencast of how to do this setup – hopefully watching this on a screen will help – at http://www.cloudera.com/blog/2009/04/20/configuring-eclipse-for-hadoop-development-a-screencast/ .

        Show
        Philip Zeyliger added a comment - Thanks Konstantin and Ravi for testing this out. I put together a screencast of how to do this setup – hopefully watching this on a screen will help – at http://www.cloudera.com/blog/2009/04/20/configuring-eclipse-for-hadoop-development-a-screencast/ .
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        1d 12h 36m 1 Philip Zeyliger 13/Apr/09 19:28
        Open Open Patch Available Patch Available
        3m 49s 2 Philip Zeyliger 13/Apr/09 19:28
        Patch Available Patch Available In Progress In Progress
        3d 21h 16m 1 Philip Zeyliger 17/Apr/09 16:45
        In Progress In Progress Resolved Resolved
        9h 8m 1 Konstantin Shvachko 18/Apr/09 01:54
        Resolved Resolved Closed Closed
        493d 19h 43m 1 Tom White 24/Aug/10 21:37

          People

          • Assignee:
            Philip Zeyliger
            Reporter:
            Philip Zeyliger
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development