Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Building against Facebook's Hadoop's jar is broken.

      1. GIRAPH-376.patch
        0.7 kB
        Nitay Joffe
      2. GIRAPH-376-2.patch
        28 kB
        Nitay Joffe
      3. GIRAPH-376-3.patch
        29 kB
        Nitay Joffe
      4. GIRAPH-376-4.patch
        30 kB
        Nitay Joffe
      5. GIRAPH-376-5.patch
        30 kB
        Nitay Joffe

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Giraph-trunk-Commit #254 (See https://builds.apache.org/job/Giraph-trunk-Commit/254/)
          GIRAPH-376: Facebook Hadoop profile broken. (nitay via aching) (Revision 1402352)

          Result = SUCCESS
          aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1402352
          Files :

          • /giraph/trunk/CHANGELOG
          • /giraph/trunk/giraph-formats-contrib/pom.xml
          • /giraph/trunk/giraph/pom.xml
          • /giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/LoggerUtils.java
          • /giraph/trunk/pom.xml
          Show
          Hudson added a comment - Integrated in Giraph-trunk-Commit #254 (See https://builds.apache.org/job/Giraph-trunk-Commit/254/ ) GIRAPH-376 : Facebook Hadoop profile broken. (nitay via aching) (Revision 1402352) Result = SUCCESS aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1402352 Files : /giraph/trunk/CHANGELOG /giraph/trunk/giraph-formats-contrib/pom.xml /giraph/trunk/giraph/pom.xml /giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/LoggerUtils.java /giraph/trunk/pom.xml
          Hide
          Avery Ching added a comment -

          Thanks Nitay Joffe.

          Show
          Avery Ching added a comment - Thanks Nitay Joffe .
          Hide
          Avery Ching added a comment -

          +1, passed mvn clean install. Committing.

          Show
          Avery Ching added a comment - +1, passed mvn clean install. Committing.
          Hide
          Nitay Joffe added a comment -

          rebased to latest trunk

          Show
          Nitay Joffe added a comment - rebased to latest trunk
          Hide
          Eugene Koontz added a comment -

          Thanks Nitay, it's a good idea to keep them sorted for human readability
          -Eugene

          Show
          Eugene Koontz added a comment - Thanks Nitay, it's a good idea to keep them sorted for human readability -Eugene
          Hide
          Nitay Joffe added a comment -

          It's purely for human readability.

          Show
          Nitay Joffe added a comment - It's purely for human readability.
          Hide
          Eugene Koontz added a comment -

          Hi Nitay,

          I haven't had a chance to test with the Facebook profile. I have worked on it to get the Hadoop 23 and Hadoop 2.0.1 profile working. Not there yet, but no fault of your patch. I'm going to continue with that over on GIRAPH-383. So your patch here is fine as far as I can see; +1.

          One question I had, though: is the lexicographic sorting just for human readability? Or is there a significance to fixing profiles?

          Thanks,

          Eugene

          Show
          Eugene Koontz added a comment - Hi Nitay, I haven't had a chance to test with the Facebook profile. I have worked on it to get the Hadoop 23 and Hadoop 2.0.1 profile working. Not there yet, but no fault of your patch. I'm going to continue with that over on GIRAPH-383 . So your patch here is fine as far as I can see; +1. One question I had, though: is the lexicographic sorting just for human readability? Or is there a significance to fixing profiles? Thanks, Eugene
          Hide
          Nitay Joffe added a comment -

          Eugene Koontz did you have something to add to this, or is it good to go in?

          Show
          Nitay Joffe added a comment - Eugene Koontz did you have something to add to this, or is it good to go in?
          Hide
          Nitay Joffe added a comment -

          Yep, here's what I did to test this patch:
          1) Edit pom.xml replace $

          {hadoop.jar.path}

          with path to my Facebook Hadoop jar. Is there a way to do this as some environment variable or something? I can also try updating the loop script to pass this in as a command line option, but that seems kinda hacky - I like that script as it is since it is completely generic (actually using it for other projects as well).
          2) dev-support/for-each-profile.sh -DskipTests clean install

          Ideally we can get rid of (1) by having FB hadoop jar published somewhere public.
          We can also make (2) the recommended "really thorough" way to test patches or something? Obviously other patches should remove -DskipTests, but right now I see some failures even in trunk and this diff didn't touch any tests so I'm assuming they're not related.

          Show
          Nitay Joffe added a comment - Yep, here's what I did to test this patch: 1) Edit pom.xml replace $ {hadoop.jar.path} with path to my Facebook Hadoop jar. Is there a way to do this as some environment variable or something? I can also try updating the loop script to pass this in as a command line option, but that seems kinda hacky - I like that script as it is since it is completely generic (actually using it for other projects as well). 2) dev-support/for-each-profile.sh -DskipTests clean install Ideally we can get rid of (1) by having FB hadoop jar published somewhere public. We can also make (2) the recommended "really thorough" way to test patches or something? Obviously other patches should remove -DskipTests, but right now I see some failures even in trunk and this diff didn't touch any tests so I'm assuming they're not related.
          Hide
          Avery Ching added a comment -

          Does this work with your new test script?

          Show
          Avery Ching added a comment - Does this work with your new test script?
          Hide
          Nitay Joffe added a comment -

          Any thoughts on this folks?

          Show
          Nitay Joffe added a comment - Any thoughts on this folks?
          Hide
          Eugene Koontz added a comment -

          Nitay, this looks encouraging! I'm having similar problems with the Hadoop 2.0.1 profile. (including the LoggerUtils.java thing you fixed). I'll take a look at this today and test with 2.0.1 and update your patch if need be. Thanks a lot for looking into this.

          -Eugene

          Show
          Eugene Koontz added a comment - Nitay, this looks encouraging! I'm having similar problems with the Hadoop 2.0.1 profile. (including the LoggerUtils.java thing you fixed). I'll take a look at this today and test with 2.0.1 and update your patch if need be. Thanks a lot for looking into this. -Eugene
          Hide
          Nitay Joffe added a comment -

          Okay yeah I see the issue now. The profiles weren't getting inherited correctly from the top level. Here's the fix. Looks like a lot but basically is just moving profile metadata up to top level so it's inherited by everyone. Couple of changes I did actually make:

          1) Sorted dependencies in pom.xmls lexicographically. No logic change.
          2) Minor hack in LoggerUtils.java because hadoop_0.23 TaskAttemptContext#setStatus() doesn't throw IOException and was causing compile error.
          3) Change group of commons-io from org.apache.commons to commons-io (was throwing a warning).
          4) Add that commons-collections dependency that was missing.

          Other than that there is no logic change. I tested on all profiles.

          Show
          Nitay Joffe added a comment - Okay yeah I see the issue now. The profiles weren't getting inherited correctly from the top level. Here's the fix. Looks like a lot but basically is just moving profile metadata up to top level so it's inherited by everyone. Couple of changes I did actually make: 1) Sorted dependencies in pom.xmls lexicographically. No logic change. 2) Minor hack in LoggerUtils.java because hadoop_0.23 TaskAttemptContext#setStatus() doesn't throw IOException and was causing compile error. 3) Change group of commons-io from org.apache.commons to commons-io (was throwing a warning). 4) Add that commons-collections dependency that was missing. Other than that there is no logic change. I tested on all profiles.
          Hide
          Avery Ching added a comment -

          This fixes the FB build, but I still get an error

          mvn clean install -DskipTests -Dhadoop=facebook -Dhadoop.jar.path=/Users/aching/git/git_svn_facebook_hadoop/build/hadoop-0.20-core.jar
          [INFO] ------------------------------------------------------------------------
          [INFO] Reactor Summary:
          [INFO]
          [INFO] Apache Giraph Parent .............................. SUCCESS [0.331s]
          [INFO] Apache Giraph ..................................... SUCCESS [1:21.741s]
          [INFO] Apache Giraph Formats ............................. FAILURE [3.311s]
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 1:25.585s
          [INFO] Finished at: Wed Oct 17 03:22:21 PDT 2012
          [INFO] Final Memory: 32M/91M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal on project giraph-formats-contrib: Could not resolve dependencies for project org.apache.giraph:giraph-formats-contrib:jar:0.2-SNAPSHOT: The following artifacts could not be resolved: org.apache.hadoop:hadoop-core:jar:0.20.1, org.apache.hadoop:hadoop-test:jar:0.20.1: Could not find artifact org.apache.hadoop:hadoop-core:jar:0.20.1 in central (http://repo1.maven.org/maven2) -> [Help 1]
          [ERROR]
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR]
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
          [ERROR]
          [ERROR] After correcting the problems, you can resume the build with the command
          [ERROR] mvn <goals> -rf :giraph-formats-contrib

          Show
          Avery Ching added a comment - This fixes the FB build, but I still get an error mvn clean install -DskipTests -Dhadoop=facebook -Dhadoop.jar.path=/Users/aching/git/git_svn_facebook_hadoop/build/hadoop-0.20-core.jar [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Giraph Parent .............................. SUCCESS [0.331s] [INFO] Apache Giraph ..................................... SUCCESS [1:21.741s] [INFO] Apache Giraph Formats ............................. FAILURE [3.311s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1:25.585s [INFO] Finished at: Wed Oct 17 03:22:21 PDT 2012 [INFO] Final Memory: 32M/91M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project giraph-formats-contrib: Could not resolve dependencies for project org.apache.giraph:giraph-formats-contrib:jar:0.2-SNAPSHOT: The following artifacts could not be resolved: org.apache.hadoop:hadoop-core:jar:0.20.1, org.apache.hadoop:hadoop-test:jar:0.20.1: Could not find artifact org.apache.hadoop:hadoop-core:jar:0.20.1 in central ( http://repo1.maven.org/maven2 ) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :giraph-formats-contrib
          Hide
          Nitay Joffe added a comment -

          Easy fix, just a missing dependency.

          Show
          Nitay Joffe added a comment - Easy fix, just a missing dependency.

            People

            • Assignee:
              Nitay Joffe
              Reporter:
              Nitay Joffe
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development