Uploaded image for project: 'Giraph'
  1. Giraph
  2. GIRAPH-631

Remove Hardcoded Dependency on Hadoop-2.0.3-alpha from YARN and replace with a more flexible Maven config

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0, 1.1.0
    • Fix Version/s: 1.0.0, 1.1.0
    • Component/s: conf and scripts
    • Labels:
      None

      Description

      Currently, Giraph's YARN profile is hardcoded to Version 2.0.3-alpha of Hadoop. This is because of two problems:

      1. Simply creating profiles that can "coexist" such as Hadoop's own -Pdist,native type mvn calls is not possible for us since we use munging and excludes in Maven to prevent compilation of the YARN code where the deps are not included (many profiles) and these excludes don't seem overridable. This has been documented online as a Maven "feature" already.

      2. Simply resetting hadoop.version for the Maven build using a -D option should work and should probably be the right fix for us but in the brief time I played with it (and with our versioning story that affects backporting not decided yet) I did not get it to work myself for Giraph-13 (this is all documented there)

      Option 2 will look like:

      mvn -Phadoop_yarn -Dhadoop.version=YOUR_FAVORITE_YARNY_HADOOP clean install 
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                initialcontext Eli Reisman
                Reporter:
                initialcontext Eli Reisman
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: