Giraph
  1. Giraph
  2. GIRAPH-168

Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than HADOOP_FACEBOOK) and remove usage of HADOOP

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      This JIRA relates to the mail thread here:

      http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser

      Currently we check for the munge flags HADOOP, HADOOP_FACEBOOK and HADOOP_NON_SECURE when using munge in a few places. Hopefully we can eliminate usage of munge in the future, but until then, we can mitigate the complexity by consolidating the number of flags checked. This JIRA renames HADOOP_FACEBOOK to HADOOP_SECURE, and removes usages of HADOOP, to handle the same conditional compilation requirements. It also makes it easier to add more maven profiles so that we can easily increase our hadoop version coverage.

      This patch modifies the existing hadoop_facebook profile to use the new HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.

      It also adds a new hadoop maven profile, hadoop_trunk, which also sets HADOOP_SECURE.

      Finally, it adds a default profile, hadoop_0.20.203. This is needed so that we can specify its dependencies separately from hadoop_trunk, because the hadoop dependencies have changed between trunk and 0.205.0 - the former requires hadoop-common, hadoop-mapreduce-client-core, and hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.

      With this patch, the following passes:

      mvn clean verify && mvn -Phadoop_trunk clean verify && mvn -Phadoop_0.20.203 clean verify
      

      Current problems:

      • I couldn't get -Phadoop_facebook to work; does this work outside of Facebook?
      1. GIRAPH-168.patch
        16 kB
        Eugene Koontz
      2. GIRAPH-168.patch
        19 kB
        Eugene Koontz
      3. GIRAPH-168.patch
        19 kB
        Eugene Koontz
      4. GIRAPH-168.patch
        19 kB
        Eugene Koontz
      5. GIRAPH-168.patch
        25 kB
        Eugene Koontz
      6. GIRAPH-168.patch
        23 kB
        Eugene Koontz

        Issue Links

          Activity

          Eugene Koontz created issue -
          Eugene Koontz made changes -
          Field Original Value New Value
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.2.0 [ 12319640 ]
          Assignee Eugene Koontz [ ekoontz ]
          Eugene Koontz made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Eugene Koontz made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Eugene Koontz made changes -
          Attachment GIRAPH-168.patch [ 12519388 ]
          Eugene Koontz made changes -
          Summary Simplify munge directive usage with new munge flag HADOOP_SECURE rather than HADOOP_FACEBOOK and HADOOP_NON_SECURE Simplify munge directive usage with new munge flag HADOOP_SECURE rather than HADOOP_FACEBOOK
          Eugene Koontz made changes -
          Summary Simplify munge directive usage with new munge flag HADOOP_SECURE rather than HADOOP_FACEBOOK Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than HADOOP_FACEBOOK) and remove usage of HADOOP
          Description This JIRA relates to the mail thread here:

          http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser

          Currently we check for the munge flags HADOOP and HADOOP_FACEBOOK and HADOOP_NON_SECURE when using munge in a few places. Hopefully we can eliminate usage of munge in the future, but until then, we can mitigate the complexity by consolidating the number of flags checked. This JIRA proposes a single flag, HADOOP_SECURE, to handle the same conditional compilation requirements. It also makes it easier to add more maven profiles so that we can easily increase our hadoop version coverage.

          This patch modifies the existing hadoop_facebook profile to use the new HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.

          It also adds a new hadoop maven profile, hadoop_trunk, which also sets HADOOP_SECURE.

          Finally, it adds a default profile, hadoop_0.20.203. This is needed so that we can specify its dependencies separately from hadoop_trunk, because the hadoop dependencies have changed between trunk and 0.205.0 - the former requires hadoop-common, hadoop-mapreduce-client-core, and hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.

          With this patch, the following passes:

          {code}
          mvn clean verify && mvn -Phadoop_trunk clean verify && mvn -Phadoop_0.20.203 clean verify
          {code}

          Current problems:

          * I left in place the usage of HADOOP_NON_SECURE, but note that the profile that uses this is hadoop_non_secure, which fails to compile on trunk: https://issues.apache.org/jira/browse/GIRAPH-167 .

          * I couldn't get -Phadoop_facebook to work; does this work outside of Facebook?

          This JIRA relates to the mail thread here:

          http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser

          Currently we check for the munge flags HADOOP, HADOOP_FACEBOOK and HADOOP_NON_SECURE when using munge in a few places. Hopefully we can eliminate usage of munge in the future, but until then, we can mitigate the complexity by consolidating the number of flags checked. This JIRA renames HADOOP_FACEBOOK to HADOOP_SECURE, and removes usages of HADOOP, to handle the same conditional compilation requirements. It also makes it easier to add more maven profiles so that we can easily increase our hadoop version coverage.

          This patch modifies the existing hadoop_facebook profile to use the new HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.

          It also adds a new hadoop maven profile, hadoop_trunk, which also sets HADOOP_SECURE.

          Finally, it adds a default profile, hadoop_0.20.203. This is needed so that we can specify its dependencies separately from hadoop_trunk, because the hadoop dependencies have changed between trunk and 0.205.0 - the former requires hadoop-common, hadoop-mapreduce-client-core, and hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.

          With this patch, the following passes:

          {code}
          mvn clean verify && mvn -Phadoop_trunk clean verify && mvn -Phadoop_0.20.203 clean verify
          {code}

          Current problems:

          * I left in place the usage of HADOOP_NON_SECURE, but note that the profile that uses this is hadoop_non_secure, which fails to compile on trunk: https://issues.apache.org/jira/browse/GIRAPH-167 .

          * I couldn't get -Phadoop_facebook to work; does this work outside of Facebook?

          Eugene Koontz made changes -
          Attachment GIRAPH-168.patch [ 12521072 ]
          Eugene Koontz made changes -
          Attachment GIRAPH-168.patch [ 12521202 ]
          Eugene Koontz made changes -
          Link This issue relates to HADOOP-6419 [ HADOOP-6419 ]
          Eugene Koontz made changes -
          Attachment GIRAPH-168.patch [ 12521541 ]
          Eugene Koontz made changes -
          Link This issue relates to GIRAPH-14 [ GIRAPH-14 ]
          Eugene Koontz made changes -
          Link This issue relates to HADOOP-6904 [ HADOOP-6904 ]
          Eugene Koontz made changes -
          Attachment GIRAPH-168.patch [ 12521788 ]
          Eugene Koontz made changes -
          Attachment GIRAPH-168.patch [ 12522023 ]
          Avery Ching made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Eugene Koontz made changes -
          Link This issue is related to GIRAPH-183 [ GIRAPH-183 ]
          Eugene Koontz made changes -
          Link This issue is related to GIRAPH-196 [ GIRAPH-196 ]
          Eugene Koontz made changes -
          Link This issue is related to GIRAPH-212 [ GIRAPH-212 ]
          Eugene Koontz made changes -
          Link This issue is related to GIRAPH-485 [ GIRAPH-485 ]

            People

            • Assignee:
              Eugene Koontz
              Reporter:
              Eugene Koontz
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development