Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1570

Add a per-job configuration knob to control loading of native hadoop libraries

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.3, 0.13.0
    • Fix Version/s: 0.14.0
    • Component/s: io
    • Labels:
      None

      Description

      As it exists today, native-hadoop libraries are loaded automatically if libhadoop.so is present; however we have sporadically seen issues (HADOOP-1545) since native direct-buffers aren't very well understood. The only way to switch off usage of these is to remove the native libraries which is a maintenence issue for large clusters...

      Hence I propose we add a per-job config knob: hadoop.native.lib (set to true by default) which can be used to control usage of native libraries even when the libraries are present e.g. we can have hadoop installed with native libraries present and then use this knob to switch off their usage in rare cases we see issues with them; thus aiding maintenence.

      1. HADOOP-1570_1_20070709.patch
        16 kB
        Arun C Murthy
      2. HADOOP-1570_2_20070713.patch
        16 kB
        Arun C Murthy

        Issue Links

          Activity

          Hide
          acmurthy Arun C Murthy added a comment -

          Straight-forward patch...

          Show
          acmurthy Arun C Murthy added a comment - Straight-forward patch...
          Show
          hadoopqa Hadoop QA added a comment - +1 http://issues.apache.org/jira/secure/attachment/12361402/HADOOP-1570_1_20070709.patch applied and successfully tested against trunk revision r554144. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/374/testReport/ Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/374/console
          Hide
          cutting Doug Cutting added a comment -

          This looks great, except the methods to set the configuration property don't belong on JobConf. JobConf should only have methods specific to jobs, settings read by the MapReduce kernel. We've not been great about enforcing this in the past, but don't want JobConf to just turn into a heap of settings. Rather these should be static methods on related code, like util.NativeCodeLoader.

          Show
          cutting Doug Cutting added a comment - This looks great, except the methods to set the configuration property don't belong on JobConf. JobConf should only have methods specific to jobs, settings read by the MapReduce kernel. We've not been great about enforcing this in the past, but don't want JobConf to just turn into a heap of settings. Rather these should be static methods on related code, like util.NativeCodeLoader.
          Hide
          acmurthy Arun C Murthy added a comment -

          Incorporated Doug's comments, thanks for the f/b!

          Show
          acmurthy Arun C Murthy added a comment - Incorporated Doug's comments, thanks for the f/b!
          Hide
          cutting Doug Cutting added a comment -

          I just committed this. Thanks, Arun!

          Show
          cutting Doug Cutting added a comment - I just committed this. Thanks, Arun!
          Hide
          hudson Hudson added a comment -
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Nightly #154 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/154/ )

            People

            • Assignee:
              acmurthy Arun C Murthy
              Reporter:
              acmurthy Arun C Murthy
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development