Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      I have noticed that org.apache.hadoop.mapred.MapReduceChildJVM doesn't forward the value of -Djava.library.path= from the parent JVM to the child JVM. Thus if one wants to use native libraries for compression the only option seems to be to manually include relevant java.library.path settings into the mapred-site.xml (as mapred.[map|reduce].child.java.opts).

      This seems to be a change in behavior compared to MR1 where TaskRunner.java used to do that:

      String libraryPath = System.getProperty("java.library.path");
          if (libraryPath == null) {
            libraryPath = workDir.getAbsolutePath();
          } else {
            libraryPath += SYSTEM_PATH_SEPARATOR + workDir;
          }
          boolean hasUserLDPath = false;
          for(int i=0; i<javaOptsSplit.length ;i++) {
            if(javaOptsSplit[i].startsWith("-Djava.library.path=")) {
              javaOptsSplit[i] += SYSTEM_PATH_SEPARATOR + libraryPath;
              hasUserLDPath = true;
              break;
            }
          }
          if(!hasUserLDPath) {
            vargs.add("-Djava.library.path=" + libraryPath);
          }
          for (int i = 0; i < javaOptsSplit.length; i++) {
            vargs.add(javaOptsSplit[i]);
          }
      

      Is this a regression or a deliberate choice?

      1. MAPREDUCE-3693-2.patch.txt
        1 kB
        Roman Shaposhnik
      2. MAPREDUCE-3693.patch.txt
        2 kB
        Roman Shaposhnik

        Issue Links

          Activity

          Roman Shaposhnik created issue -
          Arun C Murthy made changes -
          Field Original Value New Value
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Not A Problem [ 8 ]
          Eli Collins made changes -
          Resolution Not A Problem [ 8 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Eli Collins made changes -
          Summary mapreduce's MapReduceChildJVM no longer proxies value of java.library.path Job's shouldn't have to configure the location of compression libraries on the server in mapred-site.xml
          Eli Collins made changes -
          Fix Version/s 0.23.1 [ 12318883 ]
          Target Version/s 0.23.1 [ 12318883 ]
          Arun C Murthy made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Not A Problem [ 8 ]
          Roman Shaposhnik made changes -
          Attachment MAPREDUCE-3693.patch.txt [ 12511784 ]
          Roman Shaposhnik made changes -
          Resolution Not A Problem [ 8 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Assignee Roman Shaposhnik [ rvs ]
          Roman Shaposhnik made changes -
          Status Reopened [ 4 ] Patch Available [ 10002 ]
          Arun C Murthy made changes -
          Summary Job's shouldn't have to configure the location of compression libraries on the server in mapred-site.xml Add admin env to mapred-default.xml
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Roman Shaposhnik made changes -
          Attachment MAPREDUCE-3693-2.patch.txt [ 12512060 ]
          Eli Collins made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags Reviewed [ 10343 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Issue Type Bug [ 1 ] Improvement [ 4 ]
          Fix Version/s 0.23.1 [ 12318883 ]
          Priority Major [ 3 ] Minor [ 4 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Anupam Seth made changes -
          Link This issue relates to MAPREDUCE-4072 [ MAPREDUCE-4072 ]

            People

            • Assignee:
              Roman Shaposhnik
              Reporter:
              Roman Shaposhnik
            • Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development