Hadoop Common
  1. Hadoop Common
  2. HADOOP-622

Users should be able to change the environment in which there maps/reduces run.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      This would be useful with caching. So you would be avble to say, cache file X and then should be able to change the environment variable like PATH/LD_LIBRARY_PATH to include the local path hwere the file was cached.

        Activity

        Hide
        Nigel Daley added a comment -

        As a workaround, users have to start a new JVM to set the LD_LIBRARY_PATH properly before calling their native libraries. That seems very unfortunate.

        Show
        Nigel Daley added a comment - As a workaround, users have to start a new JVM to set the LD_LIBRARY_PATH properly before calling their native libraries. That seems very unfortunate.
        Hide
        Lance Riedel added a comment -

        For some native apps, the native library loads other dependencies that need a system LD_LIBRARY_PATH be set, not just the following:

        // Add java.library.path; necessary for native-hadoop libraries
        String libraryPath = System.getProperty("java.library.path");
        if (libraryPath != null)

        { vargs.add("-Djava.library.path=" + libraryPath); }

        Currently the TaskRunner gets run by inheriting the parents java processes environment, so there is no chance to change this:

        private void runChild(String[] args, File dir) throws IOException {
        this.process = Runtime.getRuntime().exec(args, null, dir);

        NOTE: the null paramater indicates that the parent env be inherited.

        Also, some of the native libraries that are untarred in the cache may be buried in a sub dir (e.g. /path/to untarred cache/myapp/lib/native.so). This full path isn't known until right before the TaskRunner starts, so either there needs to be a call back, or a relative path to the native library needs to be appended to the cache dir, and sent to the Runtime.getRuntime().exec command.

        Show
        Lance Riedel added a comment - For some native apps, the native library loads other dependencies that need a system LD_LIBRARY_PATH be set, not just the following: // Add java.library.path; necessary for native-hadoop libraries String libraryPath = System.getProperty("java.library.path"); if (libraryPath != null) { vargs.add("-Djava.library.path=" + libraryPath); } Currently the TaskRunner gets run by inheriting the parents java processes environment, so there is no chance to change this: private void runChild(String[] args, File dir) throws IOException { this.process = Runtime.getRuntime().exec(args, null, dir); NOTE: the null paramater indicates that the parent env be inherited. Also, some of the native libraries that are untarred in the cache may be buried in a sub dir (e.g. /path/to untarred cache/myapp/lib/native.so). This full path isn't known until right before the TaskRunner starts, so either there needs to be a call back, or a relative path to the native library needs to be appended to the cache dir, and sent to the Runtime.getRuntime().exec command.
        Hide
        Runping Qi added a comment -

        When the user's app depend some libbrary that use JNI, the corresponding .so files have to be available.
        One way is achieve this is thatthe user put the .so files in the lib subdir. When the tasktracker launch the child, it adds the lib dir to the library path.

        Show
        Runping Qi added a comment - When the user's app depend some libbrary that use JNI, the corresponding .so files have to be available. One way is achieve this is thatthe user put the .so files in the lib subdir. When the tasktracker launch the child, it adds the lib dir to the library path.
        Hide
        Arun C Murthy added a comment -

        Fixed as a part of HADOOP-1660.

        Show
        Arun C Murthy added a comment - Fixed as a part of HADOOP-1660 .

          People

          • Assignee:
            Owen O'Malley
            Reporter:
            Mahadev konar
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development