Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5714

ContainerExecutor does not order environment map

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
    • Fix Version/s: 3.1.0, 2.10.0, 3.0.4, 2.8.6, 2.9.3
    • Component/s: nodemanager
    • Labels:
    • Environment:

      all (linux and windows alike)

    • Hadoop Flags:
      Reviewed

      Description

      When dumping the launch container script, environment variables are dumped based on the order internally used by the map implementation (hash based). It does not take into consideration that some env variables may refer each other, and so that some env variables must be declared before those referencing them.

      In my case, I ended up having LD_LIBRARY_PATH which was depending on HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a wrong value and so native libraries weren't loaded. Jobs were running but not at their best efficiency. This is just a use case falling into that bug, but I'm sure others may happen as well.

      I already have a patch running in my production environment, i just estimate to 5 days for packaging the patch in the right fashion for JIRA + try my best to add tests.

      Note: The patch is not OS aware with a default empty implementation. I will only implement the Unix version on a 1st release. I'm not used to windows env variables syntax so it will take me more time/research for it.

        Attachments

        1. YARN-5714-branch-2.001.patch
          23 kB
          Jim Brennan
        2. YARN-5714-branch-2.8.001.patch
          23 kB
          Jim Brennan
        3. YARN-5714.009.patch
          23 kB
          Jim Brennan
        4. YARN-5714.008.patch
          23 kB
          Jim Brennan
        5. YARN-5714.007.patch
          23 kB
          Jim Brennan
        6. YARN-5714.006.patch
          22 kB
          Remi Catherinot
        7. YARN-5714.005.patch
          22 kB
          Remi Catherinot
        8. YARN-5714.004.patch
          20 kB
          Remi Catherinot
        9. YARN-5714.003.patch
          20 kB
          Remi Catherinot
        10. YARN-5714.002.patch
          20 kB
          Remi Catherinot
        11. YARN-5714.001.patch
          20 kB
          Remi Catherinot

          Issue Links

            Activity

              People

              • Assignee:
                rcatherinot Remi Catherinot
                Reporter:
                rcatherinot Remi Catherinot
              • Votes:
                0 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 120h
                  120h
                  Remaining:
                  Remaining Estimate - 120h
                  120h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified