Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6588

default values for PATH for tasks are not quite correct on Windows

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: mr-am
    • Labels:
      None

      Description

      There are several issues with the way PATH are handled for MR tasks and AMs on Windows.

      • it does not get the PWD as part of the path, unlike the non-Windows case (see MAPREDUCE-6021)
      • the MR AM doesn't get the Hadoop's bin directory in the path explicitly through mapred configuration; rather, it implicitly inherits it from the node manager (see MAPREDUCE-6577)
      • the default value for the MR task PATH uses "%...%" macros which may be expanded on the client side prematurely; this is probably OK for tasks (as it's done by the AM), but not OK for AMs; the late-binding cross-platform macros should be used

      I took a stab at addressing these as part of MAPREDUCE-6577, but I realized that it would be a bigger issue than the fix for MAPREDUCE-6577 itself. Thus, I'm filing a separate JIRA to address the Windows side of things.

      On a related note, there are quite a few brittle tests on testing these (TestMiniMRChildTask, TestMiniMRClientCluster, etc.) which are built in such a way that the tests pass against the current behavior but don't provide a lot of robust test value. We should also update those tests to make them more robust.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                sjlee0 Sangjin Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: