Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6588

default values for PATH for tasks are not quite correct on Windows

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.6.0
    • None
    • mr-am
    • None


      There are several issues with the way PATH are handled for MR tasks and AMs on Windows.

      • it does not get the PWD as part of the path, unlike the non-Windows case (see MAPREDUCE-6021)
      • the MR AM doesn't get the Hadoop's bin directory in the path explicitly through mapred configuration; rather, it implicitly inherits it from the node manager (see MAPREDUCE-6577)
      • the default value for the MR task PATH uses "%...%" macros which may be expanded on the client side prematurely; this is probably OK for tasks (as it's done by the AM), but not OK for AMs; the late-binding cross-platform macros should be used

      I took a stab at addressing these as part of MAPREDUCE-6577, but I realized that it would be a bigger issue than the fix for MAPREDUCE-6577 itself. Thus, I'm filing a separate JIRA to address the Windows side of things.

      On a related note, there are quite a few brittle tests on testing these (TestMiniMRChildTask, TestMiniMRClientCluster, etc.) which are built in such a way that the tests pass against the current behavior but don't provide a lot of robust test value. We should also update those tests to make them more robust.


        Issue Links


          This comment will be Viewable by All Users Viewable by All Users


            Unassigned Unassigned
            sjlee0 Sangjin Lee




                Issue deployment