Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6588

default values for PATH for tasks are not quite correct on Windows

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.6.0
    • None
    • mr-am
    • None

    Description

      There are several issues with the way PATH are handled for MR tasks and AMs on Windows.

      • it does not get the PWD as part of the path, unlike the non-Windows case (see MAPREDUCE-6021)
      • the MR AM doesn't get the Hadoop's bin directory in the path explicitly through mapred configuration; rather, it implicitly inherits it from the node manager (see MAPREDUCE-6577)
      • the default value for the MR task PATH uses "%...%" macros which may be expanded on the client side prematurely; this is probably OK for tasks (as it's done by the AM), but not OK for AMs; the late-binding cross-platform macros should be used

      I took a stab at addressing these as part of MAPREDUCE-6577, but I realized that it would be a bigger issue than the fix for MAPREDUCE-6577 itself. Thus, I'm filing a separate JIRA to address the Windows side of things.

      On a related note, there are quite a few brittle tests on testing these (TestMiniMRChildTask, TestMiniMRClientCluster, etc.) which are built in such a way that the tests pass against the current behavior but don't provide a lot of robust test value. We should also update those tests to make them more robust.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            sjlee0 Sangjin Lee

            Dates

              Created:
              Updated:

              Slack

                Issue deployment