Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4322

Fix command-line length abort issues on Windows

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1-win
    • Component/s: tasktracker
    • Labels:
      None
    • Environment:

      Windows, downstream applications with long aggregate classpaths

      Description

      When a task is started on the tasktracker, it creates a small batch file to invoke java and runs that batch. Within the batch file, the invocation of Java currently has -classpath ${CLASSPATH} inline to the command. That line often exceeds 8000 characters. This is ok for most linux distributions because the line limit env variable is often set much higher than this. However, for Windows this cause cmd to abort execution. This surfaces in Hadoop as an unknown failure mode for the task.

      I think the easiest and most natural way to fix this is to push the -classpath option into a config file to take the longest variable part of the line and put it somewhere that scales better.

        Attachments

        1. MAPREDUCE-4322-branch-1-win.patch
          5 kB
          Ivan Mitic
        2. MAPREDUCE-4322-branch-1-win(2).patch
          10 kB
          Ivan Mitic
        3. MAPREDUCE-4322-branch-1-win(3).patch
          13 kB
          Ivan Mitic
        4. MAPREDUCE-4322-branch-1-win(4).patch
          13 kB
          Ivan Mitic
        5. MAPREDUCE-4322-branch-1-win(5).patch
          13 kB
          Ivan Mitic

          Issue Links

            Activity

              People

              • Assignee:
                ivanmi Ivan Mitic
                Reporter:
                jgordon John Gordon
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 12h
                  12h
                  Remaining:
                  Remaining Estimate - 12h
                  12h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified