Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Not A Problem
    • Affects Version/s: 1.0.4
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      There is rare happenings during map or reduce phase, but mostly in map phase, the Exception messages:
      java.lang.Throwable: Child Error
      at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
      Caused by: java.io.IOException: Task process exit with nonzero status of 126.
      at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

      and error logs are cleaned, so It's very hard to debug.

      but I compared DefaultTaskController.java with 0.22, they use "bash command" to start the job scritp, but 1.0.4 use "bash, "-c", command".

      I removed "-c", everything is ok, 126 error code never happen again.

      I read man document of bash, it indicates when fork a new thread with write command, another thread with "bash -c" also has a writable fd. so I think it could return 126 status occasionally.

      So, there is only one line fix for this issue.

        Issue Links

          Activity

          Fengdong Yu created issue -
          Fengdong Yu made changes -
          Field Original Value New Value
          Description There is rare happenings during map or reduce phase, but mostly in map phase, the Exception messages:
          java.lang.Throwable: Child Error
          at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
          Caused by: java.io.IOException: Task process exit with nonzero status of 126.
          at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

          and error logs are cleaned, so It's very hard to debug.

          but I compared DefaultTaskController.java with 0.22, they use "bash command" to start the job scritp, but 1.0.4 use "bash, "-c", command".

          I removed "-c", everything is ok, 126 error code never happen again.

          I read man document of bash, it indicates when fork a new thread with write command, another thread with "bash -c" also has a writable fd. so it could return 126 status occasionally.
          There is rare happenings during map or reduce phase, but mostly in map phase, the Exception messages:
          java.lang.Throwable: Child Error
          at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
          Caused by: java.io.IOException: Task process exit with nonzero status of 126.
          at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

          and error logs are cleaned, so It's very hard to debug.

          but I compared DefaultTaskController.java with 0.22, they use "bash command" to start the job scritp, but 1.0.4 use "bash, "-c", command".

          I removed "-c", everything is ok, 126 error code never happen again.

          I read man document of bash, it indicates when fork a new thread with write command, another thread with "bash -c" also has a writable fd. so I think it could return 126 status occasionally.
          Fengdong Yu made changes -
          Description There is rare happenings during map or reduce phase, but mostly in map phase, the Exception messages:
          java.lang.Throwable: Child Error
          at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
          Caused by: java.io.IOException: Task process exit with nonzero status of 126.
          at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

          and error logs are cleaned, so It's very hard to debug.

          but I compared DefaultTaskController.java with 0.22, they use "bash command" to start the job scritp, but 1.0.4 use "bash, "-c", command".

          I removed "-c", everything is ok, 126 error code never happen again.

          I read man document of bash, it indicates when fork a new thread with write command, another thread with "bash -c" also has a writable fd. so I think it could return 126 status occasionally.
          There is rare happenings during map or reduce phase, but mostly in map phase, the Exception messages:
          java.lang.Throwable: Child Error
          at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
          Caused by: java.io.IOException: Task process exit with nonzero status of 126.
          at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

          and error logs are cleaned, so It's very hard to debug.

          but I compared DefaultTaskController.java with 0.22, they use "bash command" to start the job scritp, but 1.0.4 use "bash, "-c", command".

          I removed "-c", everything is ok, 126 error code never happen again.

          I read man document of bash, it indicates when fork a new thread with write command, another thread with "bash -c" also has a writable fd. so I think it could return 126 status occasionally.

          So, there is only one line fix for this issue.
          Fengdong Yu made changes -
          Attachment MAPREDUCE-4857.patch [ 12559815 ]
          Fengdong Yu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 1.0.4 [ 12323325 ]
          Luke Lu made changes -
          Link This issue relates to MAPREDUCE-2374 [ MAPREDUCE-2374 ]
          story made changes -
          Assignee Bian Zhouyong [ story ]
          story made changes -
          Assignee Bian Zhouyong [ story ]
          Harsh J made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Target Version/s 1.0.4 [ 12323325 ]
          Fix Version/s 1.0.4 [ 12323325 ]
          Resolution Not a Problem [ 8 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Fengdong Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development