Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Ubuntu 8.10 x86_64, lots of RAM and HDD spare, clean SVN_HEAD of trunk

      Description

      TestJobHistory.testJobHistoryUserLogLocation is failing, and there is an error in the log related to du failing in the mini MR cluster

        Issue Links

          Activity

          Hide
          Allen Wittenauer added a comment -

          Likely fixed forever ago.

          Show
          Allen Wittenauer added a comment - Likely fixed forever ago.
          Hide
          Hong Tang added a comment -

          Another reason we need HADOOP-4998?

          Show
          Hong Tang added a comment - Another reason we need HADOOP-4998 ?
          Hide
          Steve Loughran added a comment -

          as TestJobHistory is failing in trunk on hudson, without this du error appearing, it isn't the root cause of the test failing. However, an unexpected failure of du is odd. Downrating

          Show
          Steve Loughran added a comment - as TestJobHistory is failing in trunk on hudson, without this du error appearing, it isn't the root cause of the test failing. However, an unexpected failure of du is odd. Downrating
          Hide
          Steve Loughran added a comment -

          The same error string pops up in HADOOP-5496, where the datanode refuses to play. On this linux box the DNs do come up, yet the TT is unhappy.

          I think at the very least, the specific du command that failed should be printed out on an error, so we can see if it's the command at fault, or something else in the process.

          Show
          Steve Loughran added a comment - The same error string pops up in HADOOP-5496 , where the datanode refuses to play. On this linux box the DNs do come up, yet the TT is unhappy. I think at the very least, the specific du command that failed should be printed out on an error, so we can see if it's the command at fault, or something else in the process.
          Hide
          Steve Loughran added a comment -

          Trace of the output stream

          2009-05-19 16:38:50,362 INFO mapred.IndexCache (IndexCache.java:removeMap(140)) - Map ID attempt_200905191637_0003_m_000002_0 not found in cache
          2009-05-19 16:38:51,202 INFO mapred.JobHistory (JobHistory.java:getJobHistoryFileName(699)) - Recovered job history filename for job job_200905191637_0003 is localhost_1242747438419_job_200905191637_0003_slo_test-job-kill
          2009-05-19 16:38:51,236 INFO mapred.MiniMRCluster (MiniMRCluster.java:waitTaskTrackers(295)) - Waiting for task tracker tracker_host0.foo.com:localhost/127.0.0.1:46675 to be idle.
          2009-05-19 16:38:51,885 WARN util.ProcessTree (ProcessTree.java:sigKillInCurrentThread(91)) - Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process

          2009-05-19 16:38:51,885 INFO util.ProcessTree (ProcessTree.java:sigKillInCurrentThread(94)) - Killing process group15638 with SIGKILL. Exit code 1
          2009-05-19 16:38:51,885 INFO mapred.DefaultTaskController (DefaultTaskController.java:killTaskJVM(87)) - Process exited with exit code:0
          2009-05-19 16:38:52,236 INFO mapred.MiniMRCluster (MiniMRCluster.java:waitTaskTrackers(295)) - Waiting for task tracker tracker_host0.foo.com:localhost/127.0.0.1:46675 to be idle.
          2009-05-19 16:38:54,789 WARN util.ProcessTree (ProcessTree.java:sigKillInCurrentThread(91)) - Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process

          2009-05-19 16:38:54,790 INFO util.ProcessTree (ProcessTree.java:sigKillInCurrentThread(94)) - Killing process group15720 with SIGKILL. Exit code 1
          2009-05-19 16:38:54,790 INFO mapred.DefaultTaskController (DefaultTaskController.java:killTaskJVM(87)) - Process exited with exit code:0
          2009-05-19 16:38:54,790 INFO mapred.JvmManager (JvmManager.java:runChild(411)) - JVM : jvm_200905191637_0003_m_1766796043 exited. Number of tasks it ran: 1
          2009-05-19 16:38:55,211 WARN util.ProcessTree (ProcessTree.java:sigKillInCurrentThread(91)) - Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process

          2009-05-19 16:38:55,212 INFO util.ProcessTree (ProcessTree.java:sigKillInCurrentThread(94)) - Killing process group15689 with SIGKILL. Exit code 1
          2009-05-19 16:38:55,212 INFO mapred.DefaultTaskController (DefaultTaskController.java:killTaskJVM(87)) - Process exited with exit code:143
          2009-05-19 16:38:55,212 INFO mapred.TaskTracker (TaskTracker.java:addFreeSlot(1682)) - addFreeSlot : current free slots : 2
          2009-05-19 16:38:55,213 INFO mapred.JvmManager (JvmManager.java:runChild(411)) - JVM : jvm_200905191637_0003_m_-728591573 exited. Number of tasks it ran: 1
          2009-05-19 16:38:55,227 INFO mapred.JvmManager (JvmManager.java:runChild(411)) - JVM : jvm_200905191637_0003_m_-945732611 exited. Number of tasks it ran: 0
          2009-05-19 16:38:55,235 INFO mapred.TaskRunner (MapTaskRunner.java:close(43)) - attempt_200905191637_0003_m_000000_0 done; removing files.
          2009-05-19 16:38:55,237 INFO mapred.IndexCache (IndexCache.java:removeMap(140)) - Map ID attempt_200905191637_0003_m_000000_0 not found in cache
          2009-05-19 16:38:55,239 INFO mapred.TaskTracker (TaskTracker.java:run(659)) - Shutting down: Map-events fetcher for all reduce tasks on tracker_host0.foo.com:localhost/127.0.0.1:46675
          2009-05-19 16:38:55,239 INFO ipc.Server (Server.java:stop(1103)) - Stopping server on 46675
          2009-05-19 16:38:55,240 INFO ipc.Server (Server.java:run(997)) - IPC Server handler 0 on 46675: exiting
          2009-05-19 16:38:55,240 INFO ipc.Server (Server.java:run(997)) - IPC Server handler 1 on 46675: exiting
          2009-05-19 16:38:55,240 INFO ipc.Server (Server.java:run(997)) - IPC Server handler 2 on 46675: exiting
          2009-05-19 16:38:55,240 INFO ipc.Server (Server.java:run(997)) - IPC Server handler 3 on 46675: exiting
          2009-05-19 16:38:55,241 INFO ipc.Server (Server.java:run(539)) - Stopping IPC Server Responder
          2009-05-19 16:38:55,241 INFO mapred.TaskTracker (TaskTracker.java:shutdown(891)) - Shutting down StatusHttpServer
          2009-05-19 16:38:55,241 INFO ipc.Server (Server.java:run(352)) - Stopping IPC Server listener on 46675
          2009-05-19 16:38:55,258 ERROR mapred.TaskTracker (TaskTracker.java:offerService(1187)) - Caught exception: java.io.IOException: Expecting a line not the end of stream
          at org.apache.hadoop.fs.DF.parseExecResult(DF.java:147)
          at org.apache.hadoop.util.Shell.runCommand(Shell.java:179)
          at org.apache.hadoop.util.Shell.run(Shell.java:134)
          at org.apache.hadoop.fs.DF.getAvailable(DF.java:109)
          at org.apache.hadoop.mapred.TaskTracker.getFreeSpace(TaskTracker.java:1588)
          at org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1247)
          at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1085)
          at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1808)
          at org.apache.hadoop.mapred.MiniMRCluster$TaskTrackerRunner.run(MiniMRCluster.java:193)
          at java.lang.Thread.run(Thread.java:619)

          Show
          Steve Loughran added a comment - Trace of the output stream 2009-05-19 16:38:50,362 INFO mapred.IndexCache (IndexCache.java:removeMap(140)) - Map ID attempt_200905191637_0003_m_000002_0 not found in cache 2009-05-19 16:38:51,202 INFO mapred.JobHistory (JobHistory.java:getJobHistoryFileName(699)) - Recovered job history filename for job job_200905191637_0003 is localhost_1242747438419_job_200905191637_0003_slo_test-job-kill 2009-05-19 16:38:51,236 INFO mapred.MiniMRCluster (MiniMRCluster.java:waitTaskTrackers(295)) - Waiting for task tracker tracker_host0.foo.com:localhost/127.0.0.1:46675 to be idle. 2009-05-19 16:38:51,885 WARN util.ProcessTree (ProcessTree.java:sigKillInCurrentThread(91)) - Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process 2009-05-19 16:38:51,885 INFO util.ProcessTree (ProcessTree.java:sigKillInCurrentThread(94)) - Killing process group15638 with SIGKILL. Exit code 1 2009-05-19 16:38:51,885 INFO mapred.DefaultTaskController (DefaultTaskController.java:killTaskJVM(87)) - Process exited with exit code:0 2009-05-19 16:38:52,236 INFO mapred.MiniMRCluster (MiniMRCluster.java:waitTaskTrackers(295)) - Waiting for task tracker tracker_host0.foo.com:localhost/127.0.0.1:46675 to be idle. 2009-05-19 16:38:54,789 WARN util.ProcessTree (ProcessTree.java:sigKillInCurrentThread(91)) - Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process 2009-05-19 16:38:54,790 INFO util.ProcessTree (ProcessTree.java:sigKillInCurrentThread(94)) - Killing process group15720 with SIGKILL. Exit code 1 2009-05-19 16:38:54,790 INFO mapred.DefaultTaskController (DefaultTaskController.java:killTaskJVM(87)) - Process exited with exit code:0 2009-05-19 16:38:54,790 INFO mapred.JvmManager (JvmManager.java:runChild(411)) - JVM : jvm_200905191637_0003_m_1766796043 exited. Number of tasks it ran: 1 2009-05-19 16:38:55,211 WARN util.ProcessTree (ProcessTree.java:sigKillInCurrentThread(91)) - Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process 2009-05-19 16:38:55,212 INFO util.ProcessTree (ProcessTree.java:sigKillInCurrentThread(94)) - Killing process group15689 with SIGKILL. Exit code 1 2009-05-19 16:38:55,212 INFO mapred.DefaultTaskController (DefaultTaskController.java:killTaskJVM(87)) - Process exited with exit code:143 2009-05-19 16:38:55,212 INFO mapred.TaskTracker (TaskTracker.java:addFreeSlot(1682)) - addFreeSlot : current free slots : 2 2009-05-19 16:38:55,213 INFO mapred.JvmManager (JvmManager.java:runChild(411)) - JVM : jvm_200905191637_0003_m_-728591573 exited. Number of tasks it ran: 1 2009-05-19 16:38:55,227 INFO mapred.JvmManager (JvmManager.java:runChild(411)) - JVM : jvm_200905191637_0003_m_-945732611 exited. Number of tasks it ran: 0 2009-05-19 16:38:55,235 INFO mapred.TaskRunner (MapTaskRunner.java:close(43)) - attempt_200905191637_0003_m_000000_0 done; removing files. 2009-05-19 16:38:55,237 INFO mapred.IndexCache (IndexCache.java:removeMap(140)) - Map ID attempt_200905191637_0003_m_000000_0 not found in cache 2009-05-19 16:38:55,239 INFO mapred.TaskTracker (TaskTracker.java:run(659)) - Shutting down: Map-events fetcher for all reduce tasks on tracker_host0.foo.com:localhost/127.0.0.1:46675 2009-05-19 16:38:55,239 INFO ipc.Server (Server.java:stop(1103)) - Stopping server on 46675 2009-05-19 16:38:55,240 INFO ipc.Server (Server.java:run(997)) - IPC Server handler 0 on 46675: exiting 2009-05-19 16:38:55,240 INFO ipc.Server (Server.java:run(997)) - IPC Server handler 1 on 46675: exiting 2009-05-19 16:38:55,240 INFO ipc.Server (Server.java:run(997)) - IPC Server handler 2 on 46675: exiting 2009-05-19 16:38:55,240 INFO ipc.Server (Server.java:run(997)) - IPC Server handler 3 on 46675: exiting 2009-05-19 16:38:55,241 INFO ipc.Server (Server.java:run(539)) - Stopping IPC Server Responder 2009-05-19 16:38:55,241 INFO mapred.TaskTracker (TaskTracker.java:shutdown(891)) - Shutting down StatusHttpServer 2009-05-19 16:38:55,241 INFO ipc.Server (Server.java:run(352)) - Stopping IPC Server listener on 46675 2009-05-19 16:38:55,258 ERROR mapred.TaskTracker (TaskTracker.java:offerService(1187)) - Caught exception: java.io.IOException: Expecting a line not the end of stream at org.apache.hadoop.fs.DF.parseExecResult(DF.java:147) at org.apache.hadoop.util.Shell.runCommand(Shell.java:179) at org.apache.hadoop.util.Shell.run(Shell.java:134) at org.apache.hadoop.fs.DF.getAvailable(DF.java:109) at org.apache.hadoop.mapred.TaskTracker.getFreeSpace(TaskTracker.java:1588) at org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1247) at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1085) at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1808) at org.apache.hadoop.mapred.MiniMRCluster$TaskTrackerRunner.run(MiniMRCluster.java:193) at java.lang.Thread.run(Thread.java:619)

            People

            • Assignee:
              Unassigned
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development