[HADOOP-556] Streaming should send keep-alive signals to Reporter every 10 seconds, not every 100 records

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Assignee: Unassigned
• Reporter: Michel Tourn
• Created: 23/Sep/06 00:27
• Updated: 08/Jul/09 17:05
• Resolved: 25/Sep/06 22:46

Description

Streaming should send keep-alive signals to Reporter every 10 seconds, not every 100 records.
A first version of this was already implemented but was not satisfactory.

Now we check whether 10sec have passed:

-after reading a line of Application's stderr
(so please use as your stderr heartbeat: cerr << ".\n", not cerr << "." )

-and after outputting a record in mapper / combiner / reducer.

If 10 sec have passed then we set Reporter status.

Effects:

-the reporter status changes more often and provides useful feedback in the Web UI or in another client.
-a Task will not time out after 10 minutes just because it outputs records slowly.

No artificial heartbeat is introduced in this proposal.
The streaming Application still has to show activity (either on stdout or on stderr)