[HIVE-20512] Improve record and memory usage logging in SparkRecordHandler

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Spark
• Assignee: Bharath Krishna
• Reporter: Sahil Takiar
• Created: 06/Sep/18 19:16
• Updated: 17/Nov/22 08:55
• Resolved: 14/Nov/18 02:12

Description

We currently log memory usage and # of records processed in Spark tasks, but we should improve the methodology for how frequently we log this info. Currently we use the following code:

private long getNextLogThreshold(long currentThreshold) {
    // A very simple counter to keep track of number of rows processed by the
    // reducer. It dumps
    // every 1 million times, and quickly before that
    if (currentThreshold >= 1000000) {
      return currentThreshold + 1000000;
    }
    return 10 * currentThreshold;
  }

The issue is that after a while, the increase by 10x factor means that you have to process a huge # of records before this gets triggered.

A better approach would be to log this info at a given interval. This would help in debugging tasks that are seemingly hung.

Issue Links

• links to: Review