[DRILL-1990] Add peak memory allocation in a operator to OperatorStats.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Component/s: Execution - Relational Operators
• Assignee: Venki Korukanti
• Reporter: Venki Korukanti
• Created: 13/Jan/15 01:34
• Updated: 16/Jan/15 23:18
• Resolved: 16/Jan/15 23:18

Description

Currently we have "localMemoryAllocated" which is always set to zero as we try to fill the stats at the end of fragment execution by calling allocator.getAllocatedMemory() which at that point has already released the allocated memory. Instead if we have a stat for peak memory an allocator has seen in the lifetime of the operator execution will be useful (each operator has its own allocator).

Example query on query profile: To find aggregate of peak memory of each operator across all minor fragments in a major fragment and list them in descending order of peak memory usage

SELECT
  majorFragmentId,
  opProfile['operatorType'] opType,
  sum(opProfile['peakLocalMemoryAllocated']) aggPeakMemoryAcrossAllMinorFragments 
FROM 
  (SELECT
       majorFragmentId,
       flatten(minorFragProfile['operatorProfile']) opProfile
   FROM 
       (SELECT
             majorFragment['majorFragmentId'] majorFragmentId, 
             flatten(majorFragment['minorFragmentProfile']) minorFragProfile
         FROM
             (SELECT flatten(fragmentProfile) as majorFragment from dfs.`/tmp/a.json`)
         )
   )
-- WHERE opProfile['operatorType'] = 6 -- If want to filter to particular operator
GROUP BY 
  majorFragmentId,
  opProfile['operatorType']
ORDER BY
  aggPeakMemoryAcrossAllMinorFragments DESC;

+-----------------+------------+--------------------------------------+
| majorFragmentId |   opType   | aggPeakMemoryAcrossAllMinorFragments |
+-----------------+------------+--------------------------------------+
| 1               | 4          | 115065856                            |
| 1               | 3          | 10027008                             |
| 0               | 3          | 1671168                              |
| 3               | 6          | 1536000                              |
| 2               | 6          | 901120                               |
| 1               | 6          | 606208                               |
| 3               | 28         | 393216                               |
| 2               | 28         | 229376                               |
| 3               | 10         | 122880                               |
| 2               | 10         | 81920                                |
| 0               | 11         | 0                                    |
| 0               | 10         | 0                                    |
| 0               | 13         | 0                                    |
| 1               | 10         | 0                                    |
| 1               | 11         | 0                                    |
+-----------------+------------+--------------------------------------+