[IMPALA-4392] Add HdfsTableSink memory consumption to the query profile

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.5.0
• Fix Version/s: Impala 2.8.0
• Component/s: Backend
• Labels: memory ramp-up supportability
• Target Version: Impala 2.8.0
• Assignee: Tim Armstrong
• Reporter: Mostafa Mokhtar
• Created: 28/Oct/16 20:17
• Updated: 17/Nov/16 10:59
• Resolved: 17/Nov/16 10:59

Description

Memory consumed by HdfsTableSink is tracked by not captured in the query profile, makes it look as if there is untracked memory.

from the Memz tab

Process: Limit=201.73 GB Total=4.78 GB Peak=8.66 GB
  Free Disk IO Buffers: Total=1.40 GB Peak=1.84 GB
  RequestPool=root.mmokhtar: Total=2.90 GB Peak=7.80 GB
    Query(aa40c1753a40a8be:9c27185a00000000): Total=2.90 GB Peak=3.20 GB
      Fragment aa40c1753a40a8be:9c27185a00000006: Total=2.90 GB Peak=3.20 GB
        HDFS_SCAN_NODE (id=0): Total=1.51 GB Peak=1.97 GB
        HdfsTableSink: Total=1.38 GB Peak=1.38 GB
          HdfsTableSink Exprs: Total=124.02 MB Peak=124.02 MB
      Block Manager: Limit=161.38 GB Total=0 Peak=0
  RequestPool=root.jenkins: Total=0 Peak=284.69 MB

From query profile

        HdfsTableSink:(Total: 9m47s, non-child: 9m47s, % non-child: 100.00%)
           - BytesWritten: 6.79 GB (7290559999)
           - CompressTimer: 58s128ms
           - EncodeTimer: 8m6s
           - FilesCreated: 69 (69)
           - FinalizePartitionFileTimer: 1m18s
           - HdfsWriteTimer: 1m16s
           - PartitionsCreated: 1 (1)
           - RowsInserted: 426.90K (426901)
           - TmpFileCreateTimer: 1s253ms