[IMPALA-1440] Parquet writers seem to ignore process mem limit

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 2.0, Impala 2.3.0
• Component/s: Backend
• Labels: resource-management
• Target Version: Product Backlog
• Assignee: Tim Armstrong
• Reporter: Henry Robinson
• Created: 31/Oct/14 18:30
• Updated: 01/Jun/16 06:01
• Resolved: 01/Jun/16 06:01

Description

Running a simple insert overwrite from a partitioned table into a partitioned table blows past the memory limit, although the query is eventually cancelled with OOM.

The process limit was set to 50GB. Below, the table writer fragment uses as much as 66GB before being killed, and it seems that, after it exceeded the 50GB limit, it kept allocating.

The memory usage also seems excessive given the number of bytes written, but that's perhaps a different issue.

Plan

F01:PLAN FRAGMENT [HASH(<redacted>)]
  WRITE TO HDFS [<redacted>, OVERWRITE=true, PARTITION-KEYS=(<redacted>, <redacted>)]
  |  partitions=3857
  |  hosts=10 per-host-mem=26.00GB
  |
  01:EXCHANGE [HASH(<redacted>, <redacted>)]
     hosts=10 per-host-mem=0B
     tuple-ids=0 row-size=68B cardinality=unavailable

F00:PLAN FRAGMENT [RANDOM]
  DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=01, HASH(<redacted>, <redacted>)]
  00:SCAN HDFS [<redacted>, RANDOM]
     partitions=1320/3829 size=106.42GB
     table stats: unavailable
     hosts=10 per-host-mem=352.00MB
     tuple-ids=0 row-size=68B cardinality=unavailable

Profile excerpt

Fragment F01:
      Instance 654f7a1e99a7f447:9c1d7b6133151e86 (host=e1518.halxg.cloudera.com:22000):(Total: 8m53s, non-child: 0ns, % non-child: 0.00%)
        MemoryUsage(16s000ms): 1.72 GB, 8.57 GB, 17.79 GB, 24.84 GB, 28.88 GB, 32.23 GB, 35.81 GB, 39.49 GB, 42.03 GB, 43.78 GB, 44.20 GB, 44.95 GB, 46.02 GB, 46.43 GB, 46.85 GB, 47.98 GB, 49.07 GB, 50.17 GB, 51.57 GB, 52.65 GB, 53.22 GB, 54.06 GB, 54.06 GB, 55.11 GB, 57.09 GB, 59.23 GB, 60.24 GB, 60.98 GB, 62.02 GB, 63.16 GB, 63.50 GB, 65.27 GB, 66.66 GB
        ThreadUsage(16s000ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.00 
         - PeakMemoryUsage: 67.02 GB (71958395432)
         - PerHostPeakMemUsage: 67.02 GB (71958395432)
         - PrepareTime: 38.653us
         - RowsProduced: 27.74K (27743)
         - TotalCpuTime: 8m23s
         - TotalNetworkReceiveTime: 31s057ms
         - TotalNetworkSendTime: 0ns
         - TotalStorageWaitTime: 0ns
        BlockMgr:
           - BlockWritesOutstanding: 0
           - BlocksCreated: 0
           - BlocksRecycled: 0
           - BufferedPins: 0
           - BytesWritten: 0
           - MaxBlockSize: 8.00 MB (8388608)
           - MemoryLimit: 40.24 GB (43209306112)
           - PeakMemoryUsage: 0
           - TotalBufferWaitTime: 0ns
           - TotalEncryptionTime: 0ns
           - TotalIntegrityCheckTime: 0ns
           - TotalReadBlockTime: 0ns
        HdfsTableSink:(Total: 8m23s, non-child: 8m23s, % non-child: 100.00%)
           - BytesWritten: 11.16 GB (11981549149)
           - CompressTimer: 44s501ms
           - EncodeTimer: 12s836ms
           - FinalizePartitionFileTimer: 8m2s
           - HdfsWriteTimer: 6m19s
           - PeakMemoryUsage: 66.93 GB (71862789984)
           - RowsInserted: 27.74K (27743)
           - TmpFileCreateTimer: 6s287ms