Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1440

Parquet writers seem to ignore process mem limit

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Cannot Reproduce
    • Affects Version/s: Impala 2.0, Impala 2.3.0
    • Fix Version/s: None
    • Component/s: Backend

      Description

      Running a simple insert overwrite from a partitioned table into a partitioned table blows past the memory limit, although the query is eventually cancelled with OOM.

      The process limit was set to 50GB. Below, the table writer fragment uses as much as 66GB before being killed, and it seems that, after it exceeded the 50GB limit, it kept allocating.

      The memory usage also seems excessive given the number of bytes written, but that's perhaps a different issue.

      Plan

      F01:PLAN FRAGMENT [HASH(<redacted>)]
        WRITE TO HDFS [<redacted>, OVERWRITE=true, PARTITION-KEYS=(<redacted>, <redacted>)]
        |  partitions=3857
        |  hosts=10 per-host-mem=26.00GB
        |
        01:EXCHANGE [HASH(<redacted>, <redacted>)]
           hosts=10 per-host-mem=0B
           tuple-ids=0 row-size=68B cardinality=unavailable
      
      F00:PLAN FRAGMENT [RANDOM]
        DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=01, HASH(<redacted>, <redacted>)]
        00:SCAN HDFS [<redacted>, RANDOM]
           partitions=1320/3829 size=106.42GB
           table stats: unavailable
           hosts=10 per-host-mem=352.00MB
           tuple-ids=0 row-size=68B cardinality=unavailable
      

      Profile excerpt

      Fragment F01:
            Instance 654f7a1e99a7f447:9c1d7b6133151e86 (host=e1518.halxg.cloudera.com:22000):(Total: 8m53s, non-child: 0ns, % non-child: 0.00%)
              MemoryUsage(16s000ms): 1.72 GB, 8.57 GB, 17.79 GB, 24.84 GB, 28.88 GB, 32.23 GB, 35.81 GB, 39.49 GB, 42.03 GB, 43.78 GB, 44.20 GB, 44.95 GB, 46.02 GB, 46.43 GB, 46.85 GB, 47.98 GB, 49.07 GB, 50.17 GB, 51.57 GB, 52.65 GB, 53.22 GB, 54.06 GB, 54.06 GB, 55.11 GB, 57.09 GB, 59.23 GB, 60.24 GB, 60.98 GB, 62.02 GB, 63.16 GB, 63.50 GB, 65.27 GB, 66.66 GB
              ThreadUsage(16s000ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
               - AverageThreadTokens: 1.00 
               - PeakMemoryUsage: 67.02 GB (71958395432)
               - PerHostPeakMemUsage: 67.02 GB (71958395432)
               - PrepareTime: 38.653us
               - RowsProduced: 27.74K (27743)
               - TotalCpuTime: 8m23s
               - TotalNetworkReceiveTime: 31s057ms
               - TotalNetworkSendTime: 0ns
               - TotalStorageWaitTime: 0ns
              BlockMgr:
                 - BlockWritesOutstanding: 0
                 - BlocksCreated: 0
                 - BlocksRecycled: 0
                 - BufferedPins: 0
                 - BytesWritten: 0
                 - MaxBlockSize: 8.00 MB (8388608)
                 - MemoryLimit: 40.24 GB (43209306112)
                 - PeakMemoryUsage: 0
                 - TotalBufferWaitTime: 0ns
                 - TotalEncryptionTime: 0ns
                 - TotalIntegrityCheckTime: 0ns
                 - TotalReadBlockTime: 0ns
              HdfsTableSink:(Total: 8m23s, non-child: 8m23s, % non-child: 100.00%)
                 - BytesWritten: 11.16 GB (11981549149)
                 - CompressTimer: 44s501ms
                 - EncodeTimer: 12s836ms
                 - FinalizePartitionFileTimer: 8m2s
                 - HdfsWriteTimer: 6m19s
                 - PeakMemoryUsage: 66.93 GB (71862789984)
                 - RowsInserted: 27.74K (27743)
                 - TmpFileCreateTimer: 6s287ms
      

        Attachments

          Activity

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              henryr Henry Robinson
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: