[IMPALA-3559] Explain plan and profiles reference HDFS while query is running entirely against S3

• Type: Task
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: Impala 2.6.0
• Component/s: Frontend
• Labels: s3 supportability
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Mostafa Mokhtar
• Created: 17/May/16 22:16
• Updated: 06/Mar/18 23:51
• Resolved: 06/Mar/18 23:51

Description

When running queries against databases entirely in S3 the query profile and plan still mention HDFS

WRITE TO HDFS [tpch_300_parquet_partitioned.lineitem, OVERWRITE=false, PARTITION-KEYS=(L_SHIPDATE)]
|  partitions=2526
|  hosts=32 per-host-mem=12.53GB
|
01:EXCHANGE [HASH(L_SHIPDATE)]
|  hosts=32 per-host-mem=0B
|  tuple-ids=0 row-size=256B cardinality=1682537668
|
00:SCAN HDFS [tpch_300_text_partitioned.lineitem, RANDOM]
   partitions=2526/2526 files=2526 size=210.05GB
   table stats: 1682537668 rows total (74 partition(s) missing stats)
   column stats: all
   hosts=32 per-host-mem=960.00MB
   tuple-ids=0 row-size=256B cardinality=1682537668

        - TotalIntegrityCheckTime: 0.000ns
         - TotalReadBlockTime: 0.000ns
      HdfsTableSink:(Total: 45s194ms, non-child: 45s194ms, % non-child: 100.00%)
         - BytesWritten: 231.00 B (231)
         - CompressTimer: 5s055ms
         - EncodeTimer: 25s434ms
         - FilesCreated: 57 (57)
         - HdfsWriteTimer: 0.000ns
         - PartitionsCreated: 57 (57)
         - PeakMemoryUsage: 1011.51 MB (1060647936)
         - RowsInserted: 0 (0)
         - TmpFileCreateTimer: 4s703ms

      HDFS_SCAN_NODE (id=0):(Total: 714.997ms, non-child: 714.997ms, % non-child: 100.00%)
         - AverageHdfsReadThreadConcurrency: 0.36 
         - AverageScannerThreadConcurrency: 10.88 
         - BytesRead: 2.43 GB (2611737608)
         - BytesReadDataNodeCache: 0
         - BytesReadLocal: 0
         - BytesReadRemoteUnexpected: 0
         - BytesReadShortCircuit: 0
         - DecompressionTime: 0.000ns
         - MaxCompressedTextFileLength: 0
         - NumDisksAccessed: 0 (0)
         - NumScannerThreadsStarted: 10 (10)
         - PeakMemoryUsage: 337.63 MB (354027648)
         - PerReadThreadRawHdfsThroughput: 36.42 MB/sec
         - RemoteScanRanges: 0 (0)
         - RowsRead: 19.48M (19483458)
         - RowsReturned: 19.43M (19431740)
         - RowsReturnedRate: 27.21 M/sec
         - ScanRangesComplete: 74 (74)
         - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
         - ScannerThreadsTotalWallClockTime: 0.000ns
           - DelimiterParseTime: 3s596ms
           - MaterializeTupleTime(*): 7s094ms
           - ScannerThreadsSysTime: 0.000ns
           - ScannerThreadsUserTime: 0.000ns
         - ScannerThreadsVoluntaryContextSwitches: 0 (0)
         - TotalRawHdfsReadTime(*): 1m8s
         - TotalReadThroughput: 13.26 MB/sec

Issue Links

• duplicates: IMPALA-6050 Query profiles should clearly indicate storage layer(s) used