[IMPALA-9819] Separate data cache and HDFS scan node runtime profile metrics

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Backend
• Labels: observability
• Epic Color: ghx-label-13
• Assignee: Joe McDonnell
• Reporter: Sahil Takiar
• Created: 02/Jun/20 17:10
• Updated: 21/Dec/20 19:50

Description

When a query reads data from both a remote storage system (e.g. S3) and the data cache, the HDFS_SCAN_NODE runtime profiles are hard to reason about.

For example, in the following runtime profile snippet:

HDFS_SCAN_NODE (id=0):(Total: 59s374ms, non-child: 0.000ns, % non-child: 0.00%)
         - AverageHdfsReadThreadConcurrency: 0.62 
         - AverageScannerThreadConcurrency: 0.91 
         - BytesRead: 587.97 MB (616533483)
         - BytesReadDataNodeCache: 0
         - BytesReadLocal: 0
         - BytesReadRemoteUnexpected: 0
         - BytesReadShortCircuit: 0
         - CachedFileHandlesHitCount: 323 (323)
         - CachedFileHandlesMissCount: 94 (94)
         - CollectionItemsRead: 0 (0)
         - DataCacheHitBytes: 212.00 MB (222294996)
         - DataCacheHitCount: 107 (107)
         - DataCacheMissBytes: 375.98 MB (394238486)
         - DataCacheMissCount: 310 (310)
         - DataCachePartialHitCount: 0 (0)
         - DecompressionTime: 2s428ms
         - MaterializeTupleTime: 19s444ms
         - MaxCompressedTextFileLength: 0
         - NumColumns: 3 (3)
         - NumDictFilteredRowGroups: 0 (0)
         - NumDisksAccessed: 1 (1)
         - NumPages: 53.30K (53300)
         - NumRowGroups: 83 (83)
         - NumRowGroupsWithPageIndex: 83 (83)
         - NumScannerThreadMemUnavailable: 0 (0)
         - NumScannerThreadReservationsDenied: 0 (0)
         - NumScannerThreadsStarted: 1 (1)
         - NumScannersWithNoReads: 0 (0)
         - NumStatsFilteredPages: 0 (0)
         - NumStatsFilteredRowGroups: 0 (0)
         - PeakMemoryUsage: 16.00 MB (16781312)
         - PeakScannerThreadConcurrency: 1 (1)
         - PerReadThreadRawHdfsThroughput: 15.11 MB/sec
         - RemoteScanRanges: 0 (0)
         - RowBatchBytesEnqueued: 670.68 MB (703260541)
         - RowBatchQueueGetWaitTime: 59s368ms
         - RowBatchQueuePeakMemoryUsage: 4.17 MB (4368285)
         - RowBatchQueuePutWaitTime: 0.000ns
         - RowBatchesEnqueued: 915 (915)
         - RowsRead: 413.47M (413466507)
         - RowsReturned: 722.27K (722275)
         - RowsReturnedRate: 12.17 K/sec
         - ScanRangesComplete: 83 (83)
         - ScannerIoWaitTime: 33s454ms
         - ScannerThreadWorklessLoops: 0 (0)
         - ScannerThreadsInvoluntaryContextSwitches: 1.94K (1940)
         - ScannerThreadsTotalWallClockTime: 1m
           - ScannerThreadsSysTime: 1s181ms
           - ScannerThreadsUserTime: 20s581ms
         - ScannerThreadsVoluntaryContextSwitches: 770 (770)
         - TotalRawHdfsOpenFileTime: 3s396ms
         - TotalRawHdfsReadTime: 38s940ms
         - TotalReadThroughput: 8.86 MB/sec 

The query scanned part of the data from S3 and part of the data from the data cache.

The confusing part is that metrics such as PerReadThreadRawHdfsThroughput are measured across S3 and data cache reads. So there is no straightforward way to determine the throughput for just S3 reads. Users might want this value to determine if S3 was particularly slow for their query.

It would be nice if the scan node metrics more clearly differentiate between reads from S3 vs. the data cache. The aggregate metrics (Total metrics) are still useful, but it would be useful to have fine-grained metrics that are specific to a data storage system (e.g. either the data cache or S3).

Issue Links

• relates to: IMPALA-8341 Data cache for remote reads