---------------------------------------------------------------------------------------------- VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED ---------------------------------------------------------------------------------------------- Map 1 .......... llap SUCCEEDED 1 1 0 0 0 0 Map 5 .......... llap SUCCEEDED 10 10 0 0 0 0 Map 2 .......... llap SUCCEEDED 22 22 0 0 0 0 Map 8 .......... llap SUCCEEDED 4 4 0 0 0 0 Map 6 .......... llap SUCCEEDED 23 23 0 0 0 0 Reducer 7 ...... llap SUCCEEDED 4 4 0 0 0 0 Map 11 ......... llap SUCCEEDED 6 6 0 0 0 0 Map 9 .......... llap SUCCEEDED 22 22 0 0 0 0 Reducer 10 ..... llap SUCCEEDED 2 2 0 0 0 0 Reducer 3 ...... llap SUCCEEDED 1 1 0 0 0 0 Reducer 4 ...... llap SUCCEEDED 1 1 0 0 0 0 ---------------------------------------------------------------------------------------------- VERTICES: 11/11 [==========================>>] 100% ELAPSED TIME: 101.32 s ---------------------------------------------------------------------------------------------- INFO : Status: DAG finished successfully in 101.08 seconds INFO : DAG ID: dag_1697784846226_0000_1 INFO : INFO : Query Execution Summary INFO : ---------------------------------------------------------------------------------------------- INFO : OPERATION DURATION INFO : ---------------------------------------------------------------------------------------------- INFO : Compile Query 1.81s INFO : Prepare Plan 0.46s INFO : Get Query Coordinator (AM) 0.74s INFO : Submit Plan 0.17s INFO : Start DAG 0.36s INFO : Run DAG 101.08s INFO : ---------------------------------------------------------------------------------------------- INFO : INFO : Task Execution Summary INFO : ---------------------------------------------------------------------------------------------- INFO : VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS OUTPUT_RECORDS INFO : ---------------------------------------------------------------------------------------------- INFO : Map 1 4682.00 0 0 10,000 1,095 INFO : Map 11 9883.00 0 0 1,439,749 1,439,749 INFO : Map 2 62901.00 0 0 31,677,518 4,922,835 INFO : Map 5 8325.00 0 0 2,875,432 2,875,432 INFO : Map 6 12966.00 0 0 7,917,878 1,289,751 INFO : Map 8 6244.00 0 0 719,217 719,217 INFO : Map 9 45005.00 0 0 15,842,105 2,561,824 INFO : Reducer 10 14954.00 0 0 2,561,824 2,561,792 INFO : Reducer 3 13390.00 0 0 8,774,332 100 INFO : Reducer 4 153.00 0 0 100 0 INFO : Reducer 7 25255.00 0 0 1,289,751 1,289,705 INFO : ---------------------------------------------------------------------------------------------- INFO : INFO : LLAP IO Summary INFO : ---------------------------------------------------------------------------------------------- INFO : VERTICES ROWGROUPS META_HIT META_MISS DATA_HIT DATA_MISS TOTAL_IO INFO : ---------------------------------------------------------------------------------------------- INFO : Map 1 1 0 2 0B 970B 0.41s INFO : Map 11 149 0 34 0B 5.10MB 9.82s INFO : Map 2 3001 0 360 0B 235.10MB 432.81s INFO : Map 5 300 0 40 0B 10.34MB 11.02s INFO : Map 6 736 0 180 0B 58.95MB 56.00s INFO : Map 8 73 0 12 0B 2.58MB 5.07s INFO : Map 9 1513 0 240 0B 116.31MB 217.49s INFO : ---------------------------------------------------------------------------------------------- INFO : INFO : FileSystem Counters Summary INFO : INFO : Scheme: FILE INFO : ---------------------------------------------------------------------------------------------- INFO : VERTICES BYTES_READ READ_OPS LARGE_READ_OPS BYTES_WRITTEN WRITE_OPS INFO : ---------------------------------------------------------------------------------------------- INFO : Map 1 0B 0 0 4.86KB 0 INFO : Map 11 0B 0 0 13.25MB 0 INFO : Map 2 0B 0 0 89.17MB 0 INFO : Map 5 0B 0 0 25.79MB 0 INFO : Map 6 0B 0 0 22.45MB 0 INFO : Map 8 0B 0 0 6.73MB 0 INFO : Map 9 0B 0 0 43.37MB 0 INFO : Reducer 10 41.01MB 0 0 42.14MB 0 INFO : Reducer 3 71.08MB 0 0 4.38KB 0 INFO : Reducer 4 0B 0 0 5.33KB 0 INFO : Reducer 7 11.34MB 0 0 21.84MB 0 INFO : ---------------------------------------------------------------------------------------------- INFO : INFO : Scheme: S3A INFO : ---------------------------------------------------------------------------------------------- INFO : VERTICES BYTES_READ READ_OPS LARGE_READ_OPS BYTES_WRITTEN WRITE_OPS INFO : ---------------------------------------------------------------------------------------------- INFO : Map 1 68.53KB 4 0 0B 0 INFO : Map 11 5.54MB 68 0 0B 0 INFO : Map 2 238.48MB 480 0 0B 0 INFO : Map 5 10.71MB 80 0 0B 0 INFO : Map 6 60.68MB 360 0 0B 0 INFO : Map 8 2.69MB 24 0 0B 0 INFO : Map 9 120.03MB 480 0 0B 0 INFO : Reducer 10 0B 0 0 0B 0 INFO : Reducer 3 0B 0 0 0B 0 INFO : Reducer 4 0B 0 0 0B 0 INFO : Reducer 7 0B 0 0 0B 0 INFO : ---------------------------------------------------------------------------------------------- INFO : INFO : org.apache.tez.common.counters.DAGCounter: INFO : NUM_SUCCEEDED_TASKS: 96 INFO : TOTAL_LAUNCHED_TASKS: 96 INFO : DATA_LOCAL_TASKS: 88 INFO : AM_CPU_MILLISECONDS: 22290 INFO : AM_GC_TIME_MILLIS: 46 INFO : INITIAL_HELD_CONTAINERS: 0 INFO : TOTAL_CONTAINERS_USED: 96 INFO : TOTAL_CONTAINER_LAUNCH_COUNT: 96 INFO : TOTAL_CONTAINER_RELEASE_COUNT: 96 INFO : File System Counters: INFO : FILE_BYTES_READ: 123426528 INFO : FILE_BYTES_WRITTEN: 264749828 INFO : FILE_READ_OPS: 0 INFO : FILE_LARGE_READ_OPS: 0 INFO : FILE_WRITE_OPS: 0 INFO : S3A_BYTES_READ: 438200740 INFO : S3A_BYTES_WRITTEN: 0 INFO : S3A_READ_OPS: 1496 INFO : S3A_LARGE_READ_OPS: 0 INFO : S3A_WRITE_OPS: 0 INFO : org.apache.tez.common.counters.TaskCounter: INFO : REDUCE_INPUT_GROUPS: 8705203 INFO : REDUCE_INPUT_RECORDS: 8774510 INFO : COMBINE_INPUT_RECORDS: 0 INFO : SPILLED_RECORDS: 20460399 INFO : NUM_SHUFFLED_INPUTS: 404 INFO : NUM_SKIPPED_INPUTS: 24 INFO : NUM_FAILED_SHUFFLE_INPUTS: 0 INFO : MERGED_MAP_OUTPUTS: 30150306 INFO : TASK_DURATION_MILLIS: 1734503 INFO : INPUT_RECORDS_PROCESSED: 8941879 INFO : INPUT_SPLIT_LENGTH_BYTES: 2255775399 INFO : OUTPUT_RECORDS: 17661500 INFO : APPROXIMATE_INPUT_RECORDS: 8886255 INFO : OUTPUT_LARGE_RECORDS: 0 INFO : OUTPUT_BYTES: 449754929 INFO : OUTPUT_BYTES_WITH_OVERHEAD: 485080365 INFO : OUTPUT_BYTES_PHYSICAL: 264734098 INFO : ADDITIONAL_SPILLS_BYTES_WRITTEN: 26813280 INFO : ADDITIONAL_SPILLS_BYTES_READ: 172077930 INFO : ADDITIONAL_SPILL_COUNT: 0 INFO : SHUFFLE_CHUNK_COUNT: 68 INFO : SHUFFLE_BYTES: 264730226 INFO : SHUFFLE_BYTES_DECOMPRESSED: 485072323 INFO : SHUFFLE_BYTES_TO_MEM: 0 INFO : SHUFFLE_BYTES_TO_DISK: 0 INFO : SHUFFLE_BYTES_DISK_DIRECT: 264730226 INFO : NUM_MEM_TO_DISK_MERGES: 0 INFO : NUM_DISK_TO_DISK_MERGES: 1 INFO : SHUFFLE_PHASE_TIME: 2173 INFO : MERGE_PHASE_TIME: 3892 INFO : FIRST_EVENT_RECEIVED: 198 INFO : LAST_EVENT_RECEIVED: 1807 INFO : DATA_BYTES_VIA_EVENT: 0 INFO : HIVE: INFO : CREATED_FILES: 1 INFO : DESERIALIZE_ERRORS: 0 INFO : RECORDS_IN_Map_1: 10000 INFO : RECORDS_IN_Map_11: 1439749 INFO : RECORDS_IN_Map_2: 28800991 INFO : RECORDS_IN_Map_5: 2875432 INFO : RECORDS_IN_Map_6: 7197566 INFO : RECORDS_IN_Map_8: 719217 INFO : RECORDS_IN_Map_9: 14401261 INFO : RECORDS_OUT_0: 100 INFO : RECORDS_OUT_INTERMEDIATE_Map_1: 1095 INFO : RECORDS_OUT_INTERMEDIATE_Map_11: 1439749 INFO : RECORDS_OUT_INTERMEDIATE_Map_2: 4922835 INFO : RECORDS_OUT_INTERMEDIATE_Map_5: 2875432 INFO : RECORDS_OUT_INTERMEDIATE_Map_6: 1289751 INFO : RECORDS_OUT_INTERMEDIATE_Map_8: 719217 INFO : RECORDS_OUT_INTERMEDIATE_Map_9: 2561824 INFO : RECORDS_OUT_INTERMEDIATE_Reducer_10: 2561792 INFO : RECORDS_OUT_INTERMEDIATE_Reducer_3: 100 INFO : RECORDS_OUT_INTERMEDIATE_Reducer_4: 0 INFO : RECORDS_OUT_INTERMEDIATE_Reducer_7: 1289705 INFO : RECORDS_OUT_OPERATOR_FIL_210: 365 INFO : RECORDS_OUT_OPERATOR_FIL_215: 2875432 INFO : RECORDS_OUT_OPERATOR_FIL_219: 27504814 INFO : RECORDS_OUT_OPERATOR_FIL_226: 719217 INFO : RECORDS_OUT_OPERATOR_FIL_230: 7194926 INFO : RECORDS_OUT_OPERATOR_FIL_240: 1439749 INFO : RECORDS_OUT_OPERATOR_FIL_244: 14293543 INFO : RECORDS_OUT_OPERATOR_FIL_258: 729 INFO : RECORDS_OUT_OPERATOR_FS_264: 100 INFO : RECORDS_OUT_OPERATOR_GBY_217: 2875432 INFO : RECORDS_OUT_OPERATOR_GBY_224: 4922835 INFO : RECORDS_OUT_OPERATOR_GBY_228: 719217 INFO : RECORDS_OUT_OPERATOR_GBY_235: 1289751 INFO : RECORDS_OUT_OPERATOR_GBY_237: 1289705 INFO : RECORDS_OUT_OPERATOR_GBY_242: 1439749 INFO : RECORDS_OUT_OPERATOR_GBY_249: 2561824 INFO : RECORDS_OUT_OPERATOR_GBY_251: 2561792 INFO : RECORDS_OUT_OPERATOR_GBY_254: 4853606 INFO : RECORDS_OUT_OPERATOR_LIM_263: 100 INFO : RECORDS_OUT_OPERATOR_MAPJOIN_221: 24758945 INFO : RECORDS_OUT_OPERATOR_MAPJOIN_223: 4924002 INFO : RECORDS_OUT_OPERATOR_MAPJOIN_232: 6475966 INFO : RECORDS_OUT_OPERATOR_MAPJOIN_234: 1289773 INFO : RECORDS_OUT_OPERATOR_MAPJOIN_246: 12864622 INFO : RECORDS_OUT_OPERATOR_MAPJOIN_248: 2561824 INFO : RECORDS_OUT_OPERATOR_MAPJOIN_256: 4853606 INFO : RECORDS_OUT_OPERATOR_MAPJOIN_257: 4853606 INFO : RECORDS_OUT_OPERATOR_MAP_0: 0 INFO : RECORDS_OUT_OPERATOR_RS_212: 365 INFO : RECORDS_OUT_OPERATOR_RS_213: 365 INFO : RECORDS_OUT_OPERATOR_RS_214: 365 INFO : RECORDS_OUT_OPERATOR_RS_218: 2875432 INFO : RECORDS_OUT_OPERATOR_RS_225: 4922835 INFO : RECORDS_OUT_OPERATOR_RS_229: 719217 INFO : RECORDS_OUT_OPERATOR_RS_236: 1289751 INFO : RECORDS_OUT_OPERATOR_RS_239: 1289705 INFO : RECORDS_OUT_OPERATOR_RS_243: 1439749 INFO : RECORDS_OUT_OPERATOR_RS_250: 2561824 INFO : RECORDS_OUT_OPERATOR_RS_253: 2561792 INFO : RECORDS_OUT_OPERATOR_RS_261: 100 INFO : RECORDS_OUT_OPERATOR_SEL_211: 365 INFO : RECORDS_OUT_OPERATOR_SEL_216: 2875432 INFO : RECORDS_OUT_OPERATOR_SEL_220: 27504814 INFO : RECORDS_OUT_OPERATOR_SEL_222: 24758945 INFO : RECORDS_OUT_OPERATOR_SEL_227: 719217 INFO : RECORDS_OUT_OPERATOR_SEL_231: 7194926 INFO : RECORDS_OUT_OPERATOR_SEL_233: 6475966 INFO : RECORDS_OUT_OPERATOR_SEL_238: 1289705 INFO : RECORDS_OUT_OPERATOR_SEL_241: 1439749 INFO : RECORDS_OUT_OPERATOR_SEL_245: 14293543 INFO : RECORDS_OUT_OPERATOR_SEL_247: 12864622 INFO : RECORDS_OUT_OPERATOR_SEL_252: 2561792 INFO : RECORDS_OUT_OPERATOR_SEL_255: 4853606 INFO : RECORDS_OUT_OPERATOR_SEL_260: 100 INFO : RECORDS_OUT_OPERATOR_SEL_262: 100 INFO : RECORDS_OUT_OPERATOR_TNK_259: 100 INFO : RECORDS_OUT_OPERATOR_TS_0: 10000 INFO : RECORDS_OUT_OPERATOR_TS_26: 7197566 INFO : RECORDS_OUT_OPERATOR_TS_29: 719217 INFO : RECORDS_OUT_OPERATOR_TS_3: 28800991 INFO : RECORDS_OUT_OPERATOR_TS_49: 14401261 INFO : RECORDS_OUT_OPERATOR_TS_52: 1439749 INFO : RECORDS_OUT_OPERATOR_TS_6: 2875432 INFO : Shuffle Errors: INFO : BAD_ID: 0 INFO : CONNECTION: 0 INFO : IO_ERROR: 0 INFO : WRONG_LENGTH: 0 INFO : WRONG_MAP: 0 INFO : WRONG_REDUCE: 0 INFO : org.apache.hadoop.hive.llap.counters.LlapIOCounters: INFO : ALLOCATED_BYTES: 653934592 INFO : ALLOCATED_USED_BYTES: 519174466 INFO : CACHE_HIT_BYTES: 0 INFO : CACHE_MISS_BYTES: 428390032 INFO : CONSUMER_TIME_NS: 170361062874 INFO : DECODE_TIME_NS: 10361202915 INFO : HDFS_TIME_NS: 814496564234 INFO : IO_CPU_NS: 109391000878 INFO : IO_USER_NS: 91630000000 INFO : METADATA_CACHE_MISS: 868 INFO : NUM_DECODED_BATCHES: 5773 INFO : NUM_DECODED_ROWS: 55444216 INFO : NUM_INPUT_ROWS: 55444216 INFO : NUM_VECTOR_BATCHES: 55619 INFO : ROWS_EMITTED: 55444216 INFO : SELECTED_ROWGROUPS: 5773 INFO : TOTAL_IO_TIME_NS: 732623429076 INFO : org.apache.hadoop.hive.llap.counters.LlapWmCounters: INFO : GUARANTEED_QUEUED_NS: 0 INFO : GUARANTEED_RUNNING_NS: 0 INFO : SPECULATIVE_QUEUED_NS: 709114332106 INFO : SPECULATIVE_RUNNING_NS: 1017559853343 INFO : org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable$LlapExecutorCounters: INFO : EXECUTOR_CPU_NS: 200393162071 INFO : EXECUTOR_USER_NS: 177470000000 INFO : org.apache.hadoop.hive.ql.exec.tez.HiveInputCounters: INFO : GROUPED_INPUT_SPLITS_Map_1: 1 INFO : GROUPED_INPUT_SPLITS_Map_11: 6 INFO : GROUPED_INPUT_SPLITS_Map_2: 22 INFO : GROUPED_INPUT_SPLITS_Map_5: 10 INFO : GROUPED_INPUT_SPLITS_Map_6: 23 INFO : GROUPED_INPUT_SPLITS_Map_8: 4 INFO : GROUPED_INPUT_SPLITS_Map_9: 22 INFO : INPUT_DIRECTORIES_Map_1: 1 INFO : INPUT_DIRECTORIES_Map_11: 1 INFO : INPUT_DIRECTORIES_Map_2: 1 INFO : INPUT_DIRECTORIES_Map_5: 1 INFO : INPUT_DIRECTORIES_Map_6: 1 INFO : INPUT_DIRECTORIES_Map_8: 1 INFO : INPUT_DIRECTORIES_Map_9: 1 INFO : INPUT_FILES_Map_1: 1 INFO : INPUT_FILES_Map_11: 17 INFO : INPUT_FILES_Map_2: 120 INFO : INPUT_FILES_Map_5: 20 INFO : INPUT_FILES_Map_6: 90 INFO : INPUT_FILES_Map_8: 6 INFO : INPUT_FILES_Map_9: 120 INFO : RAW_INPUT_SPLITS_Map_1: 1 INFO : RAW_INPUT_SPLITS_Map_11: 17 INFO : RAW_INPUT_SPLITS_Map_2: 120 INFO : RAW_INPUT_SPLITS_Map_5: 20 INFO : RAW_INPUT_SPLITS_Map_6: 90 INFO : RAW_INPUT_SPLITS_Map_8: 6 INFO : RAW_INPUT_SPLITS_Map_9: 120 INFO : savedToCache: true INFO : Completed executing command(queryId=hive_20231020065526_8a03bdad-aee2-4e1a-9e5f-76db2ebd2454); Time taken: 102.942 seconds INFO : OK