Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
While running a cluster query (hs2, tez am, tez+llap), we have a handy summary attached as: tpcds78_10GB_query_summary_example.txt
However, we get no info about simply fetch tasks, we can see only this below:
0: jdbc:hive2://hs2-lbodor-aws-bug.dw-dw-team> select * from customer limit 10; INFO : Compiling command(queryId=hive_20231116094515_5f3e5f7b-da6a-4026-9b36-be63ee11ff61): select * from customer limit 10 INFO : Semantic Analysis Completed (retrial = false) INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:customer.c_customer_sk, type:bigint, comment:null), FieldSchema(name:customer.c_customer_id, type:char(16), comment:null), FieldSchema(name:customer.c_current_cdemo_sk, type:bigint, comment:null), FieldSchema(name:customer.c_current_hdemo_sk, type:bigint, comment:null), FieldSchema(name:customer.c_current_addr_sk, type:bigint, comment:null), FieldSchema(name:customer.c_first_shipto_date_sk, type:bigint, comment:null), FieldSchema(name:customer.c_first_sales_date_sk, type:bigint, comment:null), FieldSchema(name:customer.c_salutation, type:char(10), comment:null), FieldSchema(name:customer.c_first_name, type:char(20), comment:null), FieldSchema(name:customer.c_last_name, type:char(30), comment:null), FieldSchema(name:customer.c_preferred_cust_flag, type:char(1), comment:null), FieldSchema(name:customer.c_birth_day, type:int, comment:null), FieldSchema(name:customer.c_birth_month, type:int, comment:null), FieldSchema(name:customer.c_birth_year, type:int, comment:null), FieldSchema(name:customer.c_birth_country, type:varchar(20), comment:null), FieldSchema(name:customer.c_login, type:char(13), comment:null), FieldSchema(name:customer.c_email_address, type:char(50), comment:null), FieldSchema(name:customer.c_last_review_date_sk, type:bigint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20231116094515_5f3e5f7b-da6a-4026-9b36-be63ee11ff61); Time taken: 0.257 seconds INFO : Executing command(queryId=hive_20231116094515_5f3e5f7b-da6a-4026-9b36-be63ee11ff61): select * from customer limit 10 INFO : Completed executing command(queryId=hive_20231116094515_5f3e5f7b-da6a-4026-9b36-be63ee11ff61); Time taken: 0.007 seconds INFO : OK +-------------------------+-------------------------+------------------------------+------------------------------+-----------------------------+----------------------------------+---------------------------------+------------------------+------------------------+---------------------------------+---------------------------------+-----------------------+-------------------------+------------------------+---------------------------+-------------------+----------------------------------------------------+---------------------------------+ | customer.c_customer_sk | customer.c_customer_id | customer.c_current_cdemo_sk | customer.c_current_hdemo_sk | customer.c_current_addr_sk | customer.c_first_shipto_date_sk | customer.c_first_sales_date_sk | customer.c_salutation | customer.c_first_name | customer.c_last_name | customer.c_preferred_cust_flag | customer.c_birth_day | customer.c_birth_month | customer.c_birth_year | customer.c_birth_country | customer.c_login | customer.c_email_address | customer.c_last_review_date_sk | +-------------------------+-------------------------+------------------------------+------------------------------+-----------------------------+----------------------------------+---------------------------------+------------------------+------------------------+---------------------------------+---------------------------------+-----------------------+-------------------------+------------------------+---------------------------+-------------------+----------------------------------------------------+---------------------------------+ | 1 | AAAAAAAABAAAAAAA | 980124 | 7135 | 32946 | 2452238 | 2452208 | Mr. | Javier | Lewis | Y | 9 | 12 | 1936 | CHILE | NULL | Javier.Lewis@VFAxlnZEvOx.org | 2452508 | | 2 | AAAAAAAACAAAAAAA | 819667 | 1461 | 181655 | 2452318 | 2452288 | Dr. | Amy | Moses | Y | 9 | 4 | 1966 | TOGO | NULL | Amy.Moses@Ovk9KjHH.com | 2452318 | | 3 | AAAAAAAADAAAAAAA | 1473522 | 6247 | 248572 | 2449130 | 2449100 | Miss | Latisha | Hamilton | N | 18 | 9 | 1979 | NIUE | NULL | Latisha.Hamilton@V.com | 2452313 | | 4 | AAAAAAAAEAAAAAAA | 1703214 | 3986 | 189558 | 2450030 | 2450000 | Dr. | Michael | White | N | 7 | 6 | 1983 | MEXICO | NULL | Michael.White@i.org | 2452361 | | 5 | AAAAAAAAFAAAAAAA | 953372 | 4470 | 86368 | 2449438 | 2449408 | Sir | Robert | Moran | N | 8 | 5 | 1956 | FIJI | NULL | Robert.Moran@Hh.edu | 2452469 | | 6 | AAAAAAAAGAAAAAAA | 213219 | 6374 | 77082 | 2451883 | 2451853 | Ms. | Brunilda | Sharp | N | 4 | 12 | 1925 | SURINAME | NULL | Brunilda.Sharp@T3pylZEUQjm.org | 2452430 | | 7 | AAAAAAAAHAAAAAAA | 68377 | 3219 | 194814 | 2451438 | 2451408 | Ms. | Fonda | Wiles | Y | 24 | 4 | 1985 | GAMBIA | NULL | Fonda.Wiles@S9KnyEtz9hv.org | 2452360 | | 8 | AAAAAAAAIAAAAAAA | 1215897 | 2471 | 66598 | 2449406 | 2449376 | Sir | Ollie | Shipman | N | 26 | 12 | 1938 | KOREA, REPUBLIC OF | NULL | Ollie.Shipman@be.org | 2452334 | | 9 | AAAAAAAAJAAAAAAA | 1168667 | 1404 | 99388 | 2452275 | 2452245 | Sir | Karl | Gilbert | N | 26 | 10 | 1966 | MONTSERRAT | NULL | Karl.Gilbert@Crg5KyP2IxX9C4d6.edu | 2452454 | | 10 | AAAAAAAAKAAAAAAA | 1207553 | 5143 | 119580 | 2451353 | 2451323 | Ms. | Albert | Brunson | N | 15 | 10 | 1973 | JORDAN | NULL | Albert.Brunson@62.com | 2452641 | +-------------------------+-------------------------+------------------------------+------------------------------+-----------------------------+----------------------------------+---------------------------------+------------------------+------------------------+---------------------------------+---------------------------------+-----------------------+-------------------------+------------------------+---------------------------+-------------------+----------------------------------------------------+---------------------------------+
Looking at attached summary: tpcds78_10GB_query_summary_example.txt , I believe we should adapt some parts of that to HS2-only queries, basically anything which is not related to vertices and runtime outside of HS2, e.g. something like this:
INFO : Status: Query finished successfully in 101.08 seconds
INFO :
INFO : Query Execution Summary
INFO : ----------------------------------------------------------------------------------------------
INFO : OPERATION DURATION
INFO : ----------------------------------------------------------------------------------------------
INFO : Compile Query 1.81s
INFO : Prepare Plan 0.46s
INFO : FileSystem Counters Summary
INFO :
INFO : Scheme: FILE
INFO : ----------------------------------------------------------------------------------------------
INFO: BYTES_READ READ_OPS LARGE_READ_OPS BYTES_WRITTEN WRITE_OPS
INFO : ----------------------------------------------------------------------------------------------
INFO : 0B 0 0 4.86KB 0
INFO : ----------------------------------------------------------------------------------------------
INFO :
INFO : Scheme: S3A
INFO : ----------------------------------------------------------------------------------------------
INFO : BYTES_READ READ_OPS LARGE_READ_OPS BYTES_WRITTEN WRITE_OPS
INFO : ----------------------------------------------------------------------------------------------
INFO : 68.53KB 4 0 0B 0
INFO : ----------------------------------------------------------------------------------------------
INFO : File System Counters:
INFO : FILE_BYTES_READ: 123426528
INFO : FILE_BYTES_WRITTEN: 264749828
INFO : FILE_READ_OPS: 0
INFO : FILE_LARGE_READ_OPS: 0
INFO : FILE_WRITE_OPS: 0
INFO : S3A_BYTES_READ: 438200740
INFO : S3A_BYTES_WRITTEN: 0
INFO : S3A_READ_OPS: 1496
INFO : S3A_LARGE_READ_OPS: 0
INFO : S3A_WRITE_OPS: 0