[centos@ip-10-175-156-172 ~]$ beeline -u 'jdbc:hive2://localhost:10000' -n hive -f runAll.sql Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 scan complete in 2ms Connecting to jdbc:hive2://localhost:10000 Connected to: Apache Hive (version 1.1.0-cdh5.14.4) Driver: Hive JDBC (version 1.1.0-cdh5.14.4) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://localhost:10000> -- Create a Retail DB if it doesn't exist 0: jdbc:hive2://localhost:10000> CREATE DATABASE IF NOT EXISTS testdb LOCATION 's3a://cloudera-files/tmp/test-data'; INFO : Compiling command(queryId=hive_20180803212929_27b81b08-e30c-4104-99ea-cec2f55d9030): CREATE DATABASE IF NOT EXISTS testdb LOCATION 's3a://cloudera-files/tmp/test-data' INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20180803212929_27b81b08-e30c-4104-99ea-cec2f55d9030); Time taken: 1.324 seconds INFO : Executing command(queryId=hive_20180803212929_27b81b08-e30c-4104-99ea-cec2f55d9030): CREATE DATABASE IF NOT EXISTS testdb LOCATION 's3a://cloudera-files/tmp/test-data' INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20180803212929_27b81b08-e30c-4104-99ea-cec2f55d9030); Time taken: 1.497 seconds INFO : OK No rows affected (2.922 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> -- Create SQL's in the retaildb 0: jdbc:hive2://localhost:10000> use testdb; INFO : Compiling command(queryId=hive_20180803212929_53505db8-aaa7-46bf-acf5-6b672b64c150): use testdb INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20180803212929_53505db8-aaa7-46bf-acf5-6b672b64c150); Time taken: 0.085 seconds INFO : Executing command(queryId=hive_20180803212929_53505db8-aaa7-46bf-acf5-6b672b64c150): use testdb INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20180803212929_53505db8-aaa7-46bf-acf5-6b672b64c150); Time taken: 0.033 seconds INFO : OK No rows affected (0.147 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> -- Create product table, drop if exists first 0: jdbc:hive2://localhost:10000> drop table if exists products; INFO : Compiling command(queryId=hive_20180803212929_ed76bd3d-00f1-4b89-9fc5-bdb8d78124d3): drop table if exists products INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20180803212929_ed76bd3d-00f1-4b89-9fc5-bdb8d78124d3); Time taken: 0.063 seconds INFO : Executing command(queryId=hive_20180803212929_ed76bd3d-00f1-4b89-9fc5-bdb8d78124d3): drop table if exists products INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20180803212929_ed76bd3d-00f1-4b89-9fc5-bdb8d78124d3); Time taken: 0.017 seconds INFO : OK No rows affected (0.104 seconds) 0: jdbc:hive2://localhost:10000> CREATE EXTERNAL TABLE products (product_id int, product_category_id int, product_name string, product_description string, product_price float, product_image string) . . . . . . . . . . . . . . . .> stored as parquet LOCATION 's3a://cloudera-files/tmp/test-data/data/warehouse/products/'; INFO : Compiling command(queryId=hive_20180803212929_e2f1b20f-40b3-4b54-bbb0-b045b9a9a893): CREATE EXTERNAL TABLE products (product_id int, product_category_id int, product_name string, product_description string, product_price float, product_image string) stored as parquet LOCATION 's3a://cloudera-files/tmp/test-data/data/warehouse/products/' INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20180803212929_e2f1b20f-40b3-4b54-bbb0-b045b9a9a893); Time taken: 0.152 seconds INFO : Executing command(queryId=hive_20180803212929_e2f1b20f-40b3-4b54-bbb0-b045b9a9a893): CREATE EXTERNAL TABLE products (product_id int, product_category_id int, product_name string, product_description string, product_price float, product_image string) stored as parquet LOCATION 's3a://cloudera-files/tmp/test-data/data/warehouse/products/' INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20180803212929_e2f1b20f-40b3-4b54-bbb0-b045b9a9a893); Time taken: 0.383 seconds INFO : OK No rows affected (0.608 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> -- Create orders table, drop if exists first 0: jdbc:hive2://localhost:10000> drop table if exists orders; INFO : Compiling command(queryId=hive_20180803212929_027c8fe4-4408-4c93-b04e-3f3e6750f42e): drop table if exists orders INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20180803212929_027c8fe4-4408-4c93-b04e-3f3e6750f42e); Time taken: 0.008 seconds INFO : Executing command(queryId=hive_20180803212929_027c8fe4-4408-4c93-b04e-3f3e6750f42e): drop table if exists orders INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20180803212929_027c8fe4-4408-4c93-b04e-3f3e6750f42e); Time taken: 0.006 seconds INFO : OK No rows affected (0.052 seconds) 0: jdbc:hive2://localhost:10000> CREATE EXTERNAL TABLE orders (order_id int, order_date bigint, order_customer_id int, order_status string) stored as parquet LOCATION 's3a://cloudera-files/tmp/test-data/data/warehouse/orders'; INFO : Compiling command(queryId=hive_20180803212929_c4c1f2fc-cd02-4122-9f22-87c06140186b): CREATE EXTERNAL TABLE orders (order_id int, order_date bigint, order_customer_id int, order_status string) stored as parquet LOCATION 's3a://cloudera-files/tmp/test-data/data/warehouse/orders' INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20180803212929_c4c1f2fc-cd02-4122-9f22-87c06140186b); Time taken: 0.009 seconds INFO : Executing command(queryId=hive_20180803212929_c4c1f2fc-cd02-4122-9f22-87c06140186b): CREATE EXTERNAL TABLE orders (order_id int, order_date bigint, order_customer_id int, order_status string) stored as parquet LOCATION 's3a://cloudera-files/tmp/test-data/data/warehouse/orders' INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20180803212929_c4c1f2fc-cd02-4122-9f22-87c06140186b); Time taken: 0.245 seconds INFO : OK No rows affected (0.295 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> SELECT COUNT(*) FROM products; INFO : Compiling command(queryId=hive_20180803212929_643c6743-649c-42bb-a296-be17c3364f72): SELECT COUNT(*) FROM products INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20180803212929_643c6743-649c-42bb-a296-be17c3364f72); Time taken: 21.183 seconds INFO : Executing command(queryId=hive_20180803212929_643c6743-649c-42bb-a296-be17c3364f72): SELECT COUNT(*) FROM products INFO : Query ID = hive_20180803212929_643c6743-649c-42bb-a296-be17c3364f72 INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : In order to change the average load for a reducer (in bytes): INFO : set hive.exec.reducers.bytes.per.reducer= INFO : In order to limit the maximum number of reducers: INFO : set hive.exec.reducers.max= INFO : In order to set a constant number of reducers: INFO : set mapreduce.job.reduces= INFO : Starting Spark Job = 64093376-1b57-4c8b-a136-b34b325488d8 INFO : Running with YARN Application = application_1533330640119_0001 INFO : Kill Command = /opt/cloudera/parcels/CDH-5.14.4-1.cdh5.14.4.p0.3/lib/hadoop/bin/yarn application -kill application_1533330640119_0001 INFO : Query Hive on Spark job[0] stages: INFO : 0 INFO : 1 INFO : Status: Running (Hive on Spark job[0]) INFO : Job Progress Format CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost] INFO : 2018-08-03 21:30:21,037 Stage-0_0: 0(+1)/1 Stage-1_0: 0/1 INFO : 2018-08-03 21:30:24,054 Stage-0_0: 0(+1)/1 Stage-1_0: 0/1 INFO : 2018-08-03 21:30:26,062 Stage-0_0: 1/1 Finished Stage-1_0: 1/1 Finished INFO : Status: Finished successfully in 20.06 seconds INFO : =====Spark Job[64093376-1b57-4c8b-a136-b34b325488d8] statistics===== INFO : HIVE INFO : CREATED_FILES: 1 INFO : DESERIALIZE_ERRORS: 0 INFO : RECORDS_OUT_INTERMEDIATE: 1 INFO : RECORDS_IN: 1345 INFO : RECORDS_OUT_0: 1 INFO : Spark Job[64093376-1b57-4c8b-a136-b34b325488d8] Metrics INFO : ExecutorDeserializeTime: 1721 INFO : ExecutorRunTime: 3341 INFO : ResultSize: 3600 INFO : JvmGCTime: 201 INFO : ResultSerializationTime: 0 INFO : MemoryBytesSpilled: 0 INFO : DiskBytesSpilled: 0 INFO : BytesRead: 3832 INFO : RemoteBlocksFetched: 0 INFO : LocalBlocksFetched: 1 INFO : TotalBlocksFetched: 1 INFO : FetchWaitTime: 0 INFO : RemoteBytesRead: 0 INFO : ShuffleBytesWritten: 41 INFO : ShuffleWriteTime: 6190542 INFO : Execution completed successfully INFO : Completed executing command(queryId=hive_20180803212929_643c6743-649c-42bb-a296-be17c3364f72); Time taken: 33.392 seconds INFO : OK +-------+--+ | _c0 | +-------+--+ | 1345 | +-------+--+ 1 row selected (54.689 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> SELECT * from products limit 1; INFO : Compiling command(queryId=hive_20180803213030_5d9b20b9-0d1f-4ea9-a603-694fc31d2006): SELECT * from products limit 1 INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:products.product_id, type:int, comment:null), FieldSchema(name:products.product_category_id, type:int, comment:null), FieldSchema(name:products.product_name, type:string, comment:null), FieldSchema(name:products.product_description, type:string, comment:null), FieldSchema(name:products.product_price, type:float, comment:null), FieldSchema(name:products.product_image, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20180803213030_5d9b20b9-0d1f-4ea9-a603-694fc31d2006); Time taken: 0.066 seconds INFO : Executing command(queryId=hive_20180803213030_5d9b20b9-0d1f-4ea9-a603-694fc31d2006): SELECT * from products limit 1 INFO : Completed executing command(queryId=hive_20180803213030_5d9b20b9-0d1f-4ea9-a603-694fc31d2006); Time taken: 0.011 seconds INFO : OK +----------------------+-------------------------------+------------------------------------------------+-------------------------------+-------------------------+----------------------------------------------------+--+ | products.product_id | products.product_category_id | products.product_name | products.product_description | products.product_price | products.product_image | +----------------------+-------------------------------+------------------------------------------------+-------------------------------+-------------------------+----------------------------------------------------+--+ | 1 | 2 | Quest Q64 10 FT. x 10 FT. Slant Leg Instant U | | 59.97999954223633 | http://images.acmesports.sports/Quest+Q64+10+FT.+x+10+FT.+Slant+Leg+Instant+Up+Canopy | +----------------------+-------------------------------+------------------------------------------------+-------------------------------+-------------------------+----------------------------------------------------+--+ 1 row selected (1.337 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> SELECT COUNT(*) FROM orders; INFO : Compiling command(queryId=hive_20180803213030_9b7ace06-d2d9-41f9-a82c-aeeb9ad4765c): SELECT COUNT(*) FROM orders INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20180803213030_9b7ace06-d2d9-41f9-a82c-aeeb9ad4765c); Time taken: 20.657 seconds INFO : Executing command(queryId=hive_20180803213030_9b7ace06-d2d9-41f9-a82c-aeeb9ad4765c): SELECT COUNT(*) FROM orders INFO : Query ID = hive_20180803213030_9b7ace06-d2d9-41f9-a82c-aeeb9ad4765c INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : In order to change the average load for a reducer (in bytes): INFO : set hive.exec.reducers.bytes.per.reducer= INFO : In order to limit the maximum number of reducers: INFO : set hive.exec.reducers.max= INFO : In order to set a constant number of reducers: INFO : set mapreduce.job.reduces= INFO : Starting Spark Job = 6ac4b763-4b38-480d-89a6-a49e984d3c9f INFO : Running with YARN Application = application_1533330640119_0001 INFO : Kill Command = /opt/cloudera/parcels/CDH-5.14.4-1.cdh5.14.4.p0.3/lib/hadoop/bin/yarn application -kill application_1533330640119_0001 INFO : Query Hive on Spark job[1] stages: INFO : 2 INFO : 3 INFO : Status: Running (Hive on Spark job[1]) INFO : Job Progress Format CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost] INFO : 2018-08-03 21:30:49,296 Stage-2_0: 0(+1)/1 Stage-3_0: 0/1 INFO : 2018-08-03 21:30:52,310 Stage-2_0: 0(+1)/1 Stage-3_0: 0/1 INFO : 2018-08-03 21:30:54,318 Stage-2_0: 1/1 Finished Stage-3_0: 0(+1)/1 INFO : 2018-08-03 21:30:57,329 Stage-2_0: 1/1 Finished Stage-3_0: 1/1 Finished INFO : Status: Finished successfully in 9.04 seconds INFO : =====Spark Job[6ac4b763-4b38-480d-89a6-a49e984d3c9f] statistics===== INFO : HIVE INFO : CREATED_FILES: 1 INFO : DESERIALIZE_ERRORS: 0 INFO : RECORDS_OUT_INTERMEDIATE: 1 INFO : RECORDS_IN: 68883 INFO : RECORDS_OUT_0: 1 INFO : Spark Job[6ac4b763-4b38-480d-89a6-a49e984d3c9f] Metrics INFO : ExecutorDeserializeTime: 3447 INFO : ExecutorRunTime: 3964 INFO : ResultSize: 3603 INFO : JvmGCTime: 261 INFO : ResultSerializationTime: 2 INFO : MemoryBytesSpilled: 0 INFO : DiskBytesSpilled: 0 INFO : BytesRead: 3567 INFO : RemoteBlocksFetched: 2 INFO : LocalBlocksFetched: 0 INFO : TotalBlocksFetched: 2 INFO : FetchWaitTime: 6 INFO : RemoteBytesRead: 84 INFO : ShuffleBytesWritten: 42 INFO : ShuffleWriteTime: 9827620 INFO : Execution completed successfully INFO : Completed executing command(queryId=hive_20180803213030_9b7ace06-d2d9-41f9-a82c-aeeb9ad4765c); Time taken: 9.126 seconds INFO : OK +--------+--+ | _c0 | +--------+--+ | 68883 | +--------+--+ 1 row selected (29.831 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> SELECT * from orders limit 1; INFO : Compiling command(queryId=hive_20180803213030_e175a931-4417-4811-89cb-7a332225f4d8): SELECT * from orders limit 1 INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:orders.order_id, type:int, comment:null), FieldSchema(name:orders.order_date, type:bigint, comment:null), FieldSchema(name:orders.order_customer_id, type:int, comment:null), FieldSchema(name:orders.order_status, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20180803213030_e175a931-4417-4811-89cb-7a332225f4d8); Time taken: 0.059 seconds INFO : Executing command(queryId=hive_20180803213030_e175a931-4417-4811-89cb-7a332225f4d8): SELECT * from orders limit 1 INFO : Completed executing command(queryId=hive_20180803213030_e175a931-4417-4811-89cb-7a332225f4d8); Time taken: 0.014 seconds INFO : OK +------------------+--------------------+---------------------------+----------------------+--+ | orders.order_id | orders.order_date | orders.order_customer_id | orders.order_status | +------------------+--------------------+---------------------------+----------------------+--+ | 1 | 1374735600000 | 11599 | CLOSED | +------------------+--------------------+---------------------------+----------------------+--+ 1 row selected (1.173 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> DROP DATABASE testdb CASCADE; INFO : Compiling command(queryId=hive_20180803213030_33837221-5181-4386-98a5-90c3726cd705): DROP DATABASE testdb CASCADE INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20180803213030_33837221-5181-4386-98a5-90c3726cd705); Time taken: 0.051 seconds INFO : Executing command(queryId=hive_20180803213030_33837221-5181-4386-98a5-90c3726cd705): DROP DATABASE testdb CASCADE INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20180803213030_33837221-5181-4386-98a5-90c3726cd705); Time taken: 5.714 seconds INFO : OK No rows affected (5.786 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> -- Create a Retail DB if it doesn't exist 0: jdbc:hive2://localhost:10000> CREATE DATABASE IF NOT EXISTS testdb LOCATION 's3a://cloudera-files/tmp/test-data'; INFO : Compiling command(queryId=hive_20180803213131_2bda4df4-e889-4775-98b1-60a2bcbe396c): CREATE DATABASE IF NOT EXISTS testdb LOCATION 's3a://cloudera-files/tmp/test-data' INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20180803213131_2bda4df4-e889-4775-98b1-60a2bcbe396c); Time taken: 0.004 seconds INFO : Executing command(queryId=hive_20180803213131_2bda4df4-e889-4775-98b1-60a2bcbe396c): CREATE DATABASE IF NOT EXISTS testdb LOCATION 's3a://cloudera-files/tmp/test-data' INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20180803213131_2bda4df4-e889-4775-98b1-60a2bcbe396c); Time taken: 0.806 seconds INFO : OK No rows affected (0.819 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> -- Create SQL's in the retaildb 0: jdbc:hive2://localhost:10000> use testdb; INFO : Compiling command(queryId=hive_20180803213131_09c439e2-f967-47df-83e4-744ee54f5a55): use testdb INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20180803213131_09c439e2-f967-47df-83e4-744ee54f5a55); Time taken: 0.004 seconds INFO : Executing command(queryId=hive_20180803213131_09c439e2-f967-47df-83e4-744ee54f5a55): use testdb INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20180803213131_09c439e2-f967-47df-83e4-744ee54f5a55); Time taken: 0.009 seconds INFO : OK No rows affected (0.019 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> -- Create product table, drop if exists first 0: jdbc:hive2://localhost:10000> drop table if exists products; INFO : Compiling command(queryId=hive_20180803213131_f942bf05-0fba-478f-8b9d-05f47dabaa91): drop table if exists products INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20180803213131_f942bf05-0fba-478f-8b9d-05f47dabaa91); Time taken: 0.003 seconds INFO : Executing command(queryId=hive_20180803213131_f942bf05-0fba-478f-8b9d-05f47dabaa91): drop table if exists products INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20180803213131_f942bf05-0fba-478f-8b9d-05f47dabaa91); Time taken: 0.007 seconds INFO : OK No rows affected (0.021 seconds) 0: jdbc:hive2://localhost:10000> CREATE EXTERNAL TABLE products (product_id int, product_category_id int, product_name string, product_description string, product_price float, product_image string) . . . . . . . . . . . . . . . .> stored as parquet LOCATION 's3a://cloudera-files/tmp/test-data/data/warehouse/products/'; INFO : Compiling command(queryId=hive_20180803213131_e82d8151-9bf3-4d7e-a8c7-6b60e4551350): CREATE EXTERNAL TABLE products (product_id int, product_category_id int, product_name string, product_description string, product_price float, product_image string) stored as parquet LOCATION 's3a://cloudera-files/tmp/test-data/data/warehouse/products/' INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20180803213131_e82d8151-9bf3-4d7e-a8c7-6b60e4551350); Time taken: 0.006 seconds INFO : Executing command(queryId=hive_20180803213131_e82d8151-9bf3-4d7e-a8c7-6b60e4551350): CREATE EXTERNAL TABLE products (product_id int, product_category_id int, product_name string, product_description string, product_price float, product_image string) stored as parquet LOCATION 's3a://cloudera-files/tmp/test-data/data/warehouse/products/' INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20180803213131_e82d8151-9bf3-4d7e-a8c7-6b60e4551350); Time taken: 1.711 seconds INFO : OK No rows affected (1.744 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> -- Create orders table, drop if exists first 0: jdbc:hive2://localhost:10000> drop table if exists orders; INFO : Compiling command(queryId=hive_20180803213131_6def002a-daf2-4bce-abaa-7b51069a2462): drop table if exists orders INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20180803213131_6def002a-daf2-4bce-abaa-7b51069a2462); Time taken: 0.006 seconds INFO : Executing command(queryId=hive_20180803213131_6def002a-daf2-4bce-abaa-7b51069a2462): drop table if exists orders INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20180803213131_6def002a-daf2-4bce-abaa-7b51069a2462); Time taken: 0.009 seconds INFO : OK No rows affected (0.028 seconds) 0: jdbc:hive2://localhost:10000> CREATE EXTERNAL TABLE orders (order_id int, order_date bigint, order_customer_id int, order_status string) stored as parquet LOCATION 's3a://cloudera-files/tmp/test-data/data/warehouse/orders'; INFO : Compiling command(queryId=hive_20180803213131_5bd7c995-1b08-45df-a05d-0804ed9fe36e): CREATE EXTERNAL TABLE orders (order_id int, order_date bigint, order_customer_id int, order_status string) stored as parquet LOCATION 's3a://cloudera-files/tmp/test-data/data/warehouse/orders' INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20180803213131_5bd7c995-1b08-45df-a05d-0804ed9fe36e); Time taken: 0.005 seconds INFO : Executing command(queryId=hive_20180803213131_5bd7c995-1b08-45df-a05d-0804ed9fe36e): CREATE EXTERNAL TABLE orders (order_id int, order_date bigint, order_customer_id int, order_status string) stored as parquet LOCATION 's3a://cloudera-files/tmp/test-data/data/warehouse/orders' INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20180803213131_5bd7c995-1b08-45df-a05d-0804ed9fe36e); Time taken: 1.022 seconds INFO : OK No rows affected (1.05 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> SELECT COUNT(*) FROM products; INFO : Compiling command(queryId=hive_20180803213131_d5d328fb-f5b9-47c3-90e6-09b9129bdeb6): SELECT COUNT(*) FROM products INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20180803213131_d5d328fb-f5b9-47c3-90e6-09b9129bdeb6); Time taken: 20.567 seconds INFO : Executing command(queryId=hive_20180803213131_d5d328fb-f5b9-47c3-90e6-09b9129bdeb6): SELECT COUNT(*) FROM products INFO : Query ID = hive_20180803213131_d5d328fb-f5b9-47c3-90e6-09b9129bdeb6 INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : In order to change the average load for a reducer (in bytes): INFO : set hive.exec.reducers.bytes.per.reducer= INFO : In order to limit the maximum number of reducers: INFO : set hive.exec.reducers.max= INFO : In order to set a constant number of reducers: INFO : set mapreduce.job.reduces= INFO : Starting Spark Job = 47f51f5e-c33b-46ca-bd09-978531e67950 INFO : Running with YARN Application = application_1533330640119_0001 INFO : Kill Command = /opt/cloudera/parcels/CDH-5.14.4-1.cdh5.14.4.p0.3/lib/hadoop/bin/yarn application -kill application_1533330640119_0001 INFO : Query Hive on Spark job[2] stages: INFO : 5 INFO : 4 INFO : Status: Running (Hive on Spark job[2]) INFO : Job Progress Format CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost] INFO : 2018-08-03 21:31:29,787 Stage-4_0: 1/1 Finished Stage-5_0: 0(+1)/1 INFO : 2018-08-03 21:31:30,791 Stage-4_0: 1/1 Finished Stage-5_0: 1/1 Finished INFO : Status: Finished successfully in 2.01 seconds INFO : =====Spark Job[47f51f5e-c33b-46ca-bd09-978531e67950] statistics===== INFO : HIVE INFO : CREATED_FILES: 1 INFO : DESERIALIZE_ERRORS: 0 INFO : RECORDS_OUT_INTERMEDIATE: 1 INFO : RECORDS_IN: 0 INFO : RECORDS_OUT_0: 1 INFO : Spark Job[47f51f5e-c33b-46ca-bd09-978531e67950] Metrics INFO : ExecutorDeserializeTime: 47 INFO : ExecutorRunTime: 488 INFO : ResultSize: 3594 INFO : JvmGCTime: 0 INFO : ResultSerializationTime: 0 INFO : MemoryBytesSpilled: 0 INFO : DiskBytesSpilled: 0 INFO : BytesRead: 5020 INFO : RemoteBlocksFetched: 0 INFO : LocalBlocksFetched: 1 INFO : TotalBlocksFetched: 1 INFO : FetchWaitTime: 0 INFO : RemoteBytesRead: 0 INFO : ShuffleBytesWritten: 38 INFO : ShuffleWriteTime: 747310 INFO : Execution completed successfully INFO : Completed executing command(queryId=hive_20180803213131_d5d328fb-f5b9-47c3-90e6-09b9129bdeb6); Time taken: 2.115 seconds INFO : OK +------+--+ | _c0 | +------+--+ | 0 | +------+--+ 1 row selected (22.716 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> SELECT * from products limit 1; INFO : Compiling command(queryId=hive_20180803213131_55c02466-f0c2-48d8-a2dd-f0ba12dddf02): SELECT * from products limit 1 INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:products.product_id, type:int, comment:null), FieldSchema(name:products.product_category_id, type:int, comment:null), FieldSchema(name:products.product_name, type:string, comment:null), FieldSchema(name:products.product_description, type:string, comment:null), FieldSchema(name:products.product_price, type:float, comment:null), FieldSchema(name:products.product_image, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20180803213131_55c02466-f0c2-48d8-a2dd-f0ba12dddf02); Time taken: 0.031 seconds INFO : Executing command(queryId=hive_20180803213131_55c02466-f0c2-48d8-a2dd-f0ba12dddf02): SELECT * from products limit 1 INFO : Completed executing command(queryId=hive_20180803213131_55c02466-f0c2-48d8-a2dd-f0ba12dddf02); Time taken: 0.007 seconds INFO : OK +----------------------+-------------------------------+------------------------+-------------------------------+-------------------------+-------------------------+--+ | products.product_id | products.product_category_id | products.product_name | products.product_description | products.product_price | products.product_image | +----------------------+-------------------------------+------------------------+-------------------------------+-------------------------+-------------------------+--+ +----------------------+-------------------------------+------------------------+-------------------------------+-------------------------+-------------------------+--+ No rows selected (0.231 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> SELECT COUNT(*) FROM orders; INFO : Compiling command(queryId=hive_20180803213131_ffbeb78a-3e2f-45c8-ba21-fad82d991ed4): SELECT COUNT(*) FROM orders INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20180803213131_ffbeb78a-3e2f-45c8-ba21-fad82d991ed4); Time taken: 20.521 seconds INFO : Executing command(queryId=hive_20180803213131_ffbeb78a-3e2f-45c8-ba21-fad82d991ed4): SELECT COUNT(*) FROM orders INFO : Query ID = hive_20180803213131_ffbeb78a-3e2f-45c8-ba21-fad82d991ed4 INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : In order to change the average load for a reducer (in bytes): INFO : set hive.exec.reducers.bytes.per.reducer= INFO : In order to limit the maximum number of reducers: INFO : set hive.exec.reducers.max= INFO : In order to set a constant number of reducers: INFO : set mapreduce.job.reduces= INFO : Starting Spark Job = 1e8f6e2f-788d-4cfd-bed3-6fa7b8cbf694 INFO : Running with YARN Application = application_1533330640119_0001 INFO : Kill Command = /opt/cloudera/parcels/CDH-5.14.4-1.cdh5.14.4.p0.3/lib/hadoop/bin/yarn application -kill application_1533330640119_0001 INFO : Query Hive on Spark job[3] stages: INFO : 6 INFO : 7 INFO : Status: Running (Hive on Spark job[3]) INFO : Job Progress Format CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost] INFO : 2018-08-03 21:31:52,672 Stage-6_0: 1/1 Finished Stage-7_0: 1/1 Finished INFO : Status: Finished successfully in 2.01 seconds INFO : =====Spark Job[1e8f6e2f-788d-4cfd-bed3-6fa7b8cbf694] statistics===== INFO : HIVE INFO : CREATED_FILES: 1 INFO : DESERIALIZE_ERRORS: 0 INFO : RECORDS_OUT_INTERMEDIATE: 1 INFO : RECORDS_IN: 0 INFO : RECORDS_OUT_0: 1 INFO : Spark Job[1e8f6e2f-788d-4cfd-bed3-6fa7b8cbf694] Metrics INFO : ExecutorDeserializeTime: 49 INFO : ExecutorRunTime: 500 INFO : ResultSize: 3594 INFO : JvmGCTime: 0 INFO : ResultSerializationTime: 0 INFO : MemoryBytesSpilled: 0 INFO : DiskBytesSpilled: 0 INFO : BytesRead: 4576 INFO : RemoteBlocksFetched: 2 INFO : LocalBlocksFetched: 0 INFO : TotalBlocksFetched: 2 INFO : FetchWaitTime: 18 INFO : RemoteBytesRead: 76 INFO : ShuffleBytesWritten: 38 INFO : ShuffleWriteTime: 4818358 INFO : Execution completed successfully INFO : Completed executing command(queryId=hive_20180803213131_ffbeb78a-3e2f-45c8-ba21-fad82d991ed4); Time taken: 2.082 seconds INFO : OK +------+--+ | _c0 | +------+--+ | 0 | +------+--+ 1 row selected (22.632 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> SELECT * from orders limit 1; INFO : Compiling command(queryId=hive_20180803213131_75f4bc67-ebc8-4f1f-9f90-13abe153ca53): SELECT * from orders limit 1 INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:orders.order_id, type:int, comment:null), FieldSchema(name:orders.order_date, type:bigint, comment:null), FieldSchema(name:orders.order_customer_id, type:int, comment:null), FieldSchema(name:orders.order_status, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20180803213131_75f4bc67-ebc8-4f1f-9f90-13abe153ca53); Time taken: 0.033 seconds INFO : Executing command(queryId=hive_20180803213131_75f4bc67-ebc8-4f1f-9f90-13abe153ca53): SELECT * from orders limit 1 INFO : Completed executing command(queryId=hive_20180803213131_75f4bc67-ebc8-4f1f-9f90-13abe153ca53); Time taken: 0.007 seconds INFO : OK +------------------+--------------------+---------------------------+----------------------+--+ | orders.order_id | orders.order_date | orders.order_customer_id | orders.order_status | +------------------+--------------------+---------------------------+----------------------+--+ +------------------+--------------------+---------------------------+----------------------+--+ No rows selected (0.228 seconds) 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> 0: jdbc:hive2://localhost:10000> Closing: 0: jdbc:hive2://localhost:10000 [centos@ip-10-175-156-172 ~]$ [centos@ip-10-175-156-172 ~]$