Major Description
Hi Team,
We are doing a POC where we would like oour query execution to be fatser, mostly in the range of 3 to 4 seconds.
We have read carbon docuements where it has been claimed that carbondata can help to scan PETABYTES of data and present results in 3 to 4 seconds , which does not seem to be the case as per our observation.
Our table size is 1.6 billionand query is fetching only 4K records but still it takes around 22 to 25 seconds for query execution.
Below is our query that we are firing:
==============================
spark.sql("select ts,resource,metric,value from fact_timestamp_global left join tags_10_days_test on fact_timestamp_global.tags_id= tags_10_days_test.id where metric in ('Outbound Utilization (percent)','Inbound Utilization (percent)') and resource='10.212.7.98_if:<0001>' and ts>='2020-09-28 00:00:00' and ts<='2020-09-28 23:55:55'").show(false)
=================================
Definition of fact_timestamp_global is like below:
========================
spark.sql("create table Fact_timestamp_GLOBAL(ts timestamp,metric string,tags_id string,value double) partitioned by (ts2 timestamp) stored as carbondata TBLPROPERTIES ('SORT_COLUMNS'='ts,metric','SORT_SCOPE'='GLOBAL_SORT')").show()
==========================
Definition of tags_10_days_test is like below:
====================
spark.sql("create table tags_10_days_test(id string,resource string) stored as carbondata TBLPROPERTIES('SORT_COLUMNS'='id,resource')").show()
======================
Kindly go through above points and help us the query performence further.