Details
-
Bug
-
Status: Open
-
Trivial
-
Resolution: Unresolved
-
1.3.0
-
None
-
None
-
spark-2.1
Description
1.create carbonsession
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.CarbonSession._
val carbon = SparkSession.builder().config(sc.getConf) .getOrCreateCarbonSession("hdfs://localhost:54311/newCarbonStore","/tmp"
2.create a dataframe with carbonsession
import carbon.sqlContext.implicits._
carbon.sql("drop table if exists streamingtable");
val df =carbon.sparkContext.parallelize(1 to 5).toDF("colId")
3.register dataframe as carbon streaming table
df.write.format("carbondata").option("tableName","streamingTable").option("streaming","true").mode(SaveMode.Overwrite).save
4,desc formatted the table
carbon.sql("describe formatted streamingTable").show(100)
--------------------------------------------------------
col_name | data_type | comment |
--------------------------------------------------------
colid ... | int ... | MEASURE,null ... |
... | ... | ... |
##Detailed Table ... | ... | ... |
Database Name ... | default ... | ... |
Table Name ... | streamingtable ... | ... |
CARBON Store Path... | hdfs://localhost:... | ... |
Comment ... | ... | ... |
Table Block Size ... | 1024 MB ... | ... |
Table Data Size ... | 316 ... | ... |
Table Index Size ... | 283 ... | ... |
Last Update Time ... | 1515393447642 ... | ... |
SORT_SCOPE ... | LOCAL_SORT ... | LOCAL_SORT ... |
Streaming ... | false ... | ... |
SORT_SCOPE ... | LOCAL_SORT ... | LOCAL_SORT ... |
... | ... | ... |
##Detailed Column... | ... | ... |
ADAPTIVE ... | ... | ... |
SORT_COLUMNS ... | ... | ... |
--------------------------------------------------------