[KYLIN-3462] "dfs.replication=2" and compression not work in Spark cube engine

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v2.3.0, v2.3.1, v2.4.0
• Fix Version/s: v2.5.0
• Component/s: Spark Engine
• Assignee: Shao Feng Shi
• Reporter: Shao Feng Shi
• Created: 23/Jul/18 05:47
• Updated: 17/Sep/18 00:59
• Resolved: 02/Aug/18 06:12

Description

In a comparison between Spark and MR cubing, I noticed the cuboid files that Spark engine generated is 3x lager than MR, and took 4x larger more disk on HDFS than MR.

 

The reason is, the "dfs.replication=2" didn't work when Spark save to HDFS. And by default no compression for spark.

 

The converted HFiles are in the same size, the query results are the same. So this difference may easily be overlooked.