[CARBONDATA-4151] When data sampling is done on large data set using Spark's df.sample function - the size of sampled table is not matching with record size of non sampled (Raw Table)

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 2.0.1
• Fix Version/s: 2.1.0, 2.0.1
• Component/s: core
• Environment: Apache carbondata 2.0.1, spark 2.4.5, hadoop 2.7.2
• Flags: Patch
• Assignee: Unassigned
• Reporter: Amaranadh Vayyala
• Created: 16/Mar/21 15:33
• Updated: 23/Mar/21 07:08

Description

Hi Team,

When we are performing 5%, 10% data sampling on large dataset using Spark's df.sample - the size of sampled table is not matching with record size of non sampled (Raw Table).

Our Raw table size is around 11 GB, so when we perform 5%, 10% sampling then the sampled table size should come as 550 MB, 1.1 GB. However in our case they are coming as 1.5 GB and 3 GB. Which is 3 times higher than the expected number. 

Could you please check and help us in understand where is the issue?