[PARQUET-1587] Parquet Writer doesn't support to write list of Groups directly into hadoop

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 1.10.1
• Component/s: parquet-mr
• Assignee: Unassigned
• Reporter: Udit Joshi
• Created: 29/May/19 04:17
• Updated: 29/Jan/22 04:47

Description

Problem :-  Parquet Writer doesn't support to write list of Groups directly into hadoop.

Cause of Problem :- Suitable classes are not available
Constructors are not visible

All possible solution which i tried -

• ParquetWriter
• ParquetFileWriter
• ExampleParquetWriter
• ParquetRecordWriter

Gradle dependency

compile group: 'org.apache.parquet', name: 'parquet-hadoop', version: '1.10.1'

 

Is there any way to writer parquet file into Hadoop with GroupSupport?

 

Problem description

I am trying to create the object of ParquetWriter class which accepts the argument (OutputFile, Mode, WriteSupport, CompressionCodecName, int, boolean, Configuration, int, ParquetProperties). But this constructor has default access modifier. I can't able to access it.

 I also used ParquetFileWriter class but it doesn't show any group support.

 

ParquetWriter

 ParquetFileWriter

ParquetFileWriter writer = new ParquetFileWriter(HadoopOutputFile.fromPath(writePathFile, configuration), schema, Mode.CREATE, DEFAULT_BLOCK_SIZE, MAX_PADDING_SIZE_DEFAULT) ;
 writer.start();

 

 It creates  to create parquet file in hadoop but can't facilitates to write  List of Groups in parquet file

 

ExampleParquetWriter

ParquetWriter<Group> writer = ExampleParquetWriter.builder(writePathFile).withConf(configuration).withType(getSchema()).build(); 
 System.out.println("Number of groups to write:" + groups.size());
 for (Group g : groups) {
 writer.write(g);
 } 

 Doesn't support for writing  parquet file in hadoop