[PARQUET-134] Enhance ParquetWriter with file creation flag

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Assignee: Mariappan Asokan
• Reporter: Mariappan Asokan
• Created: 15/Nov/14 00:35
• Updated: 09/Mar/15 17:44
• Resolved: 05/Mar/15 02:27

Description

Currently, Parquet files can be created only with "create in exclusive mode" (like O_CREAT|O_EXCL in UNIX open system call.) This may be inconvenient in a situation where user knows that overwriting an existing file is okay. This is especially true for Parquet files created on a local disk. For example, a user might want to specify a named pipe (on UNIX) as the target, encrypt the bytes on the fly by reading from the named pipe in another process, and write to disk. The named pipe file has to exist first.

I am thinking that we can have OVERWRITE and CREATE modes for now. If appending to Parquet files is supported in the future, we can add APPEND mode later. These mode flags can be defined as constants in ParquetWriter.java.

There are 7 constructors in ParquetWriter class. I am thinking of adding one more that takes the mode flag (and all the arguments of the constructor with the most arguments today.) Also, a new constructor of ParquetFileWriter will take the mode flag as an argument and the following statement

this.out = fs.create(file, false);

will be modified so that for OVERWRITE mode, the second argument to create() will be set to true.

I can submit a patch with the above changes and a test. Committers, please give your feedback with suggestions.