[PARQUET-1773] Parquet file in invalid state while writing to S3 when calling ParquetWriter.write

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.10.0
• Component/s: parquet-mr
• Assignee: Unassigned
• Reporter: Tristan Davolt
• Created: 20/Jan/20 21:55
• Updated: 22/Sep/20 23:07
• Resolved: 22/Sep/20 23:07

Description

This may be related to PARQUET-632. I am also writing parquet to S3, but I am calling ParquetWriter.write directly. I have multiple containerized instances consuming messages from Kafka, converting them to Parquet, and then writing to S3. One instance will begin to throw this exception for all new messages. Sometimes, the container will recover. Other times, it must be restarted manually to recover. I am unable to find any "error thrown previously."

Exception:
java.io.IOException
Message:
The file being written is in an invalid state. Probably caused by an error thrown previously. Current state: BLOCK
Stacktrace:

org.apache.parquet.hadoop.ParquetFileWriter$STATE.error(ParquetFileWriter.java:168)org.apache.parquet.hadoop.ParquetFileWriter$STATE.startBlock(ParquetFileWriter.java:160)org.apache.parquet.hadoop.ParquetFileWriter.startBlock(ParquetFileWriter.java:291)org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:171)org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:114)org.apache.parquet.hadoop.ParquetWriter.close(ParquetWriter.java:308)