[PARQUET-2477] Files opened by ParquetFileWriter’s appendFile method should to be closed correctly

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.13.1
• Component/s: parquet-hadoop
• Assignee: Unassigned
• Reporter: Deng An
• Created: 18/May/24 12:52
• Updated: 23/Jun/24 03:33

Description

I try to use the `appendFile` method of `ParquetFileWriter` to merge some smaller Parquet files into one large parquet file. After I finished the merge, I tried deleting the smaller files, but couldn't actually delete them.

After reading the ParquetFileWriter source code, I found that the file he opened was not closed in the end method, which seemed unreasonable.

 

When testing on `Windows`, it was found that these implicitly opened files cannot be deleted or renamed. This seems to be related to Windows file handles. But it's worked fine on macOS.

Anyway, I will submit a patch to ensure that files opened through the appendFile function are closed correctly.

Issue Links

• links to: GitHub Pull Request #1351
• links to: GitHub Pull Request #1351