[FLINK-16371] HadoopCompressionBulkWriter fails with 'java.io.NotSerializableException'

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.10.0
• Fix Version/s: 1.10.1, 1.11.0
• Component/s: Connectors / FileSystem
• Labels: pull-request-available
• Assignee: Sivaprasanna Sethuraman
• Reporter: Sivaprasanna Sethuraman
• Created: 02/Mar/20 10:17
• Updated: 10/Mar/20 11:49
• Resolved: 10/Mar/20 11:49

Description

When using CompressWriterFactory with Hadoop compression codec, the execution fails with java.io.NotSerializableException. 

I guess this is probably to do with the the instance creation for Hadoop's CompressionCodec being done here at CompressWriterFactory.java#L59 and thus it has to be sent over the wire causing the exception to be thrown.

So I did a quick test on my end by changing the way the CompressionCodec is initialised and ran it on a Hadoop cluster, and it has been working just fine. Will raise a PR in a day or so.

Issue Links

• links to: GitHub Pull Request #11307