[NIFI-5755] Allow PutParquet prosessor to dynamically set config properties for parquet lib

• Type: New Feature
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.7.1
• Component/s: Core Framework
• Assignee: Bryan Bende
• Reporter: Ken Tore Tallakstad
• Created: 26/Oct/18 04:41
• Updated: 04/Sep/19 12:53

Description

PutParquet (NiFi 1.7.1), will fail to write parquet if incoming records contain arrays with null elements: 
Schema snippet example:

{              "name": "myfield",              "type" : ["null",

{ "type" : "array", "items" : ["null","string"], "default": null }

], "default": null
 },
 
And a corresponding data example:
"myfield" : [ null, "value1" ],
"myfield" : [ null, "value2" ],
"myfield" : [ "value3", null, "value4" ],
 
Avro does not seem to have a problem with this, but Put parquet fails with the following error: "Array contains a null element at X".
 
There is a parquet config to allow this: 
parquet.avro.write-old-list-structure=false
 
If the processor would have a box for toggling this value, or a text box to pass configs in general would be great!