[NIFI-6663] ArrayIndexOutOfBoundsException in CaptureChangeMySQL

• Type: Improvement
• Status: Open
• Priority: Trivial
• Resolution: Unresolved
• Affects Version/s: 1.9.2
• Component/s: Documentation & Website, Extensions
• Assignee: Unassigned
• Reporter: Rahul Patil
• Created: 13/Sep/19 07:40
• Updated: 13/Sep/19 07:40

Description

• Current implementaion of CaptureChangeMySQL assumes binlog_row_image=FULL
  https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-cdc-mysql-nar/1.9.2/org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL/
• But CDC Processor throws ArrayIndexOutOfBoundsException when binlog_row_image is set to Minimal at
  https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-cdc/nifi-cdc-mysql-bundle/nifi-cdc-mysql-processors/src/main/java/org/apache/nifi/cdc/mysql/event/io/UpdateRowsWriter.java#L87

  2019-06-11 18:01:09,465 ERROR [Timer-Driven Process Thread-8] o.a.n.c.m.processors.CaptureChangeMySQL CaptureChangeMySQL[id=016b1000-dfcd-1189-97d7-6bebf706ec18] CaptureChangeMySQL[id=016b1000-dfcd-1189-97d7-
  6bebf706ec18] failed to process session due to java.lang.ArrayIndexOutOfBoundsException: 5; Processor Administratively Yielded for 1 sec: java.lang.ArrayIndexOutOfBoundsException: 5
  java.lang.ArrayIndexOutOfBoundsException: 5
          at org.apache.nifi.cdc.mysql.event.io.UpdateRowsWriter.writeRow(UpdateRowsWriter.java:87)
          at org.apache.nifi.cdc.mysql.event.io.UpdateRowsWriter.lambda$writeEvent$0(UpdateRowsWriter.java:57)
          at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2627)
          at org.apache.nifi.cdc.mysql.event.io.UpdateRowsWriter.writeEvent(UpdateRowsWriter.java:51)
          at org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.outputEvents(CaptureChangeMySQL.java:973)
          at org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.onTrigger(CaptureChangeMySQL.java:619)
          at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
          at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
          at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
          at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748) 

• This is because of obvious reason of MySQL's way of writing row images 
  https://dev.mysql.com/doc/refman/5.7/en/replication-options-binary-log.html#sysvar_binlog_row_image
• Basically this is what I observed,

  binlog_format	binlog_row_image	Primary Key	CDC Processor
  ROW	FULL	YES/NO	OK
  ROW	MINIMAL	NO	OK
  ROW	MINIMAL	YES	NG

• Since not everyone who uses this processor is aware of underlying config & details, I was thinking of following fixes
  1. Include this in CaptureChangeMySQL processor description
  2. try to throw a meaningful error message
  3. Fix current implementation to support other binlog_row_image types