[NIFI-9203] Improve GrokReader to be able to handle complex Grok expressions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.15.0
• Assignee: Tamas Palfy
• Reporter: Tamas Palfy
• Created: 08/Sep/21 12:41
• Updated: 17/Sep/21 14:26
• Resolved: 17/Sep/21 14:26

Description

The current GrokReader implementation cannot handle complex expressions like in the following scenario:

Suppose we have a custom Grok pattern file:

SYSLOGBASE_ISO8601 %{TIMESTAMP_ISO8601:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:
LINE_1 %{SYSLOGBASE}%{GREEDYDATA:message}
LINE_2 %{SYSLOGBASE_ISO8601}%{GREEDYDATA:message}
LINE (?:%{LINE_1}|%{LINE_2})

If we set the Grok expression to:

%LINE

the service will fail for 2 reasons:

1. LINE_1 and LINE_2 define the same labels. The service will try to create a schema by adding fields for all labels encountered. This leads to duplicate fields in the schema which is not allowed.
2. When the used Grok library reads a record based on a complex expression it returns an array as a value as the complex expression can have multiple matches. NiFi in turn tries to handle it as a byte array.

Issue Links

• links to: GitHub Pull Request #5376