[DAFFODIL-2455] Large CSV file causes "Attempting to backtrack too far" exception

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1.0
• Assignee: Steve Lawrence
• Reporter: Steve Lawrence
• Created: 14/Jan/21 14:09
• Updated: 02/Feb/21 16:20
• Resolved: 19/Jan/21 15:10

Description

A user reports that they have an ~800MB CSV file and a schema to parse that file that looks something like this:

  <xs:element name="file">
    <xs:complexType>
      <xs:sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
        <xs:element name="row" maxOccurs="unbounded">
          <xs:complexType>
            <xs:sequence dfdl:separator=",">
              <xs:element name="field1" type="xs:string"/>
              <xs:element name="field2" type="xs:string"/>
              ...
              <xs:element name="fieldN" type="xs:string"/>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>

Parsing this large file fails with the exception:

[error] Runtime Schema Definition Error: Attempted to backtrack too far: Attempted to backtrack to byte 268427263, which exceeds maximum backtrack length of 268435456

This schema does not require any backtracking, so shouldn't be hitting this issue.