Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5486

Compliant text reader tries, but fails, to ignore empty rows

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.10.0
    • Fix Version/s: None
    • Component/s: Storage - Text & CSV
    • Labels:
      None

      Description

      The "compliant" text reader (for CSV files, etc.) has two modes: reading with headers (which creates a scalar vector per file column), or without headers (that creates a single columns vector with an array of all columns.

      When run in array mode, the code uses a class called RepeatedVarCharOutput. This class attempts to ignore empty records:

        public void finishRecord() {
          ...
          // if there were no defined fields, skip.
          if (fieldIndex > -1) {
            batchIndex++;
            recordCount++;
          }
          ...
      

      As it turns out, this works only on the first row. On the first row, the fieldIndex has its initial value of -1. But, for subsequent records, fieldIndex is never reset and retains its value from the last field set.

      The result is that the code skips the first empty row, but not empty rows elsewhere in the file.

      Further, on the first row, the logic is flawed. The part shown above is only for the row counts. Here is more of the logic:

        @Override
        public void finishRecord() {
          recordStart = characterData;
          ...
          int newOffset = ((int) (charLengthOffset - charLengthOffsetOriginal))/4;
          PlatformDependent.putInt(repeatedOffset, newOffset);
          repeatedOffset += 4;
      
          // if there were no defined fields, skip.
          if (fieldIndex > -1) {
            batchIndex++;
            recordCount++;
          }
        }
      

      Note that the underlying vector is bumped for the record, even though the row count is not. Later, this will cause the container to have one less record.

      This would be a problem if the row count (batchIndex) was used for the batch row count. But, it is not, the next level of code keeps a separate count, so the "skip empty row" logic causes the counts to get out of sync between the RepeatedVarCharOutput and the next level of reader.

      In sense, there are multiple self-cancelling bugs:

      • The skip-empty-row logic is not synchronized with the upper layer.
      • That bug is partly masked because the logic is not triggered except on the first row.
      • That bug is masked because we always set the row offset, even if we ignore the row in the row count.
      • That bug is masked because the incorrect row count is ignored when setting the container row count.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                paul-rogers Paul Rogers
                Reporter:
                paul-rogers Paul Rogers
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: