[SPARK-35640] Refactor Parquet vectorized reader to remove duplicated code paths

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 04/Jun/21 00:58
• Updated: 13/Oct/21 22:45
• Resolved: 11/Jun/21 05:40

Description

Currently in Parquet vectorized code path, there are many code duplications such as the following:

  public void readIntegers(
      int total,
      WritableColumnVector c,
      int rowId,
      int level,
      VectorizedValuesReader data) throws IOException {
    int left = total;
    while (left > 0) {
      if (this.currentCount == 0) this.readNextGroup();
      int n = Math.min(left, this.currentCount);
      switch (mode) {
        case RLE:
          if (currentValue == level) {
            data.readIntegers(n, c, rowId);
          } else {
            c.putNulls(rowId, n);
          }
          break;
        case PACKED:
          for (int i = 0; i < n; ++i) {
            if (currentBuffer[currentBufferIdx++] == level) {
              c.putInt(rowId + i, data.readInteger());
            } else {
              c.putNull(rowId + i);
            }
          }
          break;
      }
      rowId += n;
      left -= n;
      currentCount -= n;
    }
  }

This makes it hard to maintain as any change on this will need to be replicated in 20+ places. The issue becomes more serious when we are going to implement column index and complex type support for the vectorized path.

The original intention is for performance. However now days JIT compilers tend to be smart on this and will inline virtual calls as much as possible.

Issue Links

• links to: [Github] Pull Request #32777 (sunchao)