[SPARK-47085] Preformance issue on thrift API

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.1, 3.5.0
• Fix Version/s: 4.0.0, 3.5.2, 3.4.3
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Izek Greenfield
• Reporter: Izek Greenfield
• Created: 18/Feb/24 08:10
• Updated: 26/Feb/24 05:34
• Resolved: 19/Feb/24 07:57

Description

This new complexity was introduced in SPARK-39041.

before this PR the code was:

while (curRow < maxRows && iter.hasNext) {
  val sparkRow = iter.next()
  val row = ArrayBuffer[Any]()
  var curCol = 0
  while (curCol < sparkRow.length) {
    if (sparkRow.isNullAt(curCol)) {
      row += null
    } else {
      addNonNullColumnValue(sparkRow, row, curCol, timeFormatters)
    }
    curCol += 1
  }
  resultRowSet.addRow(row.toArray.asInstanceOf[Array[Object]])
  curRow += 1
}

 foreach without the O(n^2) complexity so this change just return the state to what it was before.

 

In class `RowSetUtils` there is a loop that has O(n^2) complexity:

...
 while (i < rowSize) {
          val row = rows(I)
          ...

It can be easily converted back into O( n ) complexity.

 

 

Issue Links

• is caused by: SPARK-39041 Mapping Spark Query ResultSet/Schema to TRowSet/TTableSchema directly
• links to: GitHub Pull Request #45155
• links to: GitHub Pull Request #45164
• links to: GitHub Pull Request #45165