[SPARK-34062] Call updateTableStats() from AlterTableAddPartitionCommand

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Do
• Affects Version/s: 3.2.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Max Gekk
• Created: 10/Jan/21 16:01
• Updated: 13/Jan/21 12:57
• Resolved: 13/Jan/21 12:57

Description

AlterTableAddPartitionCommand.run does almost the same as `CommandUtils.updateTableStats` while updating table stats. The code:

    if (table.stats.nonEmpty) {
      if (sparkSession.sessionState.conf.autoSizeUpdateEnabled) {
        val addedSize = CommandUtils.calculateMultipleLocationSizes(sparkSession, table.identifier,
          parts.map(_.storage.locationUri)).sum
        if (addedSize > 0) {
          val newStats = CatalogStatistics(sizeInBytes = table.stats.get.sizeInBytes + addedSize)
          catalog.alterTableStats(table.identifier, Some(newStats))
        }
      } else {
        catalog.alterTableStats(table.identifier, None)
      }
    }

can be replaced by:

CommandUtils.updateTableStats()

Issue Links

• is fixed by: SPARK-34084 ALTER TABLE .. ADD PARTITION does not update table stats
• links to: [Github] Pull Request #31114 (MaxGekk)
• links to: [Github] Pull Request #31114 (MaxGekk)