Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33507 Improve and fix cache behavior in v1 and v2
  3. SPARK-34062

Call updateTableStats() from AlterTableAddPartitionCommand

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Won't Do
    • 3.2.0
    • None
    • SQL
    • None

    Description

      AlterTableAddPartitionCommand.run does almost the same as `CommandUtils.updateTableStats` while updating table stats. The code:

          if (table.stats.nonEmpty) {
            if (sparkSession.sessionState.conf.autoSizeUpdateEnabled) {
              val addedSize = CommandUtils.calculateMultipleLocationSizes(sparkSession, table.identifier,
                parts.map(_.storage.locationUri)).sum
              if (addedSize > 0) {
                val newStats = CatalogStatistics(sizeInBytes = table.stats.get.sizeInBytes + addedSize)
                catalog.alterTableStats(table.identifier, Some(newStats))
              }
            } else {
              catalog.alterTableStats(table.identifier, None)
            }
          }
      

      can be replaced by:

      CommandUtils.updateTableStats()
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            maxgekk Max Gekk
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment