Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21338

AggregatedDialect doesn't override isCascadingTruncateTable() method

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0, 2.1.1
    • Fix Version/s: 2.3.0
    • Component/s: SQL
    • Labels:
      None

      Description

      org.apache.spark.sql.jdbc.JdbcDialect's method:
      def isCascadingTruncateTable(): Option[Boolean] = None
      is not overriden in org.apache.spark.sql.jdbc.AggregatedDialect class.
      Because of this issue - when you add more than one dialect Spark doesn't truncate table because isCascadingTruncateTable always returns default None for Aggregated Dialect.
      Can be tracked when you write dataset in jdbc mode with SaveMode.Overwrite and extra JdbcDialect added via JdbcDialects.registerDialect(...); method.
      Then inside of org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider on line 65 in createRelation method it will never get to line 67 even if isTruncate returns true.
      It will only get inside if you have one JdbcDialect and AggregatedDialect is not used.
      Fix would be to overrite this method in AggregatedDialect and also to reduce value (I guess with and clause same as in overriden getHandle method) e.g.
      override def isCascadingTruncateTable(): Option[Boolean]=
      Some(dialects.map(.isCascadingTruncateTable()).reduce( && _))

        Attachments

          Activity

            People

            • Assignee:
              huaxing Huaxin Gao
              Reporter:
              ogonchar Ostap Gonchar
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: