Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25593

JDBC write Impala, `truncate` true option in Overwrite mode for JDBC DataFrameWriter is dropping and creating the table instead of truncating.

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 2.3.2
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:

      Description

      Scenario :

      Reading data from Impala using jdbc (cloudera jdbc41 driver) and writing to Impala with command write.mode(SaveMode.Overwrite).option("truncate", true)

      Observed:
      It's dropping and trying to create a new table. failing with exception

      CAUSED BY: Exception: Syntax error
      ), Query: CREATE TABLE jdbc_spark.persons_write_200 ("personid" INTEGER , "lastname" TEXT , "firstname" TEXT , "address"
      TEXT , "city" TEXT ).

      Expected :
      It should only truncate the table.

      Note : With SaveMode.Append it's working absolutely fine.

      relates to SPARK-16463

        Attachments

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              Unassigned Assign to me
              Reporter:
              rkesh_singh rakesh
              Shepherd:
              Dongjoon Hyun

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment