Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5180 Data source API improvement (Spark 1.5)
  3. SPARK-5603

Preinsert casting and renaming rule is needed in the Analyzer

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 1.3.0
    • SQL
    • None
    • Spark 1.5 release

    Description

      For an INSERT INTO/OVERWRITE statement, we should add necessary Cast and Alias to the output of the query.

      CREATE TEMPORARY TABLE jsonTable (a int, b string)
      USING org.apache.spark.sql.json.DefaultSource
      OPTIONS (
        path '...'
      )
      INSERT OVERWRITE TABLE jsonTable SELECT a * 2, a * 4 FROM table
      

      For a*2, we should create an Alias, so the InsertableRelation can know it is the column a. For a*4, it is actually the column b in jsonTable. We should first cast it to StringType and add an Alias b to it.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            yhuai Yin Huai
            yhuai Yin Huai
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Agile

                Completed Sprint:
                Spark 1.5 release ended 14/Aug/15
                View on Board

                Slack

                  Issue deployment