Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17047

Spark 2 cannot create table when CLUSTERED.

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0, 2.1.1, 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: SQL
    • Labels:
      None
    • Flags:
      Important

      Description

      This does not work with CLUSTERED BY clause in Spark 2 now!

      CREATE TABLE test.dummy2
      (
      ID INT
      , CLUSTERED INT
      , SCATTERED INT
      , RANDOMISED INT
      , RANDOM_STRING VARCHAR(50)
      , SMALL_VC VARCHAR(10)
      , PADDING VARCHAR(10)
      )
      CLUSTERED BY (ID) INTO 256 BUCKETS
      STORED AS ORC
      TBLPROPERTIES ( "orc.compress"="SNAPPY",
      "orc.create.index"="true",
      "orc.bloom.filter.columns"="ID",
      "orc.bloom.filter.fpp"="0.05",
      "orc.stripe.size"="268435456",
      "orc.row.index.stride"="10000" )

      scala> HiveContext.sql(sqltext)
      org.apache.spark.sql.catalyst.parser.ParseException:
      Operation not allowed: CREATE TABLE ... CLUSTERED BY(line 2, pos 0)

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mich Dr Mich Talebzadeh

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment