Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12629

SparkR: DataFrame's saveAsTable method has issues with the signature and HiveContext

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.1, 2.0.0
    • Component/s: SparkR
    • Labels:
      None

      Description

      There are several issues with the DataFrame's saveAsTable method in SparkR. Here is a summary of some of them. Hope this will help to fix the issues.

      1. According to SparkR's saveAsTable(...) documentation, we can call the "saveAsTable(df, "myfile")" in order to store the dataframe.
      However, this signature isn't working. It seems that "source" and "mode" are forced according to signature.
      2. Within the method saveAsTable(...) it tries to retrieve the SQL context and tries to create/initialize source as parquet, but this is also failing because the context has to be Hive Context. Based on the error messages I see.
      3. In general the method fails when I try to call it with sqlContext
      4. Also, it seems that SQL DataFrame.saveAsTable is deprecated, we could use df.write.saveAsTable(...) instead ...

      Shivaram Venkataraman Sun Rui Felix Cheung

        Attachments

          Activity

            People

            • Assignee:
              Narine Narine Kokhlikyan
              Reporter:
              Narine Narine Kokhlikyan
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: