Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21740

DataFrame.write does not work with Phoenix JDBC Driver

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 2.0.0, 2.2.0
    • Fix Version/s: None
    • Component/s: Spark Core
    • Flags:
      Important

      Description

      The reason for this is that Phoenix JDBC driver does not support "INSERT", but "UPSERT".
      Exception for the following program:
      17/08/15 12:18:53 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1)
      org.apache.phoenix.exception.PhoenixParserException: ERROR 601 (42P00): Syntax error. Encountered "INSERT" at line 1, column 1.
      at org.apache.phoenix.exception.PhoenixParserException.newException(PhoenixParserException.java:33)

      public class HbaseJDBCSpark {
      
          private static final SparkSession sparkSession
                  = SparkSession.builder()
                          .config("spark.sql.warehouse.dir", "file:///temp")
                          .config("spark.driver.memory", "5g")
                          .master("local[*]").appName("Spark2JdbcDs").getOrCreate();
      
          static final String JDBC_URL
                  = "jdbc:phoenix:somehost:2181:/hbase-unsecure";
      
          public static void main(String[] args) {
              final Properties connectionProperties = new Properties();
              Dataset<Row> jdbcDF
                      = sparkSession.read()
                              .jdbc(JDBC_URL, "javatest", connectionProperties);
      
              jdbcDF.show();
      
              String url = JDBC_URL;
              Properties p = new Properties();
              p.put("driver", "org.apache.phoenix.jdbc.PhoenixDriver");
              //p.put("batchsize", "100000");
              jdbcDF.write().mode(SaveMode.Append).jdbc(url, "javatest", p);
              sparkSession.close();
      
          }
          // Create variables
      
      }
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              zwu.net@gmail.com Paul Wu
            • Votes:
              3 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: