Uploaded image for project: 'Bahir'
  1. Bahir
  2. BAHIR-228

Flink SQL supports kudu sink

    XMLWordPrintableJSON

    Details

      Description

      currently, for Flink-1.10.0, we can use the catalog to store our stream table sink for kudu, it should exist a kudu table sink so we can register it to catalog, and use kudu as a table in SQL environment.

      we can use kudu table sink like this:

      KuduOptions options = KuduOptions.builder() 
                  .setKuduMaster(kuduMaster) 
                  .setTableName(kuduTable) 
                  .build(); 
      KuduWriterOptions writerOptions = KuduWriterOptions.builder()          .setWriteMode(KuduWriterMode.UPSERT) 
                  .setFlushMode(FlushMode.AUTO_FLUSH_BACKGROUND) 
                  .build(); 
      KuduTableSink tableSink = KuduTableSink.builder() 
                  .setOptions(options) 
                  .setWriterOptions(writerOptions) 
                  .setTableSchema(schema) 
                  .build(); 
      tEnv.registerTableSink("kudu", tableSink);  
      tEnv.sqlUpdate("insert into kudu select * from source");
      

      I have used kudu table sink to sync data in company's production environment, the writing  speed at 5w/s in upsert mode

        Attachments

          Activity

            People

            • Assignee:
              lsy dalongliu
              Reporter:
              lsy dalongliu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: