Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2255

Customized configuration on KuduClient via KuduContext

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.4.0
    • None
    • client, spark
    • None
    • Kudu1.4.0; Spark2; Cloudera 12.5

    Description

      It seems that kuduClient can be created with customized configuration (e.g. timeout, etc.) where as there is no configuration can be passed if I use KuduContext class. KuduContext will created a kuduClient by using its default configuration settings.

      This is my scenarios, I have a Spark application, which connects to Kudu and tries to do insert / update / delete on Kudu table. I am using KuduContext.class to create a kuduContext instance and its (a)syncClient. upsertRows() method in the KuduContext class is the method that I often call to insert / update data to Kudu table.

      Basing on my basic understanding, upsertRows() method is calling writePartitionRows() method, which will create a new kudu session from Kudu syncClient. Since it does NOT allow me to set customised value for rpc timeout, it will always use the default one (30000 millisecond). The default value works for some queries, but might not work for long-running queries, which needs to insert a large resultset to Kudu table.

      Do we have a way to set customised configuration values when using KuduContext class?

      According to the Kudu configuration guide (https://kudu.apache.org/docs/configuration_reference.html), I would expect these configuration parameters can be also set in Java API.

      Could you please let me know if I miss anything? If it is not the right jira ticket to ask, could you please guide me to raise this issue?

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              mdu_sydney Min Du
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: