Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1276

Add a vectorized read/write interface for pandas DataFrame objects

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: client, python
    • Labels:
      None

      Description

      A pandas read/write interface would make Kudu significantly easier to use for average Python data users.

      The layering is as follows:

      • Writer: "Vectorized" insert that accepts a C/C++ array of values plus an array (either bits or bytes) indicating nullness for nullable slots
      • Reader: Converts a row batch to NumPy arrays with missing data representation suitable for use in pandas. Ideally should not create more than one PyString object for each observed string value. Binary can be encoded as UTF8 string, while Timestamp will need to be converted to nanoseconds for pandas

      This would also give a very performant and relatively GIL-free data ingest path to the Kudu (and Kudu consumers like Impala) without a great deal of Python+Cython coding.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jtbirdsell Jordan Birdsell
                Reporter:
                wesm Wes McKinney
              • Votes:
                3 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated: