Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-12121

[Rust] [Parquet] Arrow writer benchmarks

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0
    • Rust

    Description

      The common concern with Parquet's Arrow readers and writers is that they're slow.
      My diagnosis is that we rely on a chain of processes, which introduces overhead.
      For example, writing an Arrow RecordBatch involves the following:

      1. Iterate through arrays to create def/rep levels
      2. Extract Parquet primitive values from arrays using these levels
      3. Write primitive values, validating them in the process (when they already should be validated)
      4. Split the already materialised values into small batches for Parquet chunks (consider where we have 1e6 values in a batch)
      5. Write these batches, computing the stats of each batch, and encoding values

      The above is as a side-effect of convenience, as it would likely require a lot more effort to bypass some of the steps.

      I have ideas around going from step 1 to 5 directly, but won't know if it's better if there aren't performance benchmarks. I also struggle to see if I'm making improvements while I clean up the writer code, especially removing the allocations that I created to reduce the complexity of the level calculations.

      With ARROW-12120 (random array & batch generator), it becomes more convenient to benchmark (and test many combinations of) the Arrow writer.

      I would thus like to start adding benchmarks for the Arrow writer.

      Attachments

        Issue Links

          Activity

            People

              nevi_me Neville Dipale
              nevi_me Neville Dipale
              Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h