Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-8774

[Rust] [DataFusion] Improve threading model

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: Rust, Rust - DataFusion
    • Labels:
      None

      Description

      DataFusion currently spawns one thread per partition and this results in poor performance if there are more partitions than available cores/threads. It would be better to have a thread-pool that defaults to number of available cores.

      Here is a Google doc where we can collaborate on a design discussion.

      https://docs.google.com/document/d/1_wc6diy3YrRgEIhVIGzrO5AK8yhwfjWlmKtGnvbsrrY/edit?usp=sharing

        Attachments

          Activity

            People

            • Assignee:
              andygrove Andy Grove
              Reporter:
              andygrove Andy Grove
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: