Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29427

Create KeyValueGroupedDataset in a relational way

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.4.4
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      The scenario I'm having is that I'm reading two huge bucketed tables and since a regular join is not performant enough for my cases, I'm using groupByKey to generate two KeyValueGroupedDatasets and cogroup them to implement the merging logic I need.

      The issue with this approach is that I'm only grouping by the column that the tables are bucketed by but since I'm using groupByKey the bucketing is completely ignored and I still get a full shuffle.
      What I'm looking for is some functionality to tell Catalyst to group by a column in a relational way but then give the user a possibility to utilize the functions of the KeyValueGroupedDataset e.g. cogroup (which is not available for dataframes)

       

      At current spark (2.4.4) I see no way to do this efficiently. I think this is a valid use case which if solved would have huge performance benefits.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              hagerf Alexander Hagerf
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: