Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11309

Clean up hacky use of MemoryManager inside of HashedRelation

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • None
    • None
    • SQL

    Description

      In HashedRelation, there's a hacky creation of a new MemoryManager in order to handle broadcasting of BytesToBytesMap: https://github.com/apache/spark/blob/85e654c5ec87e666a8845bfd77185c1ea57b268a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L323

      Something similar to this has existed for a while, but the code recently became much messier as an indirect consequence of my memory manager consolidation patch. We should see about cleaning this up and removing the hack.

      Attachments

        Activity

          People

            Unassigned Unassigned
            joshrosen Josh Rosen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: