Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11792

SizeEstimator cannot provide a good size estimation of UnsafeHashedRelations

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 1.6.0
    • SQL
    • None

    Description

      At the executor side, SizeEstimator sees that an UnsafeHashedRelation has a huge graph and produce a significant larger estimated size (compared with the size of data stored in the UnsafeHashedRelation). So, the memory manger will "think" we run out of memory after several broad cast joins.

      Attachments

        Activity

          People

            yhuai Yin Huai
            yhuai Yin Huai
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: