Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26684

Add logs when allocating large memory for PooledByteBufAllocator

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Trivial
    • Resolution: Invalid
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: Shuffle
    • Labels:
      None

      Description

      Currently, Spark use `PooledByteBufAllocator` to allocate memory for channel reading. However, the allocated heap/offheap memory size is not tracked. Sometimes, this make it difficult to  find out the cause of OOM failures(for instance, direct memory oom). we have to dump the heap and use more advanced tools like MAT to locate the cause.

      Actually, we can add some logs for `PooledByteBufAllocator` when allocating large memory, which can facilitate the debugging.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                liupengcheng liupengcheng
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: