Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-18554

Memory exceeds taskmanager.memory.process.size when enabling mmap_read for RocksDB

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Not A Bug
    • Affects Version/s: 1.11.0
    • Fix Version/s: None
    • Labels:
      None

      Description

      We are testing Flink automatic memory management feature on Flink 1.11. However, YARN kept killing our containers due to the processes' physical memory exceeds the limit, although we have tuned the following configurations:

      taskmanager.memory.process.size
      taskmanager.memory.managed.fraction
      

      We suspect that it's because we have enabled mmap_read for RocksDB, since turning this options off seems to fix the issue. Maybe Flink automatic memory management is unable to account for the addition memory required when using mmap_read ?

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kien_truong Truong Duc Kien
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: