Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-19005

Document JDBC drivers as source of Metaspace leaks

    XMLWordPrintableJSON

    Details

      Description

      Hi !

      Im running a 1.11.1 flink cluster, where I execute batch jobs made with DataSet API.

      I submit these jobs every day to calculate daily data.

      In every execution, cluster's used metaspace increase by 7MB and its never released.

      This ends up with an OutOfMemoryError caused by Metaspace every 15 days and i need to restart the cluster to clean the metaspace

      taskmanager.memory.jvm-metaspace.size is set to 512mb

      Any idea of what could be causing this metaspace grow and why is it not released ?

       

      ================================================
      === Summary ======================================
      ================================================

      Case 1, reported by Guillermo Sánchez:

      • Flink 1.11.1
      • Java 11
      • Maximum Metaspace size set to 512mb
      • Custom Batch job, submitted daily
      • Requires restart every 15 days after an OOM

       Case 2, reported by Echo Lee:

      • Flink 1.11.0
      • Java 11
      • G1GC
      • WordCount Batch job, submitted every second / every 5 minutes
      • eventually fails TaskExecutor with OOM

      Case 3, reported by ShenDa

      • Flink 1.11.0
      • Java 11
      • WordCount Batch job, submitted every 5 seconds
      • growing Metaspace, eventually OOM
         

        Attachments

        1. heap_dump_after_10_executions.zip
          23.43 MB
          Guillermo Sánchez
        2. heap_dump_after_1_execution.zip
          19.96 MB
          Guillermo Sánchez
        3. heap_dump_echo_lee.tar.xz
          47.57 MB
          Echo Lee
        4. origin-jdbc-inputformat.png
          22 kB
          ShenDa
        5. modified-jdbc-inputformat.png
          27 kB
          ShenDa
        6. heapdump-after_metaspace_dropped.hprof
          53.29 MB
          Guillermo Sánchez

          Issue Links

            Activity

              People

              • Assignee:
                chesnay Chesnay Schepler
                Reporter:
                gestevez Guillermo Sánchez
              • Votes:
                1 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: