Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5670

Local RocksDB directories not cleaned up

    Details

      Description

      After cancelling a job with a RocksDB backend all files are properly cleaned up, but the parent directories still exist and are empty:

      859546fec3dac36bb9fcc8cbdd4e291e
      +- StreamFlatMap_3_0
      +- StreamFlatMap_3_3
      +- StreamFlatMap_3_4
      +- StreamFlatMap_3_5
      +- StreamFlatMap_3_6
      

      The number of empty folders varies between runs.

        Issue Links

          Activity

          Hide
          aljoscha Aljoscha Krettek added a comment -

          The problem is that we create to many subdirectories and RocksDBKeyedStateBackend.dispose will only clean up the last part of this. I recently fixed that for 1.1 here: https://github.com/apache/flink/commit/e1861db7f71d8583c40b358ee474c77210c761f9.

          I think we should also get this into Flink 1.2. What do you think Robert Metzger? I know of at least one big user of Flink for which this was a potential problem.

          Show
          aljoscha Aljoscha Krettek added a comment - The problem is that we create to many subdirectories and RocksDBKeyedStateBackend.dispose will only clean up the last part of this. I recently fixed that for 1.1 here: https://github.com/apache/flink/commit/e1861db7f71d8583c40b358ee474c77210c761f9 . I think we should also get this into Flink 1.2. What do you think Robert Metzger ? I know of at least one big user of Flink for which this was a potential problem.
          Hide
          uce Ufuk Celebi added a comment -

          Looks like an easy fix. Big +1 to fix it for 1.2 as well.

          Show
          uce Ufuk Celebi added a comment - Looks like an easy fix. Big +1 to fix it for 1.2 as well.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user aljoscha opened a pull request:

          https://github.com/apache/flink/pull/3225

          FLINK-5670 Properly clean up local RocksDB directories

          We have to change the instance path to not include too many nested
          directories, otherwise the Keyed backend cannot properly clean up the
          whole directory hierachy.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/aljoscha/flink fix-rocksdb-cleanup

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3225.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3225


          commit 8d0b3ed5437ddb09f4cef376d8d7433b7b4e27b6
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2017-01-27T11:32:21Z

          FLINK-5670 Properly clean up local RocksDB directories

          We have to change the instance path to not include too many nested
          directories, otherwise the Keyed backend cannot properly clean up the
          whole directory hierachy.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/3225 FLINK-5670 Properly clean up local RocksDB directories We have to change the instance path to not include too many nested directories, otherwise the Keyed backend cannot properly clean up the whole directory hierachy. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink fix-rocksdb-cleanup Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3225.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3225 commit 8d0b3ed5437ddb09f4cef376d8d7433b7b4e27b6 Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2017-01-27T11:32:21Z FLINK-5670 Properly clean up local RocksDB directories We have to change the instance path to not include too many nested directories, otherwise the Keyed backend cannot properly clean up the whole directory hierachy.
          Hide
          rmetzger Robert Metzger added a comment -

          I think this should go into 1.2.0 (Since the merge window is open again for a few hours right now )

          Show
          rmetzger Robert Metzger added a comment - I think this should go into 1.2.0 (Since the merge window is open again for a few hours right now )
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha commented on the issue:

          https://github.com/apache/flink/pull/3225

          @uce, @StefanRRichter Could you please have a look, I think it would be good to have that in Flink 1.2.

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/3225 @uce, @StefanRRichter Could you please have a look, I think it would be good to have that in Flink 1.2.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StefanRRichter commented on the issue:

          https://github.com/apache/flink/pull/3225

          +1 LGTM

          Show
          githubbot ASF GitHub Bot added a comment - Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3225 +1 LGTM
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha commented on the issue:

          https://github.com/apache/flink/pull/3225

          @uce are you checking this with the Job where you initially discovered it?

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/3225 @uce are you checking this with the Job where you initially discovered it?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user uce commented on the issue:

          https://github.com/apache/flink/pull/3225

          Just verified it with the job. Works fine now! All directories were cleaned up properly.

          Show
          githubbot ASF GitHub Bot added a comment - Github user uce commented on the issue: https://github.com/apache/flink/pull/3225 Just verified it with the job. Works fine now! All directories were cleaned up properly.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha commented on the issue:

          https://github.com/apache/flink/pull/3225

          Thanks 👍 😃

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/3225 Thanks 👍 😃
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha closed the pull request at:

          https://github.com/apache/flink/pull/3225

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha closed the pull request at: https://github.com/apache/flink/pull/3225
          Hide
          aljoscha Aljoscha Krettek added a comment -

          Implemented on release-1.2:
          78f090212bab6355919cf03dca3ef672911e987a

          Implemented on master:
          54b70927575d22520b12fb8111324b8f77187822

          Show
          aljoscha Aljoscha Krettek added a comment - Implemented on release-1.2: 78f090212bab6355919cf03dca3ef672911e987a Implemented on master: 54b70927575d22520b12fb8111324b8f77187822

            People

            • Assignee:
              aljoscha Aljoscha Krettek
              Reporter:
              uce Ufuk Celebi
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development