Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2536 Windows : Fix various unit test failures
  3. FLUME-2145

TestCheckpointRebuilder.testFastReplay fails on Windows due to checkpoint file being memory mapped

    Details

    • Type: Sub-task
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: v1.4.0
    • Fix Version/s: None
    • Component/s: Test, Windows
    • Labels:

      Description

      Memory mapped files are not released by JVM causing failure when attempting to clean the up.

      1. FLUME-2145.patch
        2 kB
        Roshan Naik
      2. FLUME-2145.v2.patch
        2 kB
        Roshan Naik

        Issue Links

          Activity

          Hide
          roshan_naik Roshan Naik added a comment -

          Changing files names so that memory mapped file is not same.

          Show
          roshan_naik Roshan Naik added a comment - Changing files names so that memory mapped file is not same.
          Hide
          roshan_naik Roshan Naik added a comment -

          Changing files names so that memory mapped file is not same.

          Show
          roshan_naik Roshan Naik added a comment - Changing files names so that memory mapped file is not same.
          Hide
          hshreedharan Hari Shreedharan added a comment -

          Roshan,

          This patch seems to be ok for the tests to pass, but it does look like it will leave all of the original files lurking around in the Windows case(since we use deleteQuietly in tearDown which will eat the exceptions). Can that be fixed somehow?

          Show
          hshreedharan Hari Shreedharan added a comment - Roshan, This patch seems to be ok for the tests to pass, but it does look like it will leave all of the original files lurking around in the Windows case(since we use deleteQuietly in tearDown which will eat the exceptions). Can that be fixed somehow?
          Hide
          roshan_naik Roshan Naik added a comment -

          I have seen that issue of some files being not deleted. AFAICT It is not specific to this test/patch.
          Here i have setup FileUtils.forceDeleteOnExit() inside the test as far as possible.

          I think the proper cleanup on windows needs to be addressed in another jira. agree ?

          Show
          roshan_naik Roshan Naik added a comment - I have seen that issue of some files being not deleted. AFAICT It is not specific to this test/patch. Here i have setup FileUtils.forceDeleteOnExit() inside the test as far as possible. I think the proper cleanup on windows needs to be addressed in another jira. agree ?
          Hide
          hshreedharan Hari Shreedharan added a comment -

          Roshan,

          I missed one thing yesterday. If you look at the test you can see that the channel is started using the same checkpoint directory later (after checkpoint rebuilder is done rebuilding). This issue with doing that is that the channel will restart with "checkpointDir/checkpoint" as the checkpoint and the new file created by checkpoint rebuilder is ignored. We should probably create the new checkpoint in a different directory and start the file channel with that directory as the checkpoint directory to work around this.

          As far as doing the windows specific clean up in a follow up - that sounds ok, but I'd like to make sure that if new directories (as I suggested above) are added to the tests, they get cleaned up when the test is done.

          Show
          hshreedharan Hari Shreedharan added a comment - Roshan, I missed one thing yesterday. If you look at the test you can see that the channel is started using the same checkpoint directory later (after checkpoint rebuilder is done rebuilding). This issue with doing that is that the channel will restart with "checkpointDir/checkpoint" as the checkpoint and the new file created by checkpoint rebuilder is ignored. We should probably create the new checkpoint in a different directory and start the file channel with that directory as the checkpoint directory to work around this. As far as doing the windows specific clean up in a follow up - that sounds ok, but I'd like to make sure that if new directories (as I suggested above) are added to the tests, they get cleaned up when the test is done.
          Hide
          roshan_naik Roshan Naik added a comment -

          Filed FLUME-2149

          Show
          roshan_naik Roshan Naik added a comment - Filed FLUME-2149
          Hide
          roshan_naik Roshan Naik added a comment -

          Incorporating review feedback from Hari to change checkpoint dir instead of checkpoint file

          Show
          roshan_naik Roshan Naik added a comment - Incorporating review feedback from Hari to change checkpoint dir instead of checkpoint file

            People

            • Assignee:
              roshan_naik Roshan Naik
              Reporter:
              roshan_naik Roshan Naik
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development