Nutch
  1. Nutch
  2. NUTCH-738

Close SegmentUpdater when FetchedSegments is closed

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      Currently FetchedSegments starts a SegmentUpdater, but never closes it when FetchedSegments is closed.

      (The problem was described in this mailing: http://www.mail-archive.com/nutch-user@lucene.apache.org/msg13823.html)

      1. NUTCH-738.patch
        2 kB
        Kirby Bohling
      2. FetchedSegments.patch
        1 kB
        Martina Koch

        Activity

        Hide
        Hudson added a comment -

        Integrated in Nutch-trunk #996 (See http://hudson.zones.apache.org/hudson/job/Nutch-trunk/996/)
        Close SegmentUpdater when FetchedSegments is closed.

        Show
        Hudson added a comment - Integrated in Nutch-trunk #996 (See http://hudson.zones.apache.org/hudson/job/Nutch-trunk/996/ ) Close SegmentUpdater when FetchedSegments is closed.
        Hide
        Andrzej Bialecki added a comment -

        Fixed in rev. 885150. Thank you!

        Show
        Andrzej Bialecki added a comment - Fixed in rev. 885150. Thank you!
        Hide
        Kirby Bohling added a comment -

        Alternative implementation, that uses Thread interruption as cancel semantics. Because I'm not sure that the project properly handles interruption everywhere, I capture the interruption request to ensure the shutdown is not lost.

        If this is not patch acceptable, I believe that the original patch needs to make the boolean used become volatile to conform to the Java Memory Model.

        The this.$0 in the SegmentUpdater, the segmentUpdater and the fact that segmentUpdater is a GC root (all threads are GC roots), appears to confuse the GC into not releasing this cluster of objects, thus holding this the class loader in memory.

        If segmentUpdater needs to stay final, just pass the values from FetchSegments and make SegmentUpdater become a static class. I believe that will also resolve the problem, but I didn't actually try it.

        Using this, and the patch from NUTCH-746, and adding "-XX:+CMSClassUnloadingEnabled" to Tomcat's 6.0.18 startup script will allow the classes to be unloaded.

        Show
        Kirby Bohling added a comment - Alternative implementation, that uses Thread interruption as cancel semantics. Because I'm not sure that the project properly handles interruption everywhere, I capture the interruption request to ensure the shutdown is not lost. If this is not patch acceptable, I believe that the original patch needs to make the boolean used become volatile to conform to the Java Memory Model. The this.$0 in the SegmentUpdater, the segmentUpdater and the fact that segmentUpdater is a GC root (all threads are GC roots), appears to confuse the GC into not releasing this cluster of objects, thus holding this the class loader in memory. If segmentUpdater needs to stay final, just pass the values from FetchSegments and make SegmentUpdater become a static class. I believe that will also resolve the problem, but I didn't actually try it. Using this, and the patch from NUTCH-746 , and adding "-XX:+CMSClassUnloadingEnabled" to Tomcat's 6.0.18 startup script will allow the classes to be unloaded.
        Hide
        Martina Koch added a comment -

        Patch adds close method for SegmentUpdater.

        Show
        Martina Koch added a comment - Patch adds close method for SegmentUpdater.

          People

          • Assignee:
            Andrzej Bialecki
            Reporter:
            Martina Koch
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development