Solr
  1. Solr
  2. SOLR-978

Old files are not removed from slaves after replication

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4
    • Component/s: replication (java)
    • Labels:
      None

      Description

      The files in the slave's index is not removed even after they are removed from master.
      The temp directories named index.xxxxxxx are not removed after use

      The problem is that when we do a commit on the slave after replication is done. The commit does not re-open the IndexWriter. Therefore, the deletion policy does not take affect and older files are left as is. This can keep on building up. The only solution is to re-open the index writer.

      A user has reported the issue http://markmail.org/thread/yw5n4dk2t5zbt5z5

      1. SOLR-978.patch
        6 kB
        Shalin Shekhar Mangar

        Activity

        Hide
        Shalin Shekhar Mangar added a comment -

        Changes

        1. Add a forceOpenWriter method to DirectUpdateHandler2 which is called by SnapPuller on the slave to make sure IndexWriter is open. This is to make sure IndexDeletionPolicy gets a chance to cleanup older commit points.
        2. SnapPuller calls forceOpenWriter if core.getUpdateHandler is an instance of DirectUpdateHandler2.
        3. SnapPuller calls delTree on temporary files and folders in a finally block

        I know 2 is not a very clean way. I thought of adding that method in UpdateHandler as a no-op and overriding it in DUH2. However that would mean that custom UpdateHandler implementations would silently ignore this call and we will not have a chance to warn them (which we can do with this patch). Also, DUH2 already has a protected openWriter method. However calling that method needs access to a protected lock object.

        Show
        Shalin Shekhar Mangar added a comment - Changes Add a forceOpenWriter method to DirectUpdateHandler2 which is called by SnapPuller on the slave to make sure IndexWriter is open. This is to make sure IndexDeletionPolicy gets a chance to cleanup older commit points. SnapPuller calls forceOpenWriter if core.getUpdateHandler is an instance of DirectUpdateHandler2. SnapPuller calls delTree on temporary files and folders in a finally block I know 2 is not a very clean way. I thought of adding that method in UpdateHandler as a no-op and overriding it in DUH2. However that would mean that custom UpdateHandler implementations would silently ignore this call and we will not have a chance to warn them (which we can do with this patch). Also, DUH2 already has a protected openWriter method. However calling that method needs access to a protected lock object.
        Hide
        Shalin Shekhar Mangar added a comment -

        Committed revision 737550.

        Thanks Jaco and Noble!

        Show
        Shalin Shekhar Mangar added a comment - Committed revision 737550. Thanks Jaco and Noble!
        Hide
        Grant Ingersoll added a comment -

        Bulk close Solr 1.4 issues

        Show
        Grant Ingersoll added a comment - Bulk close Solr 1.4 issues

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            Noble Paul
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development