Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-8497

Merge index does not mark the Directories it creates as 'done' and they are retained in the Directory cache.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 4.10.3
    • Fix Version/s: 6.0
    • Labels:
      None
    • Environment:

      Cloudera Search (CDH 5.4.5 Solr 4.10.3)

      Description

      After a Merge Indexes, the input directories on HDFS do not get closed (only released by the CachingDirectoryFactory). This causes the HDFSLocalityReporter to continue monitoring the input directories even after they are cleaned up/deleted. This results in a large volume of logged warnings on the Solr node.

        Activity

        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        That HDFSLocalityReporter has a really ugly performance bug currently I think. It was disabled by default after it was added. You probably want to move to a version that it allows it to be turned off.

        the input directories on HDFS do not get closed (only released by the CachingDirectoryFactory).

        Hmm...perhaps we just also need to mark them as done and we are not? I'll take a look when I can.

        Show
        markrmiller@gmail.com Mark Miller added a comment - That HDFSLocalityReporter has a really ugly performance bug currently I think. It was disabled by default after it was added. You probably want to move to a version that it allows it to be turned off. the input directories on HDFS do not get closed (only released by the CachingDirectoryFactory). Hmm...perhaps we just also need to mark them as done and we are not? I'll take a look when I can.
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        This should be the patch we need.

        Show
        markrmiller@gmail.com Mark Miller added a comment - This should be the patch we need.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 7d32456efa4ade0130c3ed0ae677aa47b29355a9 in lucene-solr's branch refs/heads/master from markrmiller
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7d32456 ]

        SOLR-8497: Merge indexes should mark it's directories as done rather than keep them around in the directory cache.

        Show
        jira-bot ASF subversion and git services added a comment - Commit 7d32456efa4ade0130c3ed0ae677aa47b29355a9 in lucene-solr's branch refs/heads/master from markrmiller [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7d32456 ] SOLR-8497 : Merge indexes should mark it's directories as done rather than keep them around in the directory cache.
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        Thanks Sivlio!

        Show
        markrmiller@gmail.com Mark Miller added a comment - Thanks Sivlio!
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        I don't think this is quite right yet. We should only mark the directory as done if it's not already being used by Solr I think.

        Show
        markrmiller@gmail.com Mark Miller added a comment - I don't think this is quite right yet. We should only mark the directory as done if it's not already being used by Solr I think.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 54e7bb5f58931cef9ead049313804c2b9a10ce88 in lucene-solr's branch refs/heads/master from markrmiller
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=54e7bb5 ]

        SOLR-8497: Only mark diectory as done if it was not previously in the cache.

        Show
        jira-bot ASF subversion and git services added a comment - Commit 54e7bb5f58931cef9ead049313804c2b9a10ce88 in lucene-solr's branch refs/heads/master from markrmiller [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=54e7bb5 ] SOLR-8497 : Only mark diectory as done if it was not previously in the cache.
        Hide
        anshumg Anshum Gupta added a comment -

        Mark Miller is this worth back porting to 5.5.1? This is a clean cherry-pick but I'll let you decide on this one.

        Show
        anshumg Anshum Gupta added a comment - Mark Miller is this worth back porting to 5.5.1? This is a clean cherry-pick but I'll let you decide on this one.

          People

          • Assignee:
            markrmiller@gmail.com Mark Miller
            Reporter:
            silviosanchezthe3rd Sivlio Sanchez
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development