Lucene - Core
  1. Lucene - Core
  2. LUCENE-5636

SegmentCommitInfo continues to list unneeded gen'd files

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.9, 6.0
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      I thought I handled it in LUCENE-5246, but turns out I didn't handle it fully. I'll upload a patch which improves the test to expose the bug. I know where it is, but I'm not sure how to fix it without breaking index back-compat. Can we do that on experimental features?

      The problem is that if you update different fields in different gens, the FieldInfos files of older gens remain referenced (still!!). I open a new issue since LUCENE-5246 is already resolved and released, so don't want to mess up our JIRA...

      The severity of the bug is that unneeded files are still referenced in the index. Everything still works correctly, it's just that .fnm files are still there. But as I wrote, I'm still not sure how to solve it without requiring apps that use dv updates to reindex.

        Issue Links

          Activity

          Hide
          Shai Erera added a comment -

          Patch improves the test to expose the bug.

          Show
          Shai Erera added a comment - Patch improves the test to expose the bug.
          Hide
          Shai Erera added a comment -

          Hmm, actually I think I can fix it w/o breaking index back-compat, by manually removing the old .fnm files in SegCommitInfo (I'll also improve how this is done in general). Assuming very little apps, if at all, write their own FieldInfosFormat, I think that's acceptable? I'll work on a patch later.

          Show
          Shai Erera added a comment - Hmm, actually I think I can fix it w/o breaking index back-compat, by manually removing the old .fnm files in SegCommitInfo (I'll also improve how this is done in general). Assuming very little apps, if at all, write their own FieldInfosFormat, I think that's acceptable? I'll work on a patch later.
          Hide
          Shai Erera added a comment -

          I chatted with Robert about this. The current situation is that the old .fnm files continue to be referenced even when not needed, however when the segment is merged, they go away (as all gen'd files). Given that there's no way to solve it without breaking back-compat, unless we introduce hacks such as checking for a ".fnm" suffix, we discussed how to solve this "going forward".

          By "going forward" I mean to not change existing segments, but if they contain future updates, write the new information in a better way. Perhaps old .fnm files will still be referenced by those segments, until they're merged away, but new segments will fix that bug.

          I think that this might be doable together with LUCENE-5618, by writing per-field gen'd DV file, so I'll try to solve it there and if it works I'll resolve that issue as appropriate.

          Show
          Shai Erera added a comment - I chatted with Robert about this. The current situation is that the old .fnm files continue to be referenced even when not needed, however when the segment is merged, they go away (as all gen'd files). Given that there's no way to solve it without breaking back-compat, unless we introduce hacks such as checking for a ".fnm" suffix, we discussed how to solve this "going forward". By "going forward" I mean to not change existing segments, but if they contain future updates, write the new information in a better way. Perhaps old .fnm files will still be referenced by those segments, until they're merged away, but new segments will fix that bug. I think that this might be doable together with LUCENE-5618 , by writing per-field gen'd DV file, so I'll try to solve it there and if it works I'll resolve that issue as appropriate.
          Hide
          ASF subversion and git services added a comment -

          Commit 1596570 from Shai Erera in branch 'dev/trunk'
          [ https://svn.apache.org/r1596570 ]

          LUCENE-5618, LUCENE-5636: write each DocValues update in a separate file; stop referencing old fieldInfos files

          Show
          ASF subversion and git services added a comment - Commit 1596570 from Shai Erera in branch 'dev/trunk' [ https://svn.apache.org/r1596570 ] LUCENE-5618 , LUCENE-5636 : write each DocValues update in a separate file; stop referencing old fieldInfos files
          Hide
          ASF subversion and git services added a comment -

          Commit 1596582 from Shai Erera in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1596582 ]

          LUCENE-5618, LUCENE-5636: write each DocValues update in a separate file; stop referencing old fieldInfos files

          Show
          ASF subversion and git services added a comment - Commit 1596582 from Shai Erera in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1596582 ] LUCENE-5618 , LUCENE-5636 : write each DocValues update in a separate file; stop referencing old fieldInfos files
          Hide
          Shai Erera added a comment -

          Fixed in LUCENE-5618

          Show
          Shai Erera added a comment - Fixed in LUCENE-5618

            People

            • Assignee:
              Shai Erera
              Reporter:
              Shai Erera
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development