Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7373

Break out Directory.syncMetaData from FSDirectory.renameFile

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.0, 6.2
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Today, when you call FSDirectory.renameFile it also calls fsync on
      the directory.

      This is OK for Lucene's current usage of this method, to rename just
      the one segments_N file on commit.

      But in playing with adding NRT replication (LUCENE-5438) to the simple
      demo Lucene server (LUCENE-5376) I found that, on spinning disks, that
      fsync is very costly, because when copying over an NRT point, we write
      to N .tmp files and then rename many files (taking seconds) in the
      end.

      I think we should just deprecate/remove the existing method, and make a new
      rename method that does only renaming, and a separate
      syncMetaData to call fsync on the directory?

      1. LUCENE-7373.patch
        16 kB
        Michael McCandless

        Activity

        Hide
        mikemccand Michael McCandless added a comment -

        Here's a patch based on trunk, deprecating renameFile and adding
        new rename and syncMetaData methods. I'll remove the
        deprecations on pushing to master.

        Show
        mikemccand Michael McCandless added a comment - Here's a patch based on trunk, deprecating renameFile and adding new rename and syncMetaData methods. I'll remove the deprecations on pushing to master.
        Hide
        rcmuir Robert Muir added a comment -

        +1

        Show
        rcmuir Robert Muir added a comment - +1
        Hide
        thetaphi Uwe Schindler added a comment -

        Stromg +1
        I discussed that a while ago with Robert Muir, but at that time we said "keep it simple because we only use renameFile for committing". But now this changes. LGTM.

        Show
        thetaphi Uwe Schindler added a comment - Stromg +1 I discussed that a while ago with Robert Muir , but at that time we said "keep it simple because we only use renameFile for committing". But now this changes. LGTM.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 621a98faa29f11d59d1f9312f7d6674519db38f3 in lucene-solr's branch refs/heads/branch_6x from Mike McCandless
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=621a98f ]

        LUCENE-7373: deprecate Directory.renameFile, which both renamed and fsync'd the directory, replacing it with separate rename and syncMetaData methods

        Show
        jira-bot ASF subversion and git services added a comment - Commit 621a98faa29f11d59d1f9312f7d6674519db38f3 in lucene-solr's branch refs/heads/branch_6x from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=621a98f ] LUCENE-7373 : deprecate Directory.renameFile, which both renamed and fsync'd the directory, replacing it with separate rename and syncMetaData methods
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit df9efb8b6da5195d6c8454cb7fd9b91147cb93fd in lucene-solr's branch refs/heads/master from Mike McCandless
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=df9efb8 ]

        LUCENE-7373: deprecate Directory.renameFile, which both renamed and fsync'd the directory, replacing it with separate rename and syncMetaData methods

        Show
        jira-bot ASF subversion and git services added a comment - Commit df9efb8b6da5195d6c8454cb7fd9b91147cb93fd in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=df9efb8 ] LUCENE-7373 : deprecate Directory.renameFile, which both renamed and fsync'd the directory, replacing it with separate rename and syncMetaData methods
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 3125fccd8dde211567c49a6fe508f80ace3b053b in lucene-solr's branch refs/heads/master from Mike McCandless
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3125fcc ]

        LUCENE-7373: remove deprecated API

        Show
        jira-bot ASF subversion and git services added a comment - Commit 3125fccd8dde211567c49a6fe508f80ace3b053b in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3125fcc ] LUCENE-7373 : remove deprecated API
        Hide
        mikemccand Michael McCandless added a comment -

        Bulk close resolved issues after 6.2.0 release.

        Show
        mikemccand Michael McCandless added a comment - Bulk close resolved issues after 6.2.0 release.

          People

          • Assignee:
            mikemccand Michael McCandless
            Reporter:
            mikemccand Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development