Solr
  1. Solr
  2. SOLR-3749

Default syncLevel cannot be configured by solrconfig.xml for updateLog(transaction log)

    Details

      Description

      In solr 4.0 environment, transaction log had been defined in three level, none/flush/fsync. The updateLog hard code the default sync level is SyncLevel.FLUSH.
      If user want to use the other levels, have to rewrite the RunUpdateProcess, to set the level.
      At best, user can set it in the solrconfig.xml, that it is easy to control and use.

      BTW, transaction log is very important for solr cloud, at best, invoke the sync to make sure kernel memory submit into the disk to avoid some corner case that maybe damage transaction log.

      1. configpatch
        0.5 kB
        Raintung Li
      2. patch.txt
        1 kB
        Raintung Li

        Activity

        Hide
        Mark Miller added a comment -

        +1

        Show
        Mark Miller added a comment - +1
        Hide
        Mark Miller added a comment -

        BTW, transaction log is very important for solr cloud, at best, invoke the sync to make sure kernel memory submit into the disk to avoid some corner case that maybe damage transaction log.

        The basic idea is that because you have replicas, fsync is less important - if there is a problem, you can recover from another node. The danger is when the whole shard goes out at once and all of the replicas lose updates - fsync every update is going to be fairly expensive to cover that kind of rare disaster (not only did the whole shard go down, but every node is missing some updates), but it should be available with config.

        Show
        Mark Miller added a comment - BTW, transaction log is very important for solr cloud, at best, invoke the sync to make sure kernel memory submit into the disk to avoid some corner case that maybe damage transaction log. The basic idea is that because you have replicas, fsync is less important - if there is a problem, you can recover from another node. The danger is when the whole shard goes out at once and all of the replicas lose updates - fsync every update is going to be fairly expensive to cover that kind of rare disaster (not only did the whole shard go down, but every node is missing some updates), but it should be available with config.
        Hide
        Mark Miller added a comment -

        And even more importantly, it should be configurable for the single node case...

        Show
        Mark Miller added a comment - And even more importantly, it should be configurable for the single node case...
        Hide
        Raintung Li added a comment - - edited

        Agree, we can recover from replicas, and all of replicas goes out for one shard that is low probability. I suggest the leader server implement the high strictly level fsync, and the replicas can defined FLUSH. As you mention, maybe only single node case. However can be configurable that is more usability. Will it apply the patch in the new version?

        Show
        Raintung Li added a comment - - edited Agree, we can recover from replicas, and all of replicas goes out for one shard that is low probability. I suggest the leader server implement the high strictly level fsync, and the replicas can defined FLUSH. As you mention, maybe only single node case. However can be configurable that is more usability. Will it apply the patch in the new version?
        Hide
        Hoss Man added a comment -

        assigning to mark to assess/commit for 4.0 since it appears he's already reviewed the patch and given a +1?

        Show
        Hoss Man added a comment - assigning to mark to assess/commit for 4.0 since it appears he's already reviewed the patch and given a +1?
        Hide
        Mark Miller added a comment -

        Committed - thanks Raintung Li!

        Show
        Mark Miller added a comment - Committed - thanks Raintung Li!
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Mark Robert Miller
        http://svn.apache.org/viewvc?view=revision&revision=1385151

        SOLR-3749: Allow default UpdateLog syncLevel to be configured by solrconfig.xml.

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1385151 SOLR-3749 : Allow default UpdateLog syncLevel to be configured by solrconfig.xml.
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Mark Robert Miller
        http://svn.apache.org/viewvc?view=revision&revision=1385151

        SOLR-3749: Allow default UpdateLog syncLevel to be configured by solrconfig.xml.

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1385151 SOLR-3749 : Allow default UpdateLog syncLevel to be configured by solrconfig.xml.
        Hide
        Uwe Schindler added a comment -

        Closed after release.

        Show
        Uwe Schindler added a comment - Closed after release.

          People

          • Assignee:
            Mark Miller
            Reporter:
            Raintung Li
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 24h
              24h
              Remaining:
              Remaining Estimate - 24h
              24h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development