Solr
  1. Solr
  2. SOLR-6640

Replication can cause index corruption.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 5.0
    • Fix Version/s: 5.0, 6.0
    • Component/s: replication (java)
    • Labels:
      None

      Description

      Test failure found on jenkins:
      http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11333/

      1 tests failed.
      REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
      
      Error Message:
      shard2 is not consistent.  Got 62 from http://127.0.0.1:57436/collection1lastClient and got 24 from http://127.0.0.1:53065/collection1
      
      Stack Trace:
      java.lang.AssertionError: shard2 is not consistent.  Got 62 from http://127.0.0.1:57436/collection1lastClient and got 24 from http://127.0.0.1:53065/collection1
              at __randomizedtesting.SeedInfo.seed([F4B371D421E391CD:7555FFCC56BCF1F1]:0)
              at org.junit.Assert.fail(Assert.java:93)
              at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1255)
              at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1234)
              at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:162)
              at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
      

      Cause of inconsistency is:

      Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch, expected segment id=yhq3vokoe1den2av9jbd3yp8, got=yhq3vokoe1den2av9jbd3yp7 (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-F4B371D421E391CD-001/tempDir-001/jetty3/index/_1_2.liv")))
         [junit4]   2> 		at org.apache.lucene.codecs.CodecUtil.checkSegmentHeader(CodecUtil.java:259)
         [junit4]   2> 		at org.apache.lucene.codecs.lucene50.Lucene50LiveDocsFormat.readLiveDocs(Lucene50LiveDocsFormat.java:88)
         [junit4]   2> 		at org.apache.lucene.codecs.asserting.AssertingLiveDocsFormat.readLiveDocs(AssertingLiveDocsFormat.java:64)
         [junit4]   2> 		at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:102)
      
      1. corruptindex.log
        4.47 MB
        Mark Miller
      2. Lucene-Solr-5.x-Linux-64bit-jdk1.8.0_20-Build-11333.txt
        889 kB
        Shalin Shekhar Mangar
      3. SOLR-6640_new_index_dir.patch
        12 kB
        Varun Thacker
      4. SOLR-6640.patch
        8 kB
        Shalin Shekhar Mangar
      5. SOLR-6640.patch
        4 kB
        Shalin Shekhar Mangar
      6. SOLR-6640.patch
        3 kB
        Varun Thacker
      7. SOLR-6640.patch
        36 kB
        Varun Thacker
      8. SOLR-6640-test.patch
        4 kB
        Shalin Shekhar Mangar
      9. SOLR-6920.patch
        16 kB
        Varun Thacker

        Issue Links

          Activity

          Hide
          Shalin Shekhar Mangar added a comment -

          The test logs from jenkins are attached.

          Show
          Shalin Shekhar Mangar added a comment - The test logs from jenkins are attached.
          Hide
          Robert Muir added a comment -

          This exception translates to: "this file _1_2.liv is mismatched and does not belong to this segment _1, instead it belongs to another segment (likely _0)"

          You can see via the suppressed checksum exception from the full stacktrace that the contents are themselves consistent and not corrupt (checksum passes), its just mismatched.

          Show
          Robert Muir added a comment - This exception translates to: "this file _1_2.liv is mismatched and does not belong to this segment _1, instead it belongs to another segment (likely _0)" You can see via the suppressed checksum exception from the full stacktrace that the contents are themselves consistent and not corrupt (checksum passes), its just mismatched.
          Hide
          Shalin Shekhar Mangar added a comment -

          Thanks Robert. In order to make sure that we are actually downloading files belonging to the correct segment, we need access to the segment version but it doesn't seem like there's an easy way to read them? Can the segment ids be exposed in the IndexCommit object? Also, can we provide a custom segment id instead of having Lucene generate them? As it stands today, we have no way of controlling the segment ids generated by replicas and therefore I'm afraid that replica recovery might end up downloading the complete index.

          Show
          Shalin Shekhar Mangar added a comment - Thanks Robert. In order to make sure that we are actually downloading files belonging to the correct segment, we need access to the segment version but it doesn't seem like there's an easy way to read them? Can the segment ids be exposed in the IndexCommit object? Also, can we provide a custom segment id instead of having Lucene generate them? As it stands today, we have no way of controlling the segment ids generated by replicas and therefore I'm afraid that replica recovery might end up downloading the complete index.
          Hide
          Robert Muir added a comment -

          In order to make sure that we are actually downloading files belonging to the correct segment, we need access to the segment version but it doesn't seem like there's an easy way to read them? Can the segment ids be exposed in the IndexCommit object?

          SegmentInfos infos = SegmentInfos.readCommit(directory, commit.getSegmentsFileName());

          Also, can we provide a custom segment id instead of having Lucene generate them?

          No: this is an internal safety mechanism from lucene. Its detecting that solr replicates the wrong file (again: in this case, the wrong deletes generation). Apparently, we cannot entrust apps to not mismatch up our index files.

          As it stands today, we have no way of controlling the segment ids generated by replicas and therefore I'm afraid that replica recovery might end up downloading the complete index.

          Maybe instead of worrying about optimizing this, its best to start with "not corrupting the index".

          Show
          Robert Muir added a comment - In order to make sure that we are actually downloading files belonging to the correct segment, we need access to the segment version but it doesn't seem like there's an easy way to read them? Can the segment ids be exposed in the IndexCommit object? SegmentInfos infos = SegmentInfos.readCommit(directory, commit.getSegmentsFileName()); Also, can we provide a custom segment id instead of having Lucene generate them? No: this is an internal safety mechanism from lucene. Its detecting that solr replicates the wrong file (again: in this case, the wrong deletes generation). Apparently, we cannot entrust apps to not mismatch up our index files. As it stands today, we have no way of controlling the segment ids generated by replicas and therefore I'm afraid that replica recovery might end up downloading the complete index. Maybe instead of worrying about optimizing this, its best to start with "not corrupting the index".
          Hide
          Varun Thacker added a comment -

          This is interesting - When I was running the test a few weeks back it was reproducible all the time on my local machine . I ran it a few times against branch_5x right now and it passed every time.

          ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=F4B371D421E391CD -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=pt_BR -Dtests.timezone=Africa/Douala -Dtests.file.encoding=US-ASCII
          

          I am digging more into what changed. Anyways I will upload a patch I have on my machine which uses the lucene segment-ids to verify what files need to be downloaded.

          Show
          Varun Thacker added a comment - This is interesting - When I was running the test a few weeks back it was reproducible all the time on my local machine . I ran it a few times against branch_5x right now and it passed every time. ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=F4B371D421E391CD -Dtests.multiplier=3 -Dtests.slow= true -Dtests.locale=pt_BR -Dtests.timezone=Africa/Douala -Dtests.file.encoding=US-ASCII I am digging more into what changed. Anyways I will upload a patch I have on my machine which uses the lucene segment-ids to verify what files need to be downloaded.
          Hide
          Varun Thacker added a comment -

          Rough patch.

          compareFile() checks if the file is present and also if the length and checksum for each file matches . Used instead of slowFileExists.

          Small refactoring -
          Removed createTempindexDir
          moveAFile() never utilized List<String> copiedfiles. So I removed it.
          Modified the DirectoryFileFetcher ctor to take only the necessary params
          In DirectoryFileFetcher saveAs was always the fileName. So I removed it.

          We need to see if we need to download all files in a segment when they are out of sync as opposed to just downloading the missing files as we do currently.

          Show
          Varun Thacker added a comment - Rough patch. compareFile() checks if the file is present and also if the length and checksum for each file matches . Used instead of slowFileExists. Small refactoring - Removed createTempindexDir moveAFile() never utilized List<String> copiedfiles. So I removed it. Modified the DirectoryFileFetcher ctor to take only the necessary params In DirectoryFileFetcher saveAs was always the fileName. So I removed it. We need to see if we need to download all files in a segment when they are out of sync as opposed to just downloading the missing files as we do currently.
          Hide
          Mark Miller added a comment -

          Cool - this is a very important issue to solve.

          Show
          Mark Miller added a comment - Cool - this is a very important issue to solve.
          Hide
          Varun Thacker added a comment -

          This test passes with the patch - ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=EDA082CD42EB33E3 -Dtests.slow=true -Dtests.locale=hi_IN -Dtests.timezone=Asia/KuchingDtests.asserts=true -Dtests.file.encoding=ISO-8859-1

          The patch does something very simple - Before we begin to download segment files, check against the current commit point which files are extra and remove them.

          For example -

          -001/jetty3/index.20141216162501726
             [junit4]   2> 22194 T113 C7 P59775 oash.SnapPuller.fetchLatestIndex SOLR-6640:: indexDir.listAll() pre remove _0.cfe _0.cfs _0.si _0_1.liv _1.fdt _1.fdx segments_1 
             [junit4]   2> 22195 T79 C6 P59766 oasup.LogUpdateProcessor.finish [collection1] webapp=/_ path=/update params={wt=javabin&version=2} {add=[0-19 (1487664160941539328)]} 0 1
             [junit4]   2> 22196 T113 C7 P59775 oash.SnapPuller.fetchLatestIndex SOLR-6640:: indexDir.listAll() post remove segments_1 
          

          So it's these files which are not getting removed when we do IW.rollback that were causing the problem -
          _0.cfe _0.cfs _0.si _0_1.liv _1.fdt _1.fdx

          I am yet to figure out whether these files should have been removed by IW.rollback() or not?

          Show
          Varun Thacker added a comment - This test passes with the patch - ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=EDA082CD42EB33E3 -Dtests.slow=true -Dtests.locale=hi_IN -Dtests.timezone=Asia/KuchingDtests.asserts=true -Dtests.file.encoding=ISO-8859-1 The patch does something very simple - Before we begin to download segment files, check against the current commit point which files are extra and remove them. For example - -001/jetty3/index.20141216162501726 [junit4] 2> 22194 T113 C7 P59775 oash.SnapPuller.fetchLatestIndex SOLR-6640:: indexDir.listAll() pre remove _0.cfe _0.cfs _0.si _0_1.liv _1.fdt _1.fdx segments_1 [junit4] 2> 22195 T79 C6 P59766 oasup.LogUpdateProcessor.finish [collection1] webapp=/_ path=/update params={wt=javabin&version=2} {add=[0-19 (1487664160941539328)]} 0 1 [junit4] 2> 22196 T113 C7 P59775 oash.SnapPuller.fetchLatestIndex SOLR-6640:: indexDir.listAll() post remove segments_1 So it's these files which are not getting removed when we do IW.rollback that were causing the problem - _0.cfe _0.cfs _0.si _0_1.liv _1.fdt _1.fdx I am yet to figure out whether these files should have been removed by IW.rollback() or not?
          Hide
          Shalin Shekhar Mangar added a comment - - edited

          I am looking at this failure too and I see another bug. I was wondering why the replica had these writes in the first place considering that recovery on startup had not completed.

          1. RecoveryStrategy publishes the state of the replica as 'recovering' before it sets the update log to buffering mode which is why the leader sends updates to this replica that affect the index.
          2. The test itself doesn't wait for a steady state e.g. by calling waitForRecovery or waitForThingsToLevelOut before starting the indexing threads. This is probably a good thing because that's what has helped us find this problem.
          3. Shouldn't the peersync also be done while update log is set to buffering mode?

          So it's these files which are not getting removed when we do IW.rollback that were causing the problem -
          _0.cfe _0.cfs _0.si _0_1.liv _1.fdt _1.fdx
          I am yet to figure out whether these files should have been removed by IW.rollback() or not?

          These files hang around because an IndexReader is open using the IndexWriter due to soft commit(s).

          Show
          Shalin Shekhar Mangar added a comment - - edited I am looking at this failure too and I see another bug. I was wondering why the replica had these writes in the first place considering that recovery on startup had not completed. RecoveryStrategy publishes the state of the replica as 'recovering' before it sets the update log to buffering mode which is why the leader sends updates to this replica that affect the index. The test itself doesn't wait for a steady state e.g. by calling waitForRecovery or waitForThingsToLevelOut before starting the indexing threads. This is probably a good thing because that's what has helped us find this problem. Shouldn't the peersync also be done while update log is set to buffering mode? So it's these files which are not getting removed when we do IW.rollback that were causing the problem - _0.cfe _0.cfs _0.si _0_1.liv _1.fdt _1.fdx I am yet to figure out whether these files should have been removed by IW.rollback() or not? These files hang around because an IndexReader is open using the IndexWriter due to soft commit(s).
          Hide
          Shalin Shekhar Mangar added a comment -

          So what is the right way to implement partial replication in this case? Force deleting the file (Varun's patch) probably won't work on windows and/or not play well with the open searchers. In SolrCloud we could just close the searcher before rollback because a replica in recovery won't get any search requests but that's not practical in standalone Solr because it'd cause downtime.

          Show
          Shalin Shekhar Mangar added a comment - So what is the right way to implement partial replication in this case? Force deleting the file (Varun's patch) probably won't work on windows and/or not play well with the open searchers. In SolrCloud we could just close the searcher before rollback because a replica in recovery won't get any search requests but that's not practical in standalone Solr because it'd cause downtime.
          Hide
          Shalin Shekhar Mangar added a comment -

          I had a discussion with Varun about this issue. We have two problems here

          1. Solr corrupts the index during replication recovery
          2. Such a corrupt index puts Solr into an infinite recovery loop

          For #1 the problem is clear – we have open searchers on uncommitted/flushed files which are mixed with files from the leader causing corruption.

          Possible solutions for #1 are either a) switch to a different index dir and move/copy files from committed segments and use the index.properties approach to open a searcher on the new index dir or b) close the searcher then rollback the writer and then download the necessary files.

          Closing the searcher.... is not as simple as it sounds because the searcher is ref counted and close() doesn't really close immediately. Also, at any time, a request might open a new searcher so it is a very involved change.

          For #2, every where we open a reader/searcher or writer, we should be ready to handle the corrupt index exceptions.

          I think we should first try to first solve the problem of corrupting the index. So let's try the deletion approach that Varun outlined. If that fails then we should switch to a new index dir, move/copy over files from commit points, fetch the missing segments from the leader and use the index.properties approach to completely move to a new index directory.

          The second problem that we need to solve is that a corrupted index trashes the server. We should be able to recover from such a scenario instead of going into an infinite recovery loop.

          Let's fix these two problems (in that order) and then figure out ways to optimize recovery.

          Longer term we need to change our code such that we can close the searchers, rollback the writer and delete uncommitted files and then attempt replication recovery.

          Also my earlier comment on non-cloud Solr was wrong:

          In SolrCloud we could just close the searcher before rollback because a replica in recovery won't get any search requests but that's not practical in standalone Solr because it'd cause downtime.

          In stand alone Solr this is not a problem because indexing and soft-commits do not happen on slaves. But anyway changing to close the searcher etc is a big change.

          Show
          Shalin Shekhar Mangar added a comment - I had a discussion with Varun about this issue. We have two problems here Solr corrupts the index during replication recovery Such a corrupt index puts Solr into an infinite recovery loop For #1 the problem is clear – we have open searchers on uncommitted/flushed files which are mixed with files from the leader causing corruption. Possible solutions for #1 are either a) switch to a different index dir and move/copy files from committed segments and use the index.properties approach to open a searcher on the new index dir or b) close the searcher then rollback the writer and then download the necessary files. Closing the searcher.... is not as simple as it sounds because the searcher is ref counted and close() doesn't really close immediately. Also, at any time, a request might open a new searcher so it is a very involved change. For #2, every where we open a reader/searcher or writer, we should be ready to handle the corrupt index exceptions. I think we should first try to first solve the problem of corrupting the index. So let's try the deletion approach that Varun outlined. If that fails then we should switch to a new index dir, move/copy over files from commit points, fetch the missing segments from the leader and use the index.properties approach to completely move to a new index directory. The second problem that we need to solve is that a corrupted index trashes the server. We should be able to recover from such a scenario instead of going into an infinite recovery loop. Let's fix these two problems (in that order) and then figure out ways to optimize recovery. Longer term we need to change our code such that we can close the searchers, rollback the writer and delete uncommitted files and then attempt replication recovery. Also my earlier comment on non-cloud Solr was wrong: In SolrCloud we could just close the searcher before rollback because a replica in recovery won't get any search requests but that's not practical in standalone Solr because it'd cause downtime. In stand alone Solr this is not a problem because indexing and soft-commits do not happen on slaves. But anyway changing to close the searcher etc is a big change.
          Hide
          Varun Thacker added a comment -

          This patch uses the following approach - Download all missing files from leader into the tempIndexDir and then copy over the remaining files from the current index directory into the tempIndexDir. Now use the index.properties method to switch to the new index.

          Main changes are in moveIndexFiles() method.

          Still very rough and needs refactoring, but wanted to share this approach and get some thoughts on it.

          Show
          Varun Thacker added a comment - This patch uses the following approach - Download all missing files from leader into the tempIndexDir and then copy over the remaining files from the current index directory into the tempIndexDir. Now use the index.properties method to switch to the new index. Main changes are in moveIndexFiles() method. Still very rough and needs refactoring, but wanted to share this approach and get some thoughts on it.
          Hide
          Shalin Shekhar Mangar added a comment -

          Here's a better patch which:

          1. Re-opens IW after rolling back the previous one
          2. Re-opens the searcher using the new IW (we can't re-open the old one because it was created using the previous IW which will be closed on rollback)
          3. Calls IW.deleteUnusedFiles repeatedly with a 1 second sleep until all unused files have been cleaned up. This may take some time because the old searcher or a virus scanner may be using these files.
          4. If we could not clean up files within 30 seconds then we fall back to a full replication
          Show
          Shalin Shekhar Mangar added a comment - Here's a better patch which: Re-opens IW after rolling back the previous one Re-opens the searcher using the new IW (we can't re-open the old one because it was created using the previous IW which will be closed on rollback) Calls IW.deleteUnusedFiles repeatedly with a 1 second sleep until all unused files have been cleaned up. This may take some time because the old searcher or a virus scanner may be using these files. If we could not clean up files within 30 seconds then we fall back to a full replication
          Hide
          Mark Miller added a comment -

          a searcher might be using some flushed but committed segments

          shouldn't that be 'not committed'?

          Show
          Mark Miller added a comment - a searcher might be using some flushed but committed segments shouldn't that be 'not committed'?
          Hide
          Shalin Shekhar Mangar added a comment -

          shouldn't that be 'not committed'?

          Umm yeah but who are you quoting?

          Show
          Shalin Shekhar Mangar added a comment - shouldn't that be 'not committed'? Umm yeah but who are you quoting?
          Hide
          Mark Miller added a comment -

          I'm quoting from the patch that was attached yesterday.

          Show
          Mark Miller added a comment - I'm quoting from the patch that was attached yesterday.
          Hide
          Shalin Shekhar Mangar added a comment -

          Ah, okay, yes you're right. I'll fix that comment.

          Show
          Shalin Shekhar Mangar added a comment - Ah, okay, yes you're right. I'll fix that comment.
          Hide
          Varun Thacker added a comment -

          Shouldn't the logic be a try-finally block instead of try-with-resource ?

          try {
                      IndexWriter indexWriter = writer.get();
                      int c = 0;
                      indexWriter.deleteUnusedFiles();
                      while (hasUnusedFiles(indexDir, commit)) {
                        indexWriter.deleteUnusedFiles();
                        LOG.info("Sleeping for 1000ms to wait for unused lucene index files to be delete-able");
                        Thread.sleep(1000);
                        c++;
                        if (c >= 30)  {
                          LOG.warn("SnapPuller unable to cleanup unused lucene index files so we must do a full copy instead");
                          isFullCopyNeeded = true;
                          break;
                        }
                      }
                      if (c > 0)  {
                        LOG.info("SnapPuller slept for " + (c * 1000) + "ms for unused lucene index files to be delete-able");
                      }
                    } finally {
                      if (writer != null) {
                        writer.decref();
                      }
                    }
          
          Show
          Varun Thacker added a comment - Shouldn't the logic be a try-finally block instead of try-with-resource ? try { IndexWriter indexWriter = writer.get(); int c = 0; indexWriter.deleteUnusedFiles(); while (hasUnusedFiles(indexDir, commit)) { indexWriter.deleteUnusedFiles(); LOG.info( "Sleeping for 1000ms to wait for unused lucene index files to be delete-able" ); Thread .sleep(1000); c++; if (c >= 30) { LOG.warn( "SnapPuller unable to cleanup unused lucene index files so we must do a full copy instead" ); isFullCopyNeeded = true ; break ; } } if (c > 0) { LOG.info( "SnapPuller slept for " + (c * 1000) + "ms for unused lucene index files to be delete-able" ); } } finally { if (writer != null ) { writer.decref(); } }
          Hide
          Mark Miller added a comment -

          Something similar to this corrupted index issue also causes this test to timeout sometimes because a sever will never recover. It keeps trying, but keeps ending up with a corrupted index problem and eventually the test framework hard kills it.

          Show
          Mark Miller added a comment - Something similar to this corrupted index issue also causes this test to timeout sometimes because a sever will never recover. It keeps trying, but keeps ending up with a corrupted index problem and eventually the test framework hard kills it.
          Hide
          Mark Miller added a comment -

          Can we get this in? I'd love to see it's affect on some common test fails over some time before 5.0 hits and see that nothing else pops out.

          Show
          Mark Miller added a comment - Can we get this in? I'd love to see it's affect on some common test fails over some time before 5.0 hits and see that nothing else pops out.
          Hide
          Anshum Gupta added a comment -

          Changing this to a Blocker for 5.0 as I think it needs to go in for 5.0.

          Show
          Anshum Gupta added a comment - Changing this to a Blocker for 5.0 as I think it needs to go in for 5.0.
          Hide
          Shalin Shekhar Mangar added a comment -

          Here's a new test called RecoveryAfterSoftCommitTest which reproduces these failures.

          My last fix isn't quite right; it leaks recovery threads and increases the recovery time by a lot. I am trying to figure out a better way.

          Show
          Shalin Shekhar Mangar added a comment - Here's a new test called RecoveryAfterSoftCommitTest which reproduces these failures. My last fix isn't quite right; it leaks recovery threads and increases the recovery time by a lot. I am trying to figure out a better way.
          Hide
          Shalin Shekhar Mangar added a comment -

          Thanks Varun for catching my mistake. The try-with-resource was wrong and I should have used try/finally to decref the writer. This patch does that.

          This patch closes the searcher itself in cloud mode and did away with re-opening the searcher. All tests pass. I'll commit this shortly and bake it on jenkins for a while.

          Show
          Shalin Shekhar Mangar added a comment - Thanks Varun for catching my mistake. The try-with-resource was wrong and I should have used try/finally to decref the writer. This patch does that. This patch closes the searcher itself in cloud mode and did away with re-opening the searcher. All tests pass. I'll commit this shortly and bake it on jenkins for a while.
          Hide
          ASF subversion and git services added a comment -

          Commit 1653281 from shalin@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1653281 ]

          SOLR-6640: Close searchers before rollback and recovery to avoid index corruption

          Show
          ASF subversion and git services added a comment - Commit 1653281 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1653281 ] SOLR-6640 : Close searchers before rollback and recovery to avoid index corruption
          Hide
          ASF subversion and git services added a comment -

          Commit 1653283 from shalin@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1653283 ]

          SOLR-6640: No need for SSL in this test

          Show
          ASF subversion and git services added a comment - Commit 1653283 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1653283 ] SOLR-6640 : No need for SSL in this test
          Hide
          ASF subversion and git services added a comment -

          Commit 1653837 from shalin@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1653837 ]

          SOLR-6640: Add changelog entry

          Show
          ASF subversion and git services added a comment - Commit 1653837 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1653837 ] SOLR-6640 : Add changelog entry
          Hide
          ASF subversion and git services added a comment -

          Commit 1653838 from shalin@apache.org in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1653838 ]

          SOLR-6640: Close searchers before rollback and recovery to avoid index corruption

          Show
          ASF subversion and git services added a comment - Commit 1653838 from shalin@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1653838 ] SOLR-6640 : Close searchers before rollback and recovery to avoid index corruption
          Hide
          ASF subversion and git services added a comment -

          Commit 1653839 from shalin@apache.org in branch 'dev/branches/lucene_solr_5_0'
          [ https://svn.apache.org/r1653839 ]

          SOLR-6640: Close searchers before rollback and recovery to avoid index corruption

          Show
          ASF subversion and git services added a comment - Commit 1653839 from shalin@apache.org in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1653839 ] SOLR-6640 : Close searchers before rollback and recovery to avoid index corruption
          Hide
          ASF subversion and git services added a comment -

          Commit 1653844 from shalin@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1653844 ]

          SOLR-6640: Exclude lock files from unused file check

          Show
          ASF subversion and git services added a comment - Commit 1653844 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1653844 ] SOLR-6640 : Exclude lock files from unused file check
          Hide
          ASF subversion and git services added a comment -

          Commit 1653845 from shalin@apache.org in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1653845 ]

          SOLR-6640: Exclude lock files from unused file check

          Show
          ASF subversion and git services added a comment - Commit 1653845 from shalin@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1653845 ] SOLR-6640 : Exclude lock files from unused file check
          Hide
          ASF subversion and git services added a comment -

          Commit 1653846 from shalin@apache.org in branch 'dev/branches/lucene_solr_5_0'
          [ https://svn.apache.org/r1653846 ]

          SOLR-6640: Exclude lock files from unused file check

          Show
          ASF subversion and git services added a comment - Commit 1653846 from shalin@apache.org in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1653846 ] SOLR-6640 : Exclude lock files from unused file check
          Hide
          Mark Miller added a comment -

          The Chaos Monkey tests are already in much better shape - I run a variety of them in normal and nightly mode on my local jenkins machine, and the 3 day performance is great.

          There are a couple things to look into with ChaosMonkeyNothingIsSafeTest, but that appears unrelated to this change.

          Show
          Mark Miller added a comment - The Chaos Monkey tests are already in much better shape - I run a variety of them in normal and nightly mode on my local jenkins machine, and the 3 day performance is great. There are a couple things to look into with ChaosMonkeyNothingIsSafeTest, but that appears unrelated to this change.
          Hide
          Mark Miller added a comment -

          Spoke too soon - one of the jobs did hit a corrupt index yesterday. I'll post the logs.

          Show
          Mark Miller added a comment - Spoke too soon - one of the jobs did hit a corrupt index yesterday. I'll post the logs.
          Hide
          Varun Thacker added a comment -

          Hi Mark Miller,

          I was unable to reproduce it just by running this on branch_5x - ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=22E5E46764398970.

          Also I don't see this line which generally comes with the the fail - "NOTE: reproduce with: ant test...."

          Are you able to reproduce it? If yes then can you tell me how? I'd also like to test it out and see why it got corrupted.

          Show
          Varun Thacker added a comment - Hi Mark Miller , I was unable to reproduce it just by running this on branch_5x - ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=22E5E46764398970 . Also I don't see this line which generally comes with the the fail - "NOTE: reproduce with: ant test...." Are you able to reproduce it? If yes then can you tell me how? I'd also like to test it out and see why it got corrupted.
          Hide
          Shalin Shekhar Mangar added a comment -

          Thanks Mark. I am trying to figure out the failure and create a smaller reproducible test case.

          Show
          Shalin Shekhar Mangar added a comment - Thanks Mark. I am trying to figure out the failure and create a smaller reproducible test case.
          Hide
          ASF subversion and git services added a comment -

          Commit 1655249 from Anshum Gupta in branch 'dev/trunk'
          [ https://svn.apache.org/r1655249 ]

          SOLR-6640: Fixing the comment

          Show
          ASF subversion and git services added a comment - Commit 1655249 from Anshum Gupta in branch 'dev/trunk' [ https://svn.apache.org/r1655249 ] SOLR-6640 : Fixing the comment
          Hide
          ASF subversion and git services added a comment -

          Commit 1655250 from Anshum Gupta in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1655250 ]

          SOLR-6640: Fixing the comment (merge from trunk)

          Show
          ASF subversion and git services added a comment - Commit 1655250 from Anshum Gupta in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1655250 ] SOLR-6640 : Fixing the comment (merge from trunk)
          Hide
          ASF subversion and git services added a comment -

          Commit 1655251 from Anshum Gupta in branch 'dev/branches/lucene_solr_5_0'
          [ https://svn.apache.org/r1655251 ]

          SOLR-6640: Fixing the comment (merge from branch_5x)

          Show
          ASF subversion and git services added a comment - Commit 1655251 from Anshum Gupta in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1655251 ] SOLR-6640 : Fixing the comment (merge from branch_5x)
          Hide
          Mark Miller added a comment -

          Hey Shalin Shekhar Mangar, shouldn't

          Collection<String> files = info.info.files(); // All files that belong to this segment

          be

          Collection<String> files = info.files(); // All files that belong to this segment

          The former skips some logic in the later - one note in particular in info.files() mentions adding live files, and in tests, I'm seeing this exhibit bad behavior and timeout waiting for .liv files to be deletable.

          Show
          Mark Miller added a comment - Hey Shalin Shekhar Mangar , shouldn't Collection<String> files = info.info.files(); // All files that belong to this segment be Collection<String> files = info.files(); // All files that belong to this segment The former skips some logic in the later - one note in particular in info.files() mentions adding live files, and in tests, I'm seeing this exhibit bad behavior and timeout waiting for .liv files to be deletable.
          Hide
          Shalin Shekhar Mangar added a comment -

          I had been ill last week so I am resuming work on this today.

          I suspect you're right, Mark. I should use the info.files(). Better yet, we can use SegmentInfos.files(Directory dir, boolean includeSegmentsFile) method and avoid iterating through SegmentCommitInfo objects.

          Show
          Shalin Shekhar Mangar added a comment - I had been ill last week so I am resuming work on this today. I suspect you're right, Mark. I should use the info.files(). Better yet, we can use SegmentInfos.files(Directory dir, boolean includeSegmentsFile) method and avoid iterating through SegmentCommitInfo objects.
          Hide
          ASF subversion and git services added a comment -

          Commit 1656630 from shalin@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1656630 ]

          SOLR-6640: Use SegmentInfos.files in unused file check

          Show
          ASF subversion and git services added a comment - Commit 1656630 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1656630 ] SOLR-6640 : Use SegmentInfos.files in unused file check
          Hide
          ASF subversion and git services added a comment -

          Commit 1656631 from shalin@apache.org in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1656631 ]

          SOLR-6640: Use SegmentInfos.files in unused file check

          Show
          ASF subversion and git services added a comment - Commit 1656631 from shalin@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1656631 ] SOLR-6640 : Use SegmentInfos.files in unused file check
          Hide
          Anshum Gupta added a comment -

          I spoke to Shalin and he suggested me to wait until Friday to see if all stays good after the last commit. If nothing fails, I'll continue with the next RC for Solr 5.0.
          By the time we get done with the vote, we'd have given the fix even more time.

          Show
          Anshum Gupta added a comment - I spoke to Shalin and he suggested me to wait until Friday to see if all stays good after the last commit. If nothing fails, I'll continue with the next RC for Solr 5.0. By the time we get done with the vote, we'd have given the fix even more time.
          Hide
          ASF subversion and git services added a comment -

          Commit 1657490 from shalin@apache.org in branch 'dev/branches/lucene_solr_5_0'
          [ https://svn.apache.org/r1657490 ]

          SOLR-6640: Use SegmentInfos.files in unused file check

          Show
          ASF subversion and git services added a comment - Commit 1657490 from shalin@apache.org in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1657490 ] SOLR-6640 : Use SegmentInfos.files in unused file check
          Hide
          Shalin Shekhar Mangar added a comment -

          I have found another failure. This time on HdfsBasicDistributedZk2Test and it is easily reproducible with:

          ant test  -Dtestcase=HdfsBasicDistributedZk2Test -Dtests.method=test -Dtests.seed=A7C02AE49EA55A58 -Dtests.nigly=true -Dtests.slow=true -Dtests.locale=iw_IL -Dtests.timezone=America/St_Vincent -Dtests.asserts=true -Dtests.file.encoding=UTF-8
          

          I'd appreciate some help in tracking down the root cause.

          Show
          Shalin Shekhar Mangar added a comment - I have found another failure. This time on HdfsBasicDistributedZk2Test and it is easily reproducible with: ant test -Dtestcase=HdfsBasicDistributedZk2Test -Dtests.method=test -Dtests.seed=A7C02AE49EA55A58 -Dtests.nigly= true -Dtests.slow= true -Dtests.locale=iw_IL -Dtests.timezone=America/St_Vincent -Dtests.asserts= true -Dtests.file.encoding=UTF-8 I'd appreciate some help in tracking down the root cause.
          Hide
          Mark Miller added a comment -

          I'm taking a look.

          Show
          Mark Miller added a comment - I'm taking a look.
          Hide
          Mark Miller added a comment -

          Those troublesome .si files...

          We don't download because we think we already have the right one I believe.

          Varun's nice issue fixes this: SOLR-6920: During replication use checksums to verify if files are the same

          Show
          Mark Miller added a comment - Those troublesome .si files... We don't download because we think we already have the right one I believe. Varun's nice issue fixes this: SOLR-6920 : During replication use checksums to verify if files are the same
          Hide
          Mark Miller added a comment -

          Of course that only works if your segments are new enough to have checksum info in that patch.

          Show
          Mark Miller added a comment - Of course that only works if your segments are new enough to have checksum info in that patch.
          Hide
          Mark Miller added a comment -

          I've been focusing on SOLR-6920 with Varun. I think that is the right path to remove the rest of these issues.

          Show
          Mark Miller added a comment - I've been focusing on SOLR-6920 with Varun. I think that is the right path to remove the rest of these issues.
          Hide
          Varun Thacker added a comment -

          1. Changed the log message in SnapPuller.compareFile() since checksum won't be present in case of exception.
          From - LOG.warn("File {} did not match. expected checksum is {} and actual is checksum {}. " + "expected length is {} and actual length is {}", filename, backupIndexFileChecksum indexFileChecksum, backupIndexFileLen, indexFileLen);
          To - LOG.warn("File {} did not match. expected length is {} and actual length is {}", filename,backupIndexFileLen, indexFileLen);

          2. In SnapPuller.downloadIndexFiles made (String) file.get(NAME)) into a variable for better readibility.

          3. The if condition still needs tweaking? We could still have a non checksummed/checksum threw error .si/.liv/segments_n file of equal length and we wouldn't re-download?

          Maybe the condition could be the check you initially proposed -
          1. If file is a .si/.liv/segments_n file download regardless
          2. else if if (!compareResult.equal || downloadCompleteIndex) then re-download?

          Show
          Varun Thacker added a comment - 1. Changed the log message in SnapPuller.compareFile() since checksum won't be present in case of exception. From - LOG.warn("File {} did not match. expected checksum is {} and actual is checksum {}. " + "expected length is {} and actual length is {}", filename, backupIndexFileChecksum indexFileChecksum, backupIndexFileLen, indexFileLen); To - LOG.warn("File {} did not match. expected length is {} and actual length is {}", filename,backupIndexFileLen, indexFileLen); 2. In SnapPuller.downloadIndexFiles made (String) file.get(NAME)) into a variable for better readibility. 3. The if condition still needs tweaking? We could still have a non checksummed/checksum threw error .si/.liv/segments_n file of equal length and we wouldn't re-download? Maybe the condition could be the check you initially proposed - 1. If file is a .si/.liv/segments_n file download regardless 2. else if if (!compareResult.equal || downloadCompleteIndex) then re-download?
          Hide
          Varun Thacker added a comment -

          Sorry this was supposed to go in SOLR-6920. I'll repost it there.

          Show
          Varun Thacker added a comment - Sorry this was supposed to go in SOLR-6920 . I'll repost it there.
          Hide
          ASF subversion and git services added a comment -

          Commit 1657969 from Mark Miller in branch 'dev/trunk'
          [ https://svn.apache.org/r1657969 ]

          SOLR-6920, SOLR-6640: A replicated index can end up corrupted when small files end up with the same file name and size.

          Show
          ASF subversion and git services added a comment - Commit 1657969 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1657969 ] SOLR-6920 , SOLR-6640 : A replicated index can end up corrupted when small files end up with the same file name and size.
          Hide
          Mark Miller added a comment -

          Since committing that to trunk, my trunk runs are looking pretty good and I still see some nasty fails on 5x. So far, so good.

          Show
          Mark Miller added a comment - Since committing that to trunk, my trunk runs are looking pretty good and I still see some nasty fails on 5x. So far, so good.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658078 from Mark Miller in branch 'dev/trunk'
          [ https://svn.apache.org/r1658078 ]

          SOLR-6920, SOLR-6640: When we so not have a checksum for a file, always download files under 100kb and other small improvements.

          Show
          ASF subversion and git services added a comment - Commit 1658078 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1658078 ] SOLR-6920 , SOLR-6640 : When we so not have a checksum for a file, always download files under 100kb and other small improvements.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658083 from Mark Miller in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1658083 ]

          SOLR-6920, SOLR-6640: A replicated index can end up corrupted when small files end up with the same file name and size.

          Show
          ASF subversion and git services added a comment - Commit 1658083 from Mark Miller in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1658083 ] SOLR-6920 , SOLR-6640 : A replicated index can end up corrupted when small files end up with the same file name and size.
          Hide
          Mark Miller added a comment -

          Anyone that can help with review of these commits, please do.

          Show
          Mark Miller added a comment - Anyone that can help with review of these commits, please do.
          Hide
          Mark Miller added a comment -

          We should not log an error when the compare method does not see the file locally - that is an expected path.

          Show
          Mark Miller added a comment - We should not log an error when the compare method does not see the file locally - that is an expected path.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658089 from Mark Miller in branch 'dev/trunk'
          [ https://svn.apache.org/r1658089 ]

          SOLR-6920, SOLR-6640: Do not log an error when a file does not exist for comparison.

          Show
          ASF subversion and git services added a comment - Commit 1658089 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1658089 ] SOLR-6920 , SOLR-6640 : Do not log an error when a file does not exist for comparison.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658090 from Mark Miller in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1658090 ]

          SOLR-6920, SOLR-6640: Do not log an error when a file does not exist for comparison.

          Show
          ASF subversion and git services added a comment - Commit 1658090 from Mark Miller in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1658090 ] SOLR-6920 , SOLR-6640 : Do not log an error when a file does not exist for comparison.
          Hide
          Gregory Chanan added a comment -

          Did a quick pass, looks good.

          One suggestion I would make is renaming the function filesToAlwaysDownloadIfChecksumFails. Checksum fails sounds like the checksum didn't match up in which case I'd hope we'd download every file, not that the checksum wasn't present. Maybe filesToAlwaysDownloadIfNoChecksums? filesToAlwaysDownloadIfNoChecksumsPresent?

          Show
          Gregory Chanan added a comment - Did a quick pass, looks good. One suggestion I would make is renaming the function filesToAlwaysDownloadIfChecksumFails. Checksum fails sounds like the checksum didn't match up in which case I'd hope we'd download every file, not that the checksum wasn't present. Maybe filesToAlwaysDownloadIfNoChecksums? filesToAlwaysDownloadIfNoChecksumsPresent?
          Hide
          Mark Miller added a comment -

          Thanks Greg, I'll incorporate - wasn't very happy with the name myself.

          I don't know that it's critical for this blocker issue, but another change we should probably make is: if the replication into an existing index fails, don't keep trying that - move to replicating to a new index folder.

          Show
          Mark Miller added a comment - Thanks Greg, I'll incorporate - wasn't very happy with the name myself. I don't know that it's critical for this blocker issue, but another change we should probably make is: if the replication into an existing index fails, don't keep trying that - move to replicating to a new index folder.
          Hide
          Mark Miller added a comment -

          I've had my local jenkins machine push it's 10 chaos monkey tests on this all day. So far so good in terms of index corruption. I'll add in the rename mentioned above and let jenkins crank overnight - let's look at moving to 5.0 in the morning when we have some more confidence.

          Show
          Mark Miller added a comment - I've had my local jenkins machine push it's 10 chaos monkey tests on this all day. So far so good in terms of index corruption. I'll add in the rename mentioned above and let jenkins crank overnight - let's look at moving to 5.0 in the morning when we have some more confidence.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658129 from Mark Miller in branch 'dev/trunk'
          [ https://svn.apache.org/r1658129 ]

          SOLR-6920, SOLR-6640: Rename method to filesToAlwaysDownloadIfNoChecksums.

          Show
          ASF subversion and git services added a comment - Commit 1658129 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1658129 ] SOLR-6920 , SOLR-6640 : Rename method to filesToAlwaysDownloadIfNoChecksums.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658130 from Mark Miller in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1658130 ]

          SOLR-6920, SOLR-6640: Rename method to filesToAlwaysDownloadIfNoChecksums.

          Show
          ASF subversion and git services added a comment - Commit 1658130 from Mark Miller in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1658130 ] SOLR-6920 , SOLR-6640 : Rename method to filesToAlwaysDownloadIfNoChecksums.
          Hide
          Shalin Shekhar Mangar added a comment -

          Thanks Mark. This has helped a lot. I wrote a test which would hit corruption at least once in 10 iterations before SOLR-6920 but I can't get it to fail anymore.

          Show
          Shalin Shekhar Mangar added a comment - Thanks Mark. This has helped a lot. I wrote a test which would hit corruption at least once in 10 iterations before SOLR-6920 but I can't get it to fail anymore.
          Hide
          Mark Miller added a comment -

          Thanks shalin - my chaosmonkey jobs went well overnight too.

          I'll look at the merge into 5.0 shortly.

          More review wouldn't hurt if anyone has time.

          Show
          Mark Miller added a comment - Thanks shalin - my chaosmonkey jobs went well overnight too. I'll look at the merge into 5.0 shortly. More review wouldn't hurt if anyone has time.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658240 from Mark Miller in branch 'dev/branches/lucene_solr_5_0'
          [ https://svn.apache.org/r1658240 ]

          SOLR-6920, SOLR-6640: A replicated index can end up corrupted when small files end up with the same file name and size.

          Show
          ASF subversion and git services added a comment - Commit 1658240 from Mark Miller in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1658240 ] SOLR-6920 , SOLR-6640 : A replicated index can end up corrupted when small files end up with the same file name and size.
          Hide
          Anshum Gupta added a comment -

          Thanks for wrapping this one up Mark. Looks good!

          I think it'd be good to declare a final int MIN_FILE_SIZE or something to use in filesToAlwaysDownloadIfNoChecksums. Also, the comment says files < 100kb would be copied but the check seems to be for 100,000 bytes or about 97kb? It's ok but I think but we should either change the comment or the number (nothing pressing).

          Also, I think the concat of 2 strings while logging was an IDE issue

          SnapPuller.java
          LOG.warn(
                          "File {} did not match. "  + "expected length is {} and actual length is {}",
                          filename, backupIndexFileChecksum, indexFileChecksum,
                      
          
          Show
          Anshum Gupta added a comment - Thanks for wrapping this one up Mark. Looks good! I think it'd be good to declare a final int MIN_FILE_SIZE or something to use in filesToAlwaysDownloadIfNoChecksums. Also, the comment says files < 100kb would be copied but the check seems to be for 100,000 bytes or about 97kb? It's ok but I think but we should either change the comment or the number (nothing pressing). Also, I think the concat of 2 strings while logging was an IDE issue SnapPuller.java LOG.warn( "File {} did not match. " + "expected length is {} and actual length is {}" , filename, backupIndexFileChecksum, indexFileChecksum,
          Hide
          Mark Miller added a comment -

          < 100kb would be copied but the check seems to be for 100,000 bytes or about 97kb?

          I went with Google's conversion using digital storage: 100000 bytes to kb = 100 kilobytes

          Show
          Mark Miller added a comment - < 100kb would be copied but the check seems to be for 100,000 bytes or about 97kb? I went with Google's conversion using digital storage: 100000 bytes to kb = 100 kilobytes
          Hide
          Anshum Gupta added a comment -

          I think we use the standard 1024 conversion factor everywhere but I think it's ok and probably not worth debating about.

          Show
          Anshum Gupta added a comment - I think we use the standard 1024 conversion factor everywhere but I think it's ok and probably not worth debating about.
          Hide
          Mark Miller added a comment -

          Yeah, I'm not really arguing one way or another - I think that debate has been had Eithet way - I simply went with Google is all.

          Show
          Mark Miller added a comment - Yeah, I'm not really arguing one way or another - I think that debate has been had Eithet way - I simply went with Google is all.
          Hide
          Anshum Gupta added a comment -

          With you on that anyways. I think this issue is now handled and I can cut an RC later in the day today/tomorrow morning. What do you think?

          Show
          Anshum Gupta added a comment - With you on that anyways. I think this issue is now handled and I can cut an RC later in the day today/tomorrow morning. What do you think?
          Hide
          Mark Miller added a comment -

          Yeah, I think we are prob good. I'll make the improvement suggested on trunk and 5x.

          Show
          Mark Miller added a comment - Yeah, I think we are prob good. I'll make the improvement suggested on trunk and 5x.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658519 from Mark Miller in branch 'dev/trunk'
          [ https://svn.apache.org/r1658519 ]

          SOLR-6920, SOLR-6640: Make constant and fix logging.

          Show
          ASF subversion and git services added a comment - Commit 1658519 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1658519 ] SOLR-6920 , SOLR-6640 : Make constant and fix logging.
          Hide
          Mark Miller added a comment -

          Thanks to all involved in this. Long time, bad time issue.

          Show
          Mark Miller added a comment - Thanks to all involved in this. Long time, bad time issue.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658524 from Mark Miller in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1658524 ]

          SOLR-6920, SOLR-6640: Make constant and fix logging.

          Show
          ASF subversion and git services added a comment - Commit 1658524 from Mark Miller in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1658524 ] SOLR-6920 , SOLR-6640 : Make constant and fix logging.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658526 from Mark Miller in branch 'dev/branches/lucene_solr_5_0'
          [ https://svn.apache.org/r1658526 ]

          SOLR-6920, SOLR-6640: Make constant and fix logging.

          Show
          ASF subversion and git services added a comment - Commit 1658526 from Mark Miller in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1658526 ] SOLR-6920 , SOLR-6640 : Make constant and fix logging.
          Hide
          Anshum Gupta added a comment -

          Bulk close after 5.0 release.

          Show
          Anshum Gupta added a comment - Bulk close after 5.0 release.

            People

            • Assignee:
              Mark Miller
              Reporter:
              Shalin Shekhar Mangar
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development