Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-12801 Fix the tests, remove BadApples and AwaitsFix annotations, improve env for test development.
  3. SOLR-13068

many cloud/autoscaling tests are using System.currentTimeMillis() for timing comparisons (under the covers)

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.7, 8.0
    • Component/s: None
    • Labels:
      None

      Description

      After rewriting TriggerSetPropertiesIntegrationTest in SOLR-13054 to use better concurrency handling/signalling and log the timestamps the triggers were firing at, i noticed we stil got a failure from Uwe's "Windows" jenkins machine (on the http2 branch, but after my fix was merged to that branch. The nature of the failure seemed to suggest that the JVM's ScheduledExecutorService.scheduleWithFixedDelay wasn't living up to it's contract – and was running successive iterations before the full delay had lapsed.

      But then i realized that in spite of using timeSource.getTimeNs() in the test, the TimeSource (being used in the test) was a lie – and under the covers System.currentTimeInMillis is being used (via TimeSource.CURRENT_TIME ) ... which IIUC is susceptible to clock drift, particularly in VMs like those used on Uwe's jenkins machines...

      Any code in the following tests that relies on the TimeSource for doing comparisons or delta calculations should be suspect...

      hossman@tray:~/lucene/dev [master] $ find solr/core/src/test -name \*.java | xargs grep CURRENT_TIME
      solr/core/src/test/org/apache/solr/cloud/autoscaling/NodeAddedTriggerTest.java:  private static final TimeSource timeSource = TimeSource.CURRENT_TIME;
      solr/core/src/test/org/apache/solr/cloud/autoscaling/ExecutePlanActionTest.java:          "mock_trigger_name", Collections.singletonList(TimeSource.CURRENT_TIME.getTimeNs()),
      solr/core/src/test/org/apache/solr/cloud/autoscaling/TriggerIntegrationTest.java:  static final TimeSource timeSource = TimeSource.CURRENT_TIME;
      solr/core/src/test/org/apache/solr/cloud/autoscaling/NodeLostTriggerTest.java:  private final TimeSource timeSource = TimeSource.CURRENT_TIME;
      solr/core/src/test/org/apache/solr/cloud/autoscaling/sim/TestSimExecutePlanAction.java:          "mock_trigger_name", Collections.singletonList(TimeSource.CURRENT_TIME.getTimeNs()),
      solr/core/src/test/org/apache/solr/cloud/HttpPartitionTest.java:    TimeOut timeOut = new TimeOut(ms, TimeUnit.MILLISECONDS, TimeSource.CURRENT_TIME);
      solr/core/src/test/org/apache/solr/cloud/TestCloudConsistency.java:    TimeOut timeOut = new TimeOut(10, TimeUnit.SECONDS, TimeSource.CURRENT_TIME);
      solr/core/src/test/org/apache/solr/cloud/TestCloudConsistency.java:    TimeOut timeOut = new TimeOut(10, TimeUnit.SECONDS, TimeSource.CURRENT_TIME);
      hossman@tray:~/lucene/dev [master] $ find -name \*.java | xargs grep TriggerIntegrationTest.timeSource
      ./solr/core/src/test/org/apache/solr/cloud/autoscaling/SearchRateTriggerIntegrationTest.java:import static org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.timeSource;
      ./solr/core/src/test/org/apache/solr/cloud/autoscaling/TriggerSetPropertiesIntegrationTest.java:import static org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.timeSource;
      ./solr/core/src/test/org/apache/solr/cloud/autoscaling/RestoreTriggerStateTest.java:import static org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.timeSource;
      ./solr/core/src/test/org/apache/solr/cloud/autoscaling/NodeLostTriggerIntegrationTest.java:          long currentTimeNanos = TriggerIntegrationTest.timeSource.getTimeNs();
      ./solr/core/src/test/org/apache/solr/cloud/autoscaling/MetricTriggerIntegrationTest.java:import static org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.timeSource;
      ./solr/core/src/test/org/apache/solr/cloud/autoscaling/NodeAddedTriggerIntegrationTest.java:          long currentTimeNanos = TriggerIntegrationTest.timeSource.getTimeNs();
      ./solr/core/src/test/org/apache/solr/cloud/autoscaling/TriggerCooldownIntegrationTest.java:import static org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.timeSource;
      

        Attachments

        1. SOLR-13068.patch
          31 kB
          Chris M. Hostetter
        2. SOLR-13068.patch
          28 kB
          Chris M. Hostetter

          Issue Links

            Activity

              People

              • Assignee:
                hossman Chris M. Hostetter
                Reporter:
                hossman Chris M. Hostetter
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: