Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2487

SyslogParser has rounding errors in timestamp parsing

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.0.1
    • Fix Version/s: 1.6.0
    • Component/s: Sinks+Sources
    • Labels:

      Description

      https://travis-ci.org/Stratio/flume/jobs/36847695#L6180

      testMultiplePorts(org.apache.flume.source.TestMultiportSyslogTCPSource) Time elapsed: 2465 sec <<< FAILURE!
      
      org.junit.ComparisonFailure: Timestamps must match expected:<141223934050[2]> but was:<141223934050[1]>
      
      at org.junit.Assert.assertEquals(Assert.java:125)
      
      at org.apache.flume.source.TestMultiportSyslogTCPSource.testMultiplePorts(TestMultiportSyslogTCPSource.java:156)
      
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      
      at java.lang.reflect.Method.invoke(Method.java:606)
      
      at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
      
      at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      
      at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
      
      at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
      
      at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
      
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
      
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
      
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
      
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
      
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
      
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
      
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
      
      at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
      
      at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
      
      at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
      
      at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
      
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      
      at java.lang.reflect.Method.invoke(Method.java:606)
      
      at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
      
      at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
      
      at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
      
      at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
      
      at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
      
      1. FLUME-2487-0.patch
        2 kB
        Santiago M. Mola

        Issue Links

          Activity

          Hide
          smolav Santiago M. Mola added a comment -

          This was a precision error produced by the use of floating point arithmetic during parsing.

          Show
          smolav Santiago M. Mola added a comment - This was a precision error produced by the use of floating point arithmetic during parsing.
          Hide
          paliwalashish Ashish Paliwal added a comment -

          testMultiplePorts() test passes at my end. How can I reproduce the error mentioned?

          Show
          paliwalashish Ashish Paliwal added a comment - testMultiplePorts() test passes at my end. How can I reproduce the error mentioned?
          Hide
          smolav Santiago M. Mola added a comment -

          Ashish Paliwal Given the non-deterministic nature of the test (dependent of the time its run), chances are that you never get this error running the test. But the bug is clearly visible in the code: floating point numbers are not exact and they lead to rounding errors. This is a well-known fact. Just run this yourself:

          @Test
          public void testXXX() {
          for (long i = 100; i < 999; i++) {
          float frac = Float.parseFloat(String.format(".%d", i));
          float multiplied = (frac * 1000f);
          long milliseconds = (long) multiplied;
          if (i != milliseconds)

          { System.out.println("PRECISION ERROR: " + i + " | " + milliseconds + " | " + frac + " | " + multiplied); }

          }
          }

          Output:
          PRECISION ERROR: 251 | 250 | 0.251 | 250.99998
          PRECISION ERROR: 253 | 252 | 0.253 | 252.99998
          PRECISION ERROR: 502 | 501 | 0.502 | 501.99997
          PRECISION ERROR: 506 | 505 | 0.506 | 505.99997
          PRECISION ERROR: 511 | 510 | 0.511 | 510.99997

          This code needs to use integer arithmetic or BigDecimal. My patch uses just integer arithmetic, but I can write one to use BigDecimal if that's preferred.

          Show
          smolav Santiago M. Mola added a comment - Ashish Paliwal Given the non-deterministic nature of the test (dependent of the time its run), chances are that you never get this error running the test. But the bug is clearly visible in the code: floating point numbers are not exact and they lead to rounding errors. This is a well-known fact. Just run this yourself: @Test public void testXXX() { for (long i = 100; i < 999; i++) { float frac = Float.parseFloat(String.format(".%d", i)); float multiplied = (frac * 1000f); long milliseconds = (long) multiplied; if (i != milliseconds) { System.out.println("PRECISION ERROR: " + i + " | " + milliseconds + " | " + frac + " | " + multiplied); } } } Output: PRECISION ERROR: 251 | 250 | 0.251 | 250.99998 PRECISION ERROR: 253 | 252 | 0.253 | 252.99998 PRECISION ERROR: 502 | 501 | 0.502 | 501.99997 PRECISION ERROR: 506 | 505 | 0.506 | 505.99997 PRECISION ERROR: 511 | 510 | 0.511 | 510.99997 This code needs to use integer arithmetic or BigDecimal. My patch uses just integer arithmetic, but I can write one to use BigDecimal if that's preferred.
          Hide
          smolav Santiago M. Mola added a comment -

          Patch updated with test case that may prove the bug.

          Show
          smolav Santiago M. Mola added a comment - Patch updated with test case that may prove the bug.
          Hide
          roshan_naik Roshan Naik added a comment -

          +1

          Show
          roshan_naik Roshan Naik added a comment - +1
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit fdb51b2af879f9fb98ffe209068a95424f700e69 in flume's branch refs/heads/flume-1.6 from Roshan Naik
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=fdb51b2 ]

          FLUME-2487. SyslogParser has rounding errors in timestamp parsing

          (Santiago M. Mola via Roshan Naik)

          Show
          jira-bot ASF subversion and git services added a comment - Commit fdb51b2af879f9fb98ffe209068a95424f700e69 in flume's branch refs/heads/flume-1.6 from Roshan Naik [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=fdb51b2 ] FLUME-2487 . SyslogParser has rounding errors in timestamp parsing (Santiago M. Mola via Roshan Naik)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 55a4666434d892f838208b315cd9ba06ea07659b in flume's branch refs/heads/trunk from Roshan Naik
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=55a4666 ]

          FLUME-2487. SyslogParser has rounding errors in timestamp parsing

          (Santiago M. Mola via Roshan Naik)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 55a4666434d892f838208b315cd9ba06ea07659b in flume's branch refs/heads/trunk from Roshan Naik [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=55a4666 ] FLUME-2487 . SyslogParser has rounding errors in timestamp parsing (Santiago M. Mola via Roshan Naik)
          Hide
          roshan_naik Roshan Naik added a comment -

          Committed. Thanks Santiago.

          Show
          roshan_naik Roshan Naik added a comment - Committed. Thanks Santiago.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in flume-trunk #699 (See https://builds.apache.org/job/flume-trunk/699/)
          FLUME-2487. SyslogParser has rounding errors in timestamp parsing (roshan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=55a4666434d892f838208b315cd9ba06ea07659b)

          • flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogParser.java
          • flume-ng-core/src/main/java/org/apache/flume/source/SyslogParser.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in flume-trunk #699 (See https://builds.apache.org/job/flume-trunk/699/ ) FLUME-2487 . SyslogParser has rounding errors in timestamp parsing (roshan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=55a4666434d892f838208b315cd9ba06ea07659b ) flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogParser.java flume-ng-core/src/main/java/org/apache/flume/source/SyslogParser.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Flume-trunk-hbase-98 #56 (See https://builds.apache.org/job/Flume-trunk-hbase-98/56/)
          FLUME-2487. SyslogParser has rounding errors in timestamp parsing (roshan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=55a4666434d892f838208b315cd9ba06ea07659b)

          • flume-ng-core/src/main/java/org/apache/flume/source/SyslogParser.java
          • flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogParser.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Flume-trunk-hbase-98 #56 (See https://builds.apache.org/job/Flume-trunk-hbase-98/56/ ) FLUME-2487 . SyslogParser has rounding errors in timestamp parsing (roshan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=55a4666434d892f838208b315cd9ba06ea07659b ) flume-ng-core/src/main/java/org/apache/flume/source/SyslogParser.java flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogParser.java

            People

            • Assignee:
              smolav Santiago M. Mola
              Reporter:
              smolav Santiago M. Mola
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development