Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2492

Flume's Kafka Source doesn't account time correctly

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: None
    • Labels:
      None

      Description

      When receiving events, Flume's Kafka source does not account correctly for passage of time. 1s batches can take significantly longer (i.e 10s or more).

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Flume-trunk-hbase-98 #26 (See https://builds.apache.org/job/Flume-trunk-hbase-98/26/)
        FLUME-2492. Flume's Kafka Source doesn't account time correctly (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9f75c40a69fa4404ea2c344fa29285a00eb082fa)

        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
        • flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceTest.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Flume-trunk-hbase-98 #26 (See https://builds.apache.org/job/Flume-trunk-hbase-98/26/ ) FLUME-2492 . Flume's Kafka Source doesn't account time correctly (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9f75c40a69fa4404ea2c344fa29285a00eb082fa ) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceTest.java
        Hide
        hudson Hudson added a comment -

        UNSTABLE: Integrated in flume-trunk #666 (See https://builds.apache.org/job/flume-trunk/666/)
        FLUME-2492. Flume's Kafka Source doesn't account time correctly (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9f75c40a69fa4404ea2c344fa29285a00eb082fa)

        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
        • flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceTest.java
        Show
        hudson Hudson added a comment - UNSTABLE: Integrated in flume-trunk #666 (See https://builds.apache.org/job/flume-trunk/666/ ) FLUME-2492 . Flume's Kafka Source doesn't account time correctly (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=9f75c40a69fa4404ea2c344fa29285a00eb082fa ) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceTest.java
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Committed! Thanks Gwen!

        Show
        hshreedharan Hari Shreedharan added a comment - Committed! Thanks Gwen!
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 99736f6eeded760069180693e94588d18eab0680 in flume's branch refs/heads/flume-1.6 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=99736f6 ]

        FLUME-2492. Flume's Kafka Source doesn't account time correctly

        (Gwen Shapira via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 99736f6eeded760069180693e94588d18eab0680 in flume's branch refs/heads/flume-1.6 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=99736f6 ] FLUME-2492 . Flume's Kafka Source doesn't account time correctly (Gwen Shapira via Hari)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 9f75c40a69fa4404ea2c344fa29285a00eb082fa in flume's branch refs/heads/trunk from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=9f75c40 ]

        FLUME-2492. Flume's Kafka Source doesn't account time correctly

        (Gwen Shapira via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 9f75c40a69fa4404ea2c344fa29285a00eb082fa in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=9f75c40 ] FLUME-2492 . Flume's Kafka Source doesn't account time correctly (Gwen Shapira via Hari)
        Hide
        gwenshap Gwen Shapira added a comment -

        proposed fix:

        Don't try to count time in loops, the inaccuracies in System.getCurrentTimeMillis accumulate.
        Instead calculate the limit and compare getCurrentTimeMillis to that.

        Show
        gwenshap Gwen Shapira added a comment - proposed fix: Don't try to count time in loops, the inaccuracies in System.getCurrentTimeMillis accumulate. Instead calculate the limit and compare getCurrentTimeMillis to that.

          People

          • Assignee:
            gwenshap Gwen Shapira
            Reporter:
            gwenshap Gwen Shapira
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development