Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2915

The kafka channel using new APIs will be stuck when the sink is avro sink

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.0
    • Fix Version/s: 1.7.0
    • Component/s: Channel
    • Labels:
      None

      Description

      The avro sink was stuck when I using the kafka channel which using the new APIs.
      After couple of hours I found the issue at KafkaChannel.java#L384:

      e.getHeaders().put(KEY_HEADER, record.key());

      and change it to:

      if (record.key() != null) {
      e.getHeaders().put(KEY_HEADER, record.key());
      }

      The reason is:
      record.key() could be null if the user didn't set it. And the avro serialize the event will throw a NullPointerException.

      1. FLUME-2915.patch
        3 kB
        Jeff Holoman
      2. FLUME-2915v2.patch
        3 kB
        Jeff Holoman

        Activity

        Hide
        jholoman Jeff Holoman added a comment -

        Thanks for spotting this. I will take a look at this over the weekend.

        Jeff

        Show
        jholoman Jeff Holoman added a comment - Thanks for spotting this. I will take a look at this over the weekend. Jeff
        Show
        jholoman Jeff Holoman added a comment - https://reviews.apache.org/r/48457/diff/
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit a619cc15748d76357f84bd3d0fa0a3e496ee61a1 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=a619cc1 ]

        FLUME-2915: The kafka channel using new APIs will be stuck when the sink is avro sink

        (Jeff Holoman via Jarek Jarcec Cecho)

        Show
        jira-bot ASF subversion and git services added a comment - Commit a619cc15748d76357f84bd3d0fa0a3e496ee61a1 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=a619cc1 ] FLUME-2915 : The kafka channel using new APIs will be stuck when the sink is avro sink (Jeff Holoman via Jarek Jarcec Cecho)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit f2bdd57fc4f1eaa411c08e71e67eed766af7b731 in flume's branch refs/heads/flume-1.7 from Jarek Jarcec Cecho
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=f2bdd57 ]

        FLUME-2915: The kafka channel using new APIs will be stuck when the sink is avro sink

        (Jeff Holoman via Jarek Jarcec Cecho)

        Show
        jira-bot ASF subversion and git services added a comment - Commit f2bdd57fc4f1eaa411c08e71e67eed766af7b731 in flume's branch refs/heads/flume-1.7 from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=f2bdd57 ] FLUME-2915 : The kafka channel using new APIs will be stuck when the sink is avro sink (Jeff Holoman via Jarek Jarcec Cecho)
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        Thank you Jeff Holoman for your contribution and Grant Henke / Attila Simon for review!

        Show
        jarcec Jarek Jarcec Cecho added a comment - Thank you Jeff Holoman for your contribution and Grant Henke / Attila Simon for review!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Flume-trunk-hbase-1 #160 (See https://builds.apache.org/job/Flume-trunk-hbase-1/160/)
        FLUME-2915: The kafka channel using new APIs will be stuck when the (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=a619cc15748d76357f84bd3d0fa0a3e496ee61a1)

        • flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
        • flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Flume-trunk-hbase-1 #160 (See https://builds.apache.org/job/Flume-trunk-hbase-1/160/ ) FLUME-2915 : The kafka channel using new APIs will be stuck when the (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=a619cc15748d76357f84bd3d0fa0a3e496ee61a1 ) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java

          People

          • Assignee:
            jholoman Jeff Holoman
            Reporter:
            zhangzhonglai Julian Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development