Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2956

hive sink not sending heartbeat correctly

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.5.2
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Environment:

      linux CentOS 6.6

      Description

      Flume has been configured in order to populate a hive table as follow :

      agentSCDR.sinks.hive1.type = hive
      agentSCDR.sinks.hive1.channel = channel1
      agentSCDR.sinks.hive1.hive.metastore = thrift://myserver:9083
      agentSCDR.sinks.hive1.hive.txnsPerBatchAsk = 10
      agentSCDR.sinks.hive1.hive.database = myDatabase
      agentSCDR.sinks.hive1.hive.table = my_table
      agentSCDR.sinks.hive1.hive.partition = %Y
      agentSCDR.sinks.hive1.heartBeatInterval = 60
      agentSCDR.sinks.hive1.serializer = DELIMITED
      agentSCDR.sinks.hive1.serializer.delimiter = ;
      agentSCDR.sinks.hive1.serializer.serdeSeparator = ;
      agentSCDR.sinks.hive1.serializer.fieldnames = field1,field2,field3...

      My dataflow is irregular at best and I can have more than 30 minutes of inactivity, so I have a heartbeatInterval at 60s in order to keep my txn alive.

      The issue is that this heartBeat is only sent when flume is trying to inject some datas into hive instead of having one every 60s.

      1. grep -i heartbeat flume-agentSCDR.log
        15 juil. 2016 13:40:43,008 INFO [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:238) - Sending heartbeat on batch TxnIds=[3755...3764] on endPoint = {metaStoreUri=...
        15 juil. 2016 14:12:21,001 INFO [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:231) - Sending heartbeat on batch TxnIds=[3785...3794] on endPoint = {metaStoreUri=...
        15 juil. 2016 14:27:56,963 INFO [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:231) - Sending heartbeat on batch TxnIds=[3795...3804] on endPoint = {metaStoreUri=...
        ...

        Attachments

        1. hive-sink-patch.txt
          3 kB
          He Jiang
        2. flume-hive-sink-1.8.0.jar
          43 kB
          He Jiang

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              olivier.brobecker Olivier brobecker
            • Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: