Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.5.2
-
None
-
None
-
linux CentOS 6.6
Description
Flume has been configured in order to populate a hive table as follow :
agentSCDR.sinks.hive1.type = hive
agentSCDR.sinks.hive1.channel = channel1
agentSCDR.sinks.hive1.hive.metastore = thrift://myserver:9083
agentSCDR.sinks.hive1.hive.txnsPerBatchAsk = 10
agentSCDR.sinks.hive1.hive.database = myDatabase
agentSCDR.sinks.hive1.hive.table = my_table
agentSCDR.sinks.hive1.hive.partition = %Y
agentSCDR.sinks.hive1.heartBeatInterval = 60
agentSCDR.sinks.hive1.serializer = DELIMITED
agentSCDR.sinks.hive1.serializer.delimiter = ;
agentSCDR.sinks.hive1.serializer.serdeSeparator = ;
agentSCDR.sinks.hive1.serializer.fieldnames = field1,field2,field3...
My dataflow is irregular at best and I can have more than 30 minutes of inactivity, so I have a heartbeatInterval at 60s in order to keep my txn alive.
The issue is that this heartBeat is only sent when flume is trying to inject some datas into hive instead of having one every 60s.
- grep -i heartbeat flume-agentSCDR.log
15 juil. 2016 13:40:43,008 INFO [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:238) - Sending heartbeat on batch TxnIds=[3755...3764] on endPoint = {metaStoreUri=...
15 juil. 2016 14:12:21,001 INFO [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:231) - Sending heartbeat on batch TxnIds=[3785...3794] on endPoint = {metaStoreUri=...
15 juil. 2016 14:27:56,963 INFO [hive-hive1-call-runner-0] (org.apache.flume.sink.hive.HiveWriter$2.call:231) - Sending heartbeat on batch TxnIds=[3795...3804] on endPoint = {metaStoreUri=...
...
