Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Won't Fix
-
0.9.3
-
None
Description
In agent,I config as: rpcSource(333)| agentDFOChain("157.5.210.6:17876","157.5.210.5:17876")
In collector,I config as: collector(17876)|myCustomplugin
At agent mashine, I use a flume rpc client send 1000 event per sencond, after about five munite, i looked the agent monitor page, the metric "source.LazyOpenSource.ThriftEventSource.queueCapacity " is 1000 ,which is the default size of the blocke queue in thrifteventsource,the "Source.LazyOpenSource.ThriftEventSource.enqueued" is 1357
when I use flume shell commit reconfig agent: exec config agent_01 'rpcSource(333)' 'agentDFOChain("157.5.210.6:17876","157.5.210.5:17876")'
I found the rpcsource cann't be reopen, the netstat command can't found port 3333 is in listen, in log, I found the SpawningLogicalNode thread is spawn on every heartbeat
[2011-12-12 08:03:15,670] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.handlers.thrift.ThriftEventSource 164] Closed server on port 3333...
[2011-12-12 08:03:15,670] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.handlers.thrift.ThriftEventSource 167] Queue still has 898 elements ...
[2011-12-12 08:03:18,614] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen =>
[2011-12-12 08:03:18,614] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:03:18,615] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:03:23,630] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)")
[2011-12-12 08:03:23,630] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen =>
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >
[2011-12-12 08:03:23,661] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen =>
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >
[2011-12-12 08:03:28,643] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen =>
[2011-12-12 08:03:28,644] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:03:28,644] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:03:33,659] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)")
[2011-12-12 08:03:33,659] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen =>
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >
[2011-12-12 08:03:33,659] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen =>
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >
[2011-12-12 08:03:38,665] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen =>
[2011-12-12 08:03:38,704] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:03:38,704] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:03:43,672] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)")
[2011-12-12 08:03:43,673] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen =>
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >
[2011-12-12 08:03:43,673] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen =>
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >
[2011-12-12 08:03:48,680] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen =>
[2011-12-12 08:03:48,680] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:03:48,680] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:03:53,684] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)")
[2011-12-12 08:03:53,685] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen =>
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >
[2011-12-12 08:03:53,764] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen =>
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >
[2011-12-12 08:03:58,688] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen =>
[2011-12-12 08:03:58,688] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:03:58,688] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:04:03,699] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)")
[2011-12-12 08:04:03,699] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen =>
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >
[2011-12-12 08:04:03,699] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen =>
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >
[2011-12-12 08:04:08,732] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen =>
[2011-12-12 08:04:08,771] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:04:08,773] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:04:13,741] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)")
[2011-12-12 08:04:13,741] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen =>
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >
[2011-12-12 08:04:13,742] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen =>
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >
[2011-12-12 08:04:18,755] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen =>
[2011-12-12 08:04:18,755] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:04:18,755] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:04:23,764] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)")
[2011-12-12 08:04:23,764] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen =>
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >
[2011-12-12 08:04:23,819] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen =>
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >
[2011-12-12 08:04:28,772] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen =>
[2011-12-12 08:04:28,773] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:04:28,773] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:04:33,782] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)")
[2011-12-12 08:04:33,782] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen =>
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >
[2011-12-12 08:04:33,782] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen =>
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >
[2011-12-12 08:04:38,792] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen =>
[2011-12-12 08:04:38,792] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 164] Setting dfo failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:04:38,792] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 73] Setting failover chain to < failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") ? diskFailover insistentAppend stubbornAppend insistentOpen failChain("{ lazyOpen =>
{ stubbornAppend => %s } } ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)") >[2011-12-12 08:04:43,796] [INFO ] [SpawningLogicalNode pm_C04_JQ3_CNA03_E6000_a] [com.cloudera.flume.agent.AgentFailChainSink 135] Setting best effort failover chain to failChain("{ lazyOpen => { stubbornAppend => %s }
} ","tsink(\"157.5.210.5\",17876)","tsink(\"157.5.210.6\",17876)")