[FLUME-2207] HDFS file suffix style index suffix in ElasticSearchSink

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.5.0
• Component/s: Sinks+Sources
• Assignee: Unassigned
• Reporter: Dib Ghosh
• Created: 08/Oct/13 18:55
• Updated: 18/Feb/16 16:46

Description

Currently ElasticSearchSink indexName is appended with "YYYY-MM-DD" to generate actual ElasticSearch index name and allow index rolling on a daily basis. This works well for simpler use cases but is restrictive imho. User should be able to add indexSuffix as per their own use case and also keep the indexSuffix empty if requirement demands.

agent.sinks.essink.indexName = flume becomes flume-2013-10-10.

It will be really useful to have HDFS Sink filepath style indexSuffix naming i.e. "YYYY" will roll the indexes yearly, "YYYY-MM" will create indices monthly.

Issue Links

• is depended upon by: FLUME-2884 Tag not allowed in ElasticSearch indexNameBuilder properties
• is duplicated by: FLUME-2576 Elasticsearch sink: customizing dateFormat for IndexNameBuilder does not work
• is duplicated by: FLUME-2876 ElasticSearchSink: indexnameBuilderContext.putAll bug fixes
• is duplicated by: FLUME-2391 Turn Elasticsearch sink index timestamp parametrizable
• is part of: FLUME-2435 elasticsearchsink: IndexNameBuilder configuration fails
• is related to: FLUME-2424 allow index timestamp format option for ES sink
• links to: Review Board