Flume
  1. Flume
  2. FLUME-2207

HDFS file suffix style index suffix in ElasticSearchSink

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v1.5.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Currently ElasticSearchSink indexName is appended with "YYYY-MM-DD" to generate actual ElasticSearch index name and allow index rolling on a daily basis. This works well for simpler use cases but is restrictive imho. User should be able to add indexSuffix as per their own use case and also keep the indexSuffix empty if requirement demands.

      agent.sinks.essink.indexName = flume becomes flume-2013-10-10.

      It will be really useful to have HDFS Sink filepath style indexSuffix naming i.e. "YYYY" will roll the indexes yearly, "YYYY-MM" will create indices monthly.

      1. FLUME-2207-1.patch
        0.8 kB
        Satoshi Iijima

        Issue Links

          Activity

          Hide
          Satoshi Iijima added a comment -

          No, not yet.

          Hari or other commiters - Please review and merge this patch. This is a small bug.

          Show
          Satoshi Iijima added a comment - No, not yet. Hari or other commiters - Please review and merge this patch. This is a small bug.
          Hide
          Jean lagarutte added a comment -

          thanks for your response
          is this patch already integrated in the trunk branch ?

          Show
          Jean lagarutte added a comment - thanks for your response is this patch already integrated in the trunk branch ?
          Hide
          Satoshi Iijima added a comment -

          Hi,
          Example is below. Please try it.

          a1.sinks.k1.type = elasticsearch
          a1.sinks.k1.indexNameBuilder.dateFormat = yyyy-MM-dd-HH
          a1.sinks.k1.indexNameBuilder.timeZone = America/Los_Angeles
          a1.sinks.k1.indexName = foo_index
          a1.sinks.k1.indexType = bar_type
          a1.sinks.k1.hostNames = 127.0.0.1:9200
          a1.sinks.k1.channel = c1

          Show
          Satoshi Iijima added a comment - Hi, Example is below. Please try it. a1.sinks.k1.type = elasticsearch a1.sinks.k1.indexNameBuilder.dateFormat = yyyy-MM-dd-HH a1.sinks.k1.indexNameBuilder.timeZone = America/Los_Angeles a1.sinks.k1.indexName = foo_index a1.sinks.k1.indexType = bar_type a1.sinks.k1.hostNames = 127.0.0.1:9200 a1.sinks.k1.channel = c1
          Hide
          Jean lagarutte added a comment -

          hi
          how can we configure the TimeBasedIndexNameBuilder with your patch.
          what is the propertie name and the format ?

          regards

          Show
          Jean lagarutte added a comment - hi how can we configure the TimeBasedIndexNameBuilder with your patch. what is the propertie name and the format ? regards
          Hide
          Dripple added a comment -

          Hi,

          Any chances it gets fixed in 1.5.x version ?

          Cheers
          D.

          Show
          Dripple added a comment - Hi, Any chances it gets fixed in 1.5.x version ? Cheers D.
          Hide
          Satoshi Iijima added a comment -

          This feature has already been implemented by TimeBasedIndexNameBuilder which is the default indexNameBuilder. But it does not work because of a small bug. I have attached a patch.

          Show
          Satoshi Iijima added a comment - This feature has already been implemented by TimeBasedIndexNameBuilder which is the default indexNameBuilder. But it does not work because of a small bug. I have attached a patch.

            People

            • Assignee:
              Unassigned
              Reporter:
              Dib Ghosh
            • Votes:
              4 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development