Flume
  1. Flume
  2. FLUME-2206

ElasticSearchSink ttl field modification to mimic Elasticsearch way of specifying TTL

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: v1.5.0
    • Fix Version/s: v1.5.0
    • Component/s: Sinks+Sources
    • Labels:

      Description

      According to current ElasticSearch documentation of TTL:

      "In case you do not specify a time unit like d (days), m (minutes), h (hours), ms (milliseconds) or w (weeks), milliseconds is used as default unit.
      If no default is set and no _ttl value is given then the document has an infinite _ttl and will not expire."

      Current flume ElasticSearchSink index TTL configuration:

      agent.sinks.myESsink.ttl = <some integer value> and defaults to 5 days.

      It will be good to have a qualifier like "d" / "s" / "m" / "w" / "h" to mimic the TTL configuration in ElasticSearch mapping.

      When user specifies agent.sinks.myESsink.ttl = "3w" index TTL should be set to 3 weeks while agent.sinks.myESsink.ttl = 3 will set the TTL to 3 days.

        Activity

        Hide
        Dib Ghosh added a comment -

        I have already made the changes and currently testing it locally. Can somebody please assign the JIRA ticket to me?

        Thanks

        Show
        Dib Ghosh added a comment - I have already made the changes and currently testing it locally. Can somebody please assign the JIRA ticket to me? Thanks
        Hide
        Hari Shreedharan added a comment -

        Done

        Show
        Hari Shreedharan added a comment - Done
        Hide
        Dib Ghosh added a comment -

        Thanks Hari..

        Show
        Dib Ghosh added a comment - Thanks Hari..
        Hide
        Dib Ghosh added a comment -
        Show
        Dib Ghosh added a comment - Opened reviewboard: https://reviews.apache.org/r/14614/
        Hide
        Dib Ghosh added a comment -

        patch for accepting qualified TTL values in ElasticSearch sink settings of flume along with existing way of specifying TTL.

        Show
        Dib Ghosh added a comment - patch for accepting qualified TTL values in ElasticSearch sink settings of flume along with existing way of specifying TTL.
        Hide
        Hari Shreedharan added a comment -

        +1

        Show
        Hari Shreedharan added a comment - +1
        Hide
        Hari Shreedharan added a comment -

        I made a couple of minor changes and committed this, rev: 6dfe63c. Thanks Dib!

        Show
        Hari Shreedharan added a comment - I made a couple of minor changes and committed this, rev: 6dfe63c. Thanks Dib!

          People

          • Assignee:
            Dib Ghosh
            Reporter:
            Dib Ghosh
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development