Flume
  1. Flume
  2. FLUME-1972

ElasticSearchSink now allows the id of the document to be provided by a custom class

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: v1.3.1
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Sometimes, you want to set the id on the document that is being written to elasticsearch so that you can later search by that id or overwrite previous versions of the same document (as they have the same id).

      This enhancement allows developers to provide a custom class that can read the incoming Avro event and determine the ID from it.

      We currently have this enhancement working successfully against Flume 1.3.1. A patch against trunk will be provided shortly.

      1. FLUME-1972-1.4.0.patch
        14 kB
        Edward Sargisson
      2. FLUME-1972-1.4.0-1.patch
        17 kB
        Edward Sargisson
      3. FLUME-1972-1.4.0-2.patch
        4 kB
        Edward Sargisson
      4. FLUME-1972-1.4.0-3.patch
        24 kB
        Edward Sargisson
      5. FLUME-1972-1.4.0-4.patch
        24 kB
        Edward Sargisson

        Issue Links

          Activity

          Edward Sargisson created issue -
          Edward Sargisson made changes -
          Field Original Value New Value
          Attachment FLUME-1972-1.4.0.patch [ 12577256 ]
          Edward Sargisson made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Edward Sargisson added a comment -

          FLUME-1927-1.4.0-1.patch is now the patch to use. It's based on flume-1.4 branch at commit 5b9d31f1ad228 (2013-04-07).

          Show
          Edward Sargisson added a comment - FLUME-1927-1.4.0-1.patch is now the patch to use. It's based on flume-1.4 branch at commit 5b9d31f1ad228 (2013-04-07).
          Edward Sargisson made changes -
          Attachment FLUME-1972-1.4.0-1.patch [ 12577841 ]
          Edward Sargisson made changes -
          Summary ElasticSearcSink now allows the id of the document to be provided by a custom class ElasticSearchSink now allows the id of the document to be provided by a custom class
          Hide
          Edward Sargisson added a comment -

          Based on review comments I am attaching FLUME-1972-1.4.0-2.patch which is an incremental patch on FLUME-1972-1.4.0-1 which includes an example ElasticsearchIdProvider, associated unit test and documentation updates.

          Show
          Edward Sargisson added a comment - Based on review comments I am attaching FLUME-1972 -1.4.0-2.patch which is an incremental patch on FLUME-1972 -1.4.0-1 which includes an example ElasticsearchIdProvider, associated unit test and documentation updates.
          Edward Sargisson made changes -
          Attachment FLUME-1972-1.4.0-2.patch [ 12578033 ]
          Hide
          Edward Sargisson added a comment -

          New cumulative patch with rework from review attached. FLUME-1972-1.4.0-3

          Show
          Edward Sargisson added a comment - New cumulative patch with rework from review attached. FLUME-1972 -1.4.0-3
          Edward Sargisson made changes -
          Attachment FLUME-1972-1.4.0-3.patch [ 12578258 ]
          Edward Sargisson made changes -
          Remote Link This issue links to "Review Board (Web Link)" [ 12156 ]
          Hide
          Edward Sargisson added a comment -

          New cumulative patch, FLUME-1972-1.4.0-4.patch, attached. Removes the instantiation of the ElasticsearchNullIdProvider if the user-configured provider class cannot be instantiated.

          Show
          Edward Sargisson added a comment - New cumulative patch, FLUME-1972 -1.4.0-4.patch, attached. Removes the instantiation of the ElasticsearchNullIdProvider if the user-configured provider class cannot be instantiated.
          Edward Sargisson made changes -
          Attachment FLUME-1972-1.4.0-4.patch [ 12579457 ]
          Hide
          Edward Sargisson added a comment -

          See flume-1782 for the latest patch.

          Show
          Edward Sargisson added a comment - See flume-1782 for the latest patch.
          Edward Sargisson made changes -
          Link This issue is related to FLUME-2015 [ FLUME-2015 ]
          Hide
          Edward Sargisson added a comment -

          FLUME-2015 is a better solution to this problem. Closing.

          Show
          Edward Sargisson added a comment - FLUME-2015 is a better solution to this problem. Closing.
          Edward Sargisson made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Won't Fix [ 2 ]
          Edward Sargisson made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Mike Percy made changes -
          Fix Version/s v1.4.0 [ 12323372 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          2m 9s 1 Edward Sargisson 05/Apr/13 18:40
          Patch Available Patch Available Resolved Resolved
          21d 2h 37m 1 Edward Sargisson 26/Apr/13 21:17
          Resolved Resolved Closed Closed
          5s 1 Edward Sargisson 26/Apr/13 21:17

            People

            • Assignee:
              Unassigned
              Reporter:
              Edward Sargisson
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development