Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-3201

ElasticsearchIO should allow the user to optionally pass id, type and index per document

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.5.0
    • Component/s: io-java-elasticsearch
    • Labels:
      None

      Description

      Dynamic documents id: Today the ESIO only inserts the payload of the ES documents. Elasticsearch generates a document id for each record inserted. So each new insertion is considered as a new document. Users want to be able to update documents using the IO. So, for the write part of the IO, users should be able to provide a document id so that they could update already stored documents. Providing an id for the documents could also help the user on indempotency.
      Dynamic ES type and ES index: In some cases (streaming pipeline with high throughput) partitioning the PCollection to allow to plug to different ESIO instances (pointing to different index/type) is not very practical, the users would like to be able to set ES index/type per document.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                timrobertson100 Tim Robertson
                Reporter:
                echauchot Etienne Chauchot
              • Votes:
                4 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: