Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6168

Allow modification of JSON value before writing to ElasticSearch

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: io-java-elasticsearch
    • Labels:
      None

      Description

      I have an Apache Beam streaming job which reads data from Kafka and writes to ElasticSearch using ElasticSearchIO.

      The issue I'm having is that messages in Kafka already have key field, and using ElasticSearchIO.Write.withIdFn() I'm mapping this field to document _id field in ElasticSearch.

      Having a big volume of data I don't want the key field to be also written to ElasticSearch as part of _source.

      Is there an option/workaround that would allow doing that?

        Attachments

          Activity

            People

            • Assignee:
              timrobertson100 Tim Robertson
              Reporter:
              marknorkin Mark Norkin
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: