Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6654

Support SDF within ElasticsearchIO

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: In Progress
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: io-java-elasticsearch
    • Labels:
      None

      Description

      Similar to HbaseIO.ReadAll, it would be very useful to support a SDF based transform which can compose queries based off of input elements. 

      Implementation would split the work similarly to how the current Source API implementation works (leveraging slices within Elasticsearch).

      This work can facilitate a transition of ElasticsearchIO to SDF when more runners have implemented support.  Until then, I propose leaving the read() implementation alone and add a readAll method and associated transform to be implemented as an SDF.  Once enough runner support is ready, the existing Read transform would be updated to use the same SDF as the ReadAll transform with a null input element.

      A prototype implementation currently exists (albeit developed in a bubble) at https://github.com/0xdecaf/beam/tree/feature/elasticsearchio.readall 

       

        Attachments

          Activity

            People

            • Assignee:
              tmoulton Tony Moulton
              Reporter:
              tmoulton Tony Moulton
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: