Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-13137

make ElasticsearchIO$BoundedElasticsearchSource#getEstimatedSizeBytes deterministic

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.37.0
    • io-java-elasticsearch
    • None

    Description

      Index size estimation is statistical in ES and varies. But it is the base for splitting so it needs to be more deterministic because that causes flakiness in the UTests in testSplit and testSizes and maybe entails sub-optimal splitting in production in some cases.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. BEAM-5172 org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest is flaky

          • P1 - Critical bug: data loss, total loss of function, or loss of testing signal due to test failures or flakiness
          • Resolved
          links to

          Web Link GitHub Pull Request #16507

          Activity

            People

              egalpin Evan Galpin
              echauchot Etienne Chauchot
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 10m
                  2h 10m