Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7450

Support unbounded reads with HCatalogIO

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.14.0
    • Component/s: io-java-hcatalog
    • Labels:
      None

      Description

      1. Current version of HcatalogIO is a bounded source.
      2. While migrating our jobs to aws, we realized that it would be helpful to have an unbounded hcat reader that can behave as an unbounded source and polls for new partitions as and when they become available.
      3. I have used splittable pardo(s) to do this. There is a flag that can be set to treat this as a bounded source which will terminate if that flag is set.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jhalarua Ankit Jhalaria
                Reporter:
                jhalarua Ankit Jhalaria
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 12h 40m
                  12h 40m