Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7450

Support unbounded reads with HCatalogIO

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.14.0
    • Component/s: io-java-hcatalog
    • Labels:
      None

      Description

      1. Current version of HcatalogIO is a bounded source.
      2. While migrating our jobs to aws, we realized that it would be helpful to have an unbounded hcat reader that can behave as an unbounded source and polls for new partitions as and when they become available.
      3. I have used splittable pardo(s) to do this. There is a flag that can be set to treat this as a bounded source which will terminate if that flag is set.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              jhalarua Ankit Jhalaria
              Reporter:
              jhalarua Ankit Jhalaria

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 12h 40m
                12h 40m

                  Issue deployment