Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-449

Add sequentialDo function for injecting arbitrary non-parallel code

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.4, 0.11.0
    • Component/s: Core
    • Labels:
      None

      Description

      I've been noodling on this one for awhile: how to add the ability to execute some code if and only if one or more targets are created, and have that executed code (optionally) return one or more new PCollections as a result. I was thinking that this functionality could be wired in to libraries to do things like bulk loading HBase tables or running Sqoop jobs as part of Crunch pipelines automatically.

        Attachments

        1. CRUNCH-449e.patch
          84 kB
          Josh Wills
        2. CRUNCH-449d.patch
          82 kB
          Josh Wills
        3. CRUNCH-449c.patch
          71 kB
          Josh Wills
        4. CRUNCH-449b.patch
          58 kB
          Josh Wills
        5. CRUNCH-449.patch
          60 kB
          Josh Wills

          Activity

            People

            • Assignee:
              jwills Josh Wills
              Reporter:
              jwills Josh Wills
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: