Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-663

Expose Record-level File Path to Processing Functions

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: Core
    • Labels:
      None

      Description

      We have some processing pipelines where we want to know the file path that each record being processed came from.  It would be nice if this could be exposed to the DoFns in our pipelines.

       

      This same desire was expressed a little over 1 year ago on the mailing list:
      http://mail-archives.apache.org/mod_mbox/crunch-user/201611.mbox/%3CCAG-tO+Y42KRFiocg1RJT4qFcyvkPjFSfZa4z=wk34AriP4weTw@mail.gmail.com%3E

       

      Unfortunately, that thread dead-ended.

       

      I will use the comments section and a patch to propose a simple, albeit slightly hacky solution.  Another alternative would be to create a new Source that provides a PCollection<Pair<Path, Record>>, but I'm not sure of the effort it would take to create that.

        Attachments

        1. CRUNCH-663.patch
          3 kB
          Ben Roling
        2. CRUNCH-663-v2.patch
          6 kB
          Ben Roling

          Activity

            People

            • Assignee:
              jwills Josh Wills
              Reporter:
              ben.roling Ben Roling
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: