Uploaded image for project: 'Crunch (Retired)'
  1. Crunch (Retired)
  2. CRUNCH-146

Jobs given Avro input data without '.avro' extension succeed, but do nothing

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.4.0
    • 0.5.0
    • Core
    • None

    Description

      The Crunch AvroInputFormat class explicitly checks each input path for a '.avro' extension.

      If none of the input paths have such an extension, the job continues and succeeds even though no map tasks get allocated, and none of the pipeline stages are executed.

      ( I notice this behaviour is taken from Avro's own input format. It does happen in standard Avro mapreduce too therefore, but only in the old mapred API - the new mapreduce API version will happily read files regardless of extension. I'm going to report this as a bug on Avro as well...)

      Attachments

        1. CRUNCH-146.patch
          2 kB
          Dave Beech
        2. CRUNCH-146-errorcheck.patch
          2 kB
          Dave Beech

        Activity

          People

            jwills Josh Wills
            dbeech Dave Beech
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: