Pig
  1. Pig
  2. PIG-2505

AvroStorage won't read any file not ending in .avro

    Details

    • Patch Info:
      Patch Available
    • Hadoop Flags:
      Reviewed

      Description

      AvroStorage will not load any files that do not end in .avro; This is problematic when you are reading part of the output of a hadoop job, such as 'part-000001', etc.

      1. PIG-2505.patch
        1 kB
        Russell Jurney
      2. PIG-2505.tests.patch
        2 kB
        Russell Jurney
      3. PIG-2505-2.patch
        4 kB
        Daniel Dai
      4. expected_testFileWithNoExtension.avro
        5 kB
        Daniel Dai

        Activity

        Hide
        Russell Jurney added a comment -

        Removes the check for .avro in PigAvroInputFormat.java

        Show
        Russell Jurney added a comment - Removes the check for .avro in PigAvroInputFormat.java
        Hide
        Russell Jurney added a comment -

        Test added for the other patch.

        Show
        Russell Jurney added a comment - Test added for the other patch.
        Hide
        Daniel Dai added a comment -

        Shall we check the directory name ends with .avro in this case?

        Show
        Daniel Dai added a comment - Shall we check the directory name ends with .avro in this case?
        Hide
        Russell Jurney added a comment -

        We want it to read any directory or file, regardless of filename.

        Show
        Russell Jurney added a comment - We want it to read any directory or file, regardless of filename.
        Hide
        Daniel Dai added a comment -

        The input file for test test_no_extension is not in the patch. Can you attach?

        Show
        Daniel Dai added a comment - The input file for test test_no_extension is not in the patch. Can you attach?
        Hide
        Daniel Dai added a comment -

        Merge patches and resync with trunk. Still waiting for test_no_extension.

        Show
        Daniel Dai added a comment - Merge patches and resync with trunk. Still waiting for test_no_extension.
        Hide
        Russell Jurney added a comment -

        Coming up.

        Show
        Russell Jurney added a comment - Coming up.
        Hide
        Russell Jurney added a comment -

        I don't know how to submit a patch when a binary file is added. So here's a recipe.

        cd contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/avro_test_files/
        cp test_record.avro test_no_extension

        Show
        Russell Jurney added a comment - I don't know how to submit a patch when a binary file is added. So here's a recipe. cd contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/avro_test_files/ cp test_record.avro test_no_extension
        Hide
        Daniel Dai added a comment -

        Let me try

        Show
        Daniel Dai added a comment - Let me try
        Hide
        Daniel Dai added a comment -

        Patch committed to 0.10/trunk. Thanks Russell!

        Show
        Daniel Dai added a comment - Patch committed to 0.10/trunk. Thanks Russell!
        Hide
        Jonathan Coveney added a comment -

        Daniel: the test_no_exception file didn't get added.

        Show
        Jonathan Coveney added a comment - Daniel: the test_no_exception file didn't get added.
        Hide
        Daniel Dai added a comment -

        Thanks, added now.

        Show
        Daniel Dai added a comment - Thanks, added now.

          People

          • Assignee:
            Russell Jurney
            Reporter:
            Russell Jurney
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development