Pig
  1. Pig
  2. PIG-2505

AvroStorage won't read any file not ending in .avro

    Details

    • Patch Info:
      Patch Available
    • Hadoop Flags:
      Reviewed

      Description

      AvroStorage will not load any files that do not end in .avro; This is problematic when you are reading part of the output of a hadoop job, such as 'part-000001', etc.

      1. expected_testFileWithNoExtension.avro
        5 kB
        Daniel Dai
      2. PIG-2505.patch
        1 kB
        Russell Jurney
      3. PIG-2505.tests.patch
        2 kB
        Russell Jurney
      4. PIG-2505-2.patch
        4 kB
        Daniel Dai

        Activity

        Russell Jurney created issue -
        Russell Jurney made changes -
        Field Original Value New Value
        Labels avro fun happy pants pig pig_avro pig_udf
        Tags pig
        Hide
        Russell Jurney added a comment -

        Removes the check for .avro in PigAvroInputFormat.java

        Show
        Russell Jurney added a comment - Removes the check for .avro in PigAvroInputFormat.java
        Russell Jurney made changes -
        Attachment PIG-2505.patch [ 12513185 ]
        Russell Jurney made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Russell Jurney added a comment -

        Test added for the other patch.

        Show
        Russell Jurney added a comment - Test added for the other patch.
        Russell Jurney made changes -
        Attachment PIG-2505.tests.patch [ 12516265 ]
        Russell Jurney made changes -
        Fix Version/s 0.10 [ 12316246 ]
        Affects Version/s 0.10 [ 12316246 ]
        Hide
        Daniel Dai added a comment -

        Shall we check the directory name ends with .avro in this case?

        Show
        Daniel Dai added a comment - Shall we check the directory name ends with .avro in this case?
        Hide
        Russell Jurney added a comment -

        We want it to read any directory or file, regardless of filename.

        Show
        Russell Jurney added a comment - We want it to read any directory or file, regardless of filename.
        Hide
        Daniel Dai added a comment -

        The input file for test test_no_extension is not in the patch. Can you attach?

        Show
        Daniel Dai added a comment - The input file for test test_no_extension is not in the patch. Can you attach?
        Hide
        Daniel Dai added a comment -

        Merge patches and resync with trunk. Still waiting for test_no_extension.

        Show
        Daniel Dai added a comment - Merge patches and resync with trunk. Still waiting for test_no_extension.
        Daniel Dai made changes -
        Attachment PIG-2505-2.patch [ 12518765 ]
        Hide
        Russell Jurney added a comment -

        Coming up.

        Show
        Russell Jurney added a comment - Coming up.
        Hide
        Russell Jurney added a comment -

        I don't know how to submit a patch when a binary file is added. So here's a recipe.

        cd contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/avro_test_files/
        cp test_record.avro test_no_extension

        Show
        Russell Jurney added a comment - I don't know how to submit a patch when a binary file is added. So here's a recipe. cd contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/avro_test_files/ cp test_record.avro test_no_extension
        Hide
        Daniel Dai added a comment -

        Let me try

        Show
        Daniel Dai added a comment - Let me try
        Daniel Dai made changes -
        Attachment PIG-2505-2.patch [ 12518765 ]
        Daniel Dai made changes -
        Attachment PIG-2505-2.patch [ 12519007 ]
        Attachment expected_testFileWithNoExtension.avro [ 12519008 ]
        Hide
        Daniel Dai added a comment -

        Patch committed to 0.10/trunk. Thanks Russell!

        Show
        Daniel Dai added a comment - Patch committed to 0.10/trunk. Thanks Russell!
        Daniel Dai made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Assignee Russell Jurney [ rjurney ]
        Fix Version/s 0.11 [ 12318878 ]
        Resolution Fixed [ 1 ]
        Hide
        Jonathan Coveney added a comment -

        Daniel: the test_no_exception file didn't get added.

        Show
        Jonathan Coveney added a comment - Daniel: the test_no_exception file didn't get added.
        Hide
        Daniel Dai added a comment -

        Thanks, added now.

        Show
        Daniel Dai added a comment - Thanks, added now.
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Gavin made changes -
        Assignee Russell Jurney [ rjurney ] Russell Jurney [ russell.jurney ]

          People

          • Assignee:
            Russell Jurney
            Reporter:
            Russell Jurney
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development