Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2579

Support for multiple input schemas in AvroStorage

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.9.2, 0.11
    • Fix Version/s: 0.11
    • Component/s: piggybank
    • Labels:
      None
    • Patch Info:
      Patch Available
    • Hadoop Flags:
      Reviewed

      Description

      This is a barebones patch for AvroStorage which enables support of multiple input schemas. The assumption is that the input consists of avro files having different schemas that can be unioned, e.g., flat records.

      A simple illustrative example is attached (avro_storage_union_schema_test.tar.gz): run create_avro1.pig, followed by create_avro2.pig, followed by read_avro.pig.

        Attachments

        1. avro_storage_union_schema_test.tar.gz
          0.6 kB
          Stan Rosenberg
        2. avro_storage_union_schema.patch
          50 kB
          Stan Rosenberg
        3. PIG-2579-2.patch
          51 kB
          Cheolsoo Park
        4. PIG-2579-2-avro_test_files.tar.gz
          1 kB
          Cheolsoo Park
        5. PIG-2579-3.patch
          51 kB
          Cheolsoo Park
        6. PIG-2579-4.patch
          58 kB
          Cheolsoo Park
        7. PIG-2579-5.patch
          58 kB
          Cheolsoo Park
        8. PIG-2579-6.patch
          58 kB
          Cheolsoo Park

          Activity

            People

            • Assignee:
              cheolsoo Cheolsoo Park
              Reporter:
              srosenberg Stan Rosenberg
            • Votes:
              3 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: