Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2824

Pushing checking number of fields into LoadFunc

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.9.0, 0.10.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      As described in PIG-1188, if users define a schema (w or w/o types), we need to check the number of fields after loading data, so if there are less fields we need to pad null fields, and if there are more fields we need to throw them away.

      For schema with types, Pig used to insert a Foreach after the loader for type casting which also checks #fields. For schema without types there was no such Foreach, thus PIG-1188 inserted one just for checking #fields. Unfortunately, Foreach is too expensive for such checking, and ideally we can push it into the loader.

        Attachments

        1. 2824.patch
          60 kB
          Jie Li
        2. 2824.png
          45 kB
          Jie Li

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jay23jack Jie Li
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: