Pig
  1. Pig
  2. PIG-2824

Pushing checking number of fields into LoadFunc

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.9.0, 0.10.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      As described in PIG-1188, if users define a schema (w or w/o types), we need to check the number of fields after loading data, so if there are less fields we need to pad null fields, and if there are more fields we need to throw them away.

      For schema with types, Pig used to insert a Foreach after the loader for type casting which also checks #fields. For schema without types there was no such Foreach, thus PIG-1188 inserted one just for checking #fields. Unfortunately, Foreach is too expensive for such checking, and ideally we can push it into the loader.

      1. 2824.png
        45 kB
        Jie Li
      2. 2824.patch
        60 kB
        Jie Li

        Issue Links

          Activity

          Jie Li made changes -
          Link This issue is related to PIG-2661 [ PIG-2661 ]
          Alan Gates made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jie Li made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jie Li made changes -
          Attachment 2824.patch [ 12537251 ]
          Jie Li made changes -
          Field Original Value New Value
          Attachment 2824.png [ 12536912 ]
          Jie Li created issue -

            People

            • Assignee:
              Unassigned
              Reporter:
              Jie Li
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development