Pig
  1. Pig
  2. PIG-1371

Pig should handle deep casting of complex types

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 0.7.0
    • Component/s: None
    • Labels:
      None

      Description

      Consider input data in BinStorage format which has a field of bag type - bg:

      {t:(i:int)}

      . In the load statement if the schema specified has the type for this field specified as bg:

      {t:(c:chararray)}

      , the current behavior is that Pig thinks of the field to be of type specified in the load statement (bg:

      {t:(c:chararray)}

      ) but no deep cast from bag of int (the real data) to bag of chararray (the user specified schema) is made.

      There are two issues currently:
      1) The TypeCastInserter only considers the byte 'type' between the loader presented schema and user specified schema to decided whether to introduce a cast or not. In the above case since both schema have the type "bag" no cast is inserted. This check has to be extended to consider the full FieldSchema (with inner subschema) in order to decide whether a cast is needed.
      2) POCast should be changed to handle casting a complex type to the type specified the user supplied FieldSchema. Here is there is one issue to be considered - if the user specified the cast type to be bg:

      {t:(i:int, j:int)}

      and the real data had only one field what should the result of the cast be:

      • A bag with two fields - the int field and a null? - In this approach pig is assuming the lone field in the data is the first field which might be incorrect if it in fact is the second field.
      • A null bag to indicate that the bag is of unknown value - this is the one I personally prefer
      • The cast throws an IncompatibleCastException

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          233d 7h 57m 1 Daniel Dai 02/Dec/10 01:54
          Olga Natkovich made changes -
          Fix Version/s 0.9.0 [ 12315191 ]
          Fix Version/s 0.7.0 [ 12314397 ]
          Daniel Dai made changes -
          Link This issue is related to PIG-1758 [ PIG-1758 ]
          Daniel Dai made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]
          Hide
          Daniel Dai added a comment -

          It is actually solved by PIG-613. Verified the test case in partial patch works in trunk.

          Show
          Daniel Dai added a comment - It is actually solved by PIG-613 . Verified the test case in partial patch works in trunk.
          Alan Gates made changes -
          Fix Version/s 0.9.0 [ 12315191 ]
          Alan Gates made changes -
          Assignee Richard Ding [ rding ] Alan Gates [ alangates ]
          Olga Natkovich made changes -
          Fix Version/s 0.8.0 [ 12314562 ]
          Hide
          Olga Natkovich added a comment -

          It does not look like we will have time to do this in 0.8.0

          Show
          Olga Natkovich added a comment - It does not look like we will have time to do this in 0.8.0
          Olga Natkovich made changes -
          Assignee Richard Ding [ rding ]
          Hide
          Olga Natkovich added a comment -

          Richard, can you take a look and see how feasible this is for 0.8.0, thanks

          Show
          Olga Natkovich added a comment - Richard, can you take a look and see how feasible this is for 0.8.0, thanks
          Pradeep Kamath made changes -
          Field Original Value New Value
          Attachment PIG-1371-partial.patch [ 12442907 ]
          Hide
          Pradeep Kamath added a comment -

          partial patch - attaching here for future reference

          Show
          Pradeep Kamath added a comment - partial patch - attaching here for future reference
          Pradeep Kamath created issue -

            People

            • Assignee:
              Alan Gates
              Reporter:
              Pradeep Kamath
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development