Pig
  1. Pig
  2. PIG-2877

Make SchemaTuple work in foreach (and thus, in loads)

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Implemented
    • Affects Version/s: None
    • Fix Version/s: 0.11
    • Component/s: None
    • Labels:
      None

      Description

      This wires foreaches to use SchemaTuple. As an aside, while SchemaTuple is not turned on by default, I made sure that it worked with ant test-commit if it were (there were some edge cases and lifecycle stuff that needed to be cleaned up).

      Further, I refactored some of the tests that I needed to work with for other reasons to be much cleaner.

      1. PIG-2877-0.patch
        91 kB
        Jonathan Coveney
      2. PIG-2877-1.patch
        92 kB
        Jonathan Coveney
      3. PIG-2877-2.patch
        106 kB
        Jonathan Coveney
      4. PIG-2877-3.patch
        109 kB
        Jonathan Coveney

        Activity

        Hide
        Dmitriy V. Ryaboy added a comment -

        great to see this. Reviewboard?
        Is a branch with this patch available on github somewhere?

        Show
        Dmitriy V. Ryaboy added a comment - great to see this. Reviewboard? Is a branch with this patch available on github somewhere?
        Hide
        Dmitriy V. Ryaboy added a comment -

        oh.. and can you post a patch with --no-prefix?

        Show
        Dmitriy V. Ryaboy added a comment - oh.. and can you post a patch with --no-prefix?
        Hide
        Jonathan Coveney added a comment -
        Show
        Jonathan Coveney added a comment - https://reviews.apache.org/r/7336/ and a new patch
        Hide
        Dmitriy V. Ryaboy added a comment -

        how badly will it mess you up if we remove mandatory foreaches after loads?

        Show
        Dmitriy V. Ryaboy added a comment - how badly will it mess you up if we remove mandatory foreaches after loads?
        Hide
        Jonathan Coveney added a comment -

        What's the new pattern? When will the casting in the case of an AS statement happen? It won't screw me up, it just means for it to work on loads I'll have to move the logic to wherever the new casting post-load happens. That said, I support that change.

        Show
        Jonathan Coveney added a comment - What's the new pattern? When will the casting in the case of an AS statement happen? It won't screw me up, it just means for it to work on loads I'll have to move the logic to wherever the new casting post-load happens. That said, I support that change.
        Hide
        Dmitriy V. Ryaboy added a comment -

        See PIG-2824 and my longish comment at the bottom of PIG-2661

        Show
        Dmitriy V. Ryaboy added a comment - See PIG-2824 and my longish comment at the bottom of PIG-2661
        Hide
        Jonathan Coveney added a comment -

        Yeah, this would be totally compatible with that. The coercion has to happen somewhere, so whenever that is finished, I can just thread the Schema through there and do it. I think that this is actually ideal, b/c getting rid of the Foreach and directly doing the insertion should be a nice speed boost.

        Show
        Jonathan Coveney added a comment - Yeah, this would be totally compatible with that. The coercion has to happen somewhere, so whenever that is finished, I can just thread the Schema through there and do it. I think that this is actually ideal, b/c getting rid of the Foreach and directly doing the insertion should be a nice speed boost.
        Hide
        Jonathan Coveney added a comment -

        Julien,

        I took into account your comments. Let me know what you think.

        Show
        Jonathan Coveney added a comment - Julien, I took into account your comments. Let me know what you think.
        Hide
        Julien Le Dem added a comment -

        +1

        Show
        Julien Le Dem added a comment - +1

          People

          • Assignee:
            Jonathan Coveney
            Reporter:
            Jonathan Coveney
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development