Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-1745

Disable converting bytes loading from BinStorage

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.9.0
    • Component/s: impl
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      1. Converting bytes loading from BinStorage() will now result an error.
      2. If user clearly knows the caster for the BinStorage, he/she can pass the caster class name to the construct of BinStorage. For example, if data is load from PigStorage (or other LoadFunc using Utf8StorageConverter) and store using BinStorage, user can pass "org.apache.pig.builtin.Utf8StorageConverter" to the construct of BinStorage (or pass "Utf8StorageConverter", since we will search for "org.apache.pig.builtin" package by default). By doing this, converting bytes to other type will still work.
      Show
      1. Converting bytes loading from BinStorage() will now result an error. 2. If user clearly knows the caster for the BinStorage, he/she can pass the caster class name to the construct of BinStorage. For example, if data is load from PigStorage (or other LoadFunc using Utf8StorageConverter) and store using BinStorage, user can pass "org.apache.pig.builtin.Utf8StorageConverter" to the construct of BinStorage (or pass "Utf8StorageConverter", since we will search for "org.apache.pig.builtin" package by default). By doing this, converting bytes to other type will still work.

      Description

      If we load bytes from BinStorage, we don't actually know how we get these bytes originally, and we will not have a way to cast those bytes. Ideally we shall encode caster into BinStorage data file, but we are not there yet. Currrently bytesToXXX methods for BinStorage is wrong and it results unexpected errors. Eg.

      a = load '1.txt' as (a0, a1, a2);
      store a into '1.bin' as BinStorage();
      
      a = load '1.bin' using BinStorage as (a0, a1, a2);
      b = foreach a generate (long)a0;
      dump b;
      

      The code will run but produce wrong data. It's less confusing if we throw an exception in this case.

      Release Notes:
      Pig will throw exception in the case we want to convert bytes loading from BinStorage

        Attachments

        1. PIG-1745-1.patch
          9 kB
          Daniel Dai
        2. PIG-1745-2.patch
          13 kB
          Daniel Dai
        3. PIG-1745-3.patch
          13 kB
          Daniel Dai

          Issue Links

            Activity

              People

              • Assignee:
                daijy Daniel Dai
                Reporter:
                daijy Daniel Dai
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: