Pig
  1. Pig
  2. PIG-1745

Disable converting bytes loading from BinStorage

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.9.0
    • Component/s: impl
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      1. Converting bytes loading from BinStorage() will now result an error.
      2. If user clearly knows the caster for the BinStorage, he/she can pass the caster class name to the construct of BinStorage. For example, if data is load from PigStorage (or other LoadFunc using Utf8StorageConverter) and store using BinStorage, user can pass "org.apache.pig.builtin.Utf8StorageConverter" to the construct of BinStorage (or pass "Utf8StorageConverter", since we will search for "org.apache.pig.builtin" package by default). By doing this, converting bytes to other type will still work.
      Show
      1. Converting bytes loading from BinStorage() will now result an error. 2. If user clearly knows the caster for the BinStorage, he/she can pass the caster class name to the construct of BinStorage. For example, if data is load from PigStorage (or other LoadFunc using Utf8StorageConverter) and store using BinStorage, user can pass "org.apache.pig.builtin.Utf8StorageConverter" to the construct of BinStorage (or pass "Utf8StorageConverter", since we will search for "org.apache.pig.builtin" package by default). By doing this, converting bytes to other type will still work.

      Description

      If we load bytes from BinStorage, we don't actually know how we get these bytes originally, and we will not have a way to cast those bytes. Ideally we shall encode caster into BinStorage data file, but we are not there yet. Currrently bytesToXXX methods for BinStorage is wrong and it results unexpected errors. Eg.

      a = load '1.txt' as (a0, a1, a2);
      store a into '1.bin' as BinStorage();
      
      a = load '1.bin' using BinStorage as (a0, a1, a2);
      b = foreach a generate (long)a0;
      dump b;
      

      The code will run but produce wrong data. It's less confusing if we throw an exception in this case.

      Release Notes:
      Pig will throw exception in the case we want to convert bytes loading from BinStorage

      1. PIG-1745-3.patch
        13 kB
        Daniel Dai
      2. PIG-1745-2.patch
        13 kB
        Daniel Dai
      3. PIG-1745-1.patch
        9 kB
        Daniel Dai

        Issue Links

          Activity

            People

            • Assignee:
              Daniel Dai
              Reporter:
              Daniel Dai
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development