Hive
  1. Hive
  2. HIVE-62

Add ability to describe nested types

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.3.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Add ability to describe nested types.

      1. patch-62.txt
        22 kB
        Ashish Thusoo
      2. patch-62_2.txt
        22 kB
        Ashish Thusoo

        Activity

        Carl Steinbach made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Carl Steinbach made changes -
        Fix Version/s 0.3.0 [ 12313637 ]
        Fix Version/s 0.6.0 [ 12314524 ]
        Component/s Query Processor [ 12312586 ]
        Zheng Shao made changes -
        Fix Version/s 0.6.0 [ 12314524 ]
        Fix Version/s 0.2.0 [ 12313565 ]
        dhruba borthakur made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        dhruba borthakur added a comment -

        I just committed this. Thanks Ashish!

        Show
        dhruba borthakur added a comment - I just committed this. Thanks Ashish!
        Ashish Thusoo made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.20.0 [ 12313506 ]
        Affects Version/s 0.20.0 [ 12313506 ]
        Hide
        Ashish Thusoo added a comment -

        submit patch.

        Show
        Ashish Thusoo added a comment - submit patch.
        Ashish Thusoo made changes -
        Attachment patch-62_2.txt [ 12393984 ]
        Hide
        Ashish Thusoo added a comment -

        Added comments to the test case.

        Show
        Ashish Thusoo added a comment - Added comments to the test case.
        Hide
        Namit Jain added a comment -

        Can you add more comments - specially around the test case : describe_xpath.q ?
        Otherwise, it looks OK.

        +1

        Show
        Namit Jain added a comment - Can you add more comments - specially around the test case : describe_xpath.q ? Otherwise, it looks OK. +1
        Ashish Thusoo made changes -
        Field Original Value New Value
        Attachment patch-62.txt [ 12393973 ]
        Hide
        Ashish Thusoo added a comment -

        Patch added.

        Show
        Ashish Thusoo added a comment - Patch added.
        Hide
        Zheng Shao added a comment -

        Talked with Ashish face-to-face. I agree both approaches have values.

        We can have "describe type" after we added "create type" support.

        Show
        Zheng Shao added a comment - Talked with Ashish face-to-face. I agree both approaches have values. We can have "describe type" after we added "create type" support.
        Hide
        Ashish Thusoo added a comment -

        yes that is a possibility. One problem from a usability angle though is that even if I know the path to the field I would have to go though multiple describe types to get to the member that I am interested in finding the type of. e.g.

        {int a; lObj b; }

        Obj

        { int c; }

        In order to find the type for c I would have to

        describe T;

        which would give me

        int a
        Obj b;

        then I would have to

        describe type Obj;

        which would give me

        int c;

        instead of just saying

        describe T.b

        Having a TYPE keyword though is a good idea, though Oracle and other OORDBMS do not make this explict.

        Check out the Oracle type syntax at

        http://download.oracle.com/docs/cd/B19306_01/server.102/b14357/ch12019.htm

        though Oracle itself does not support the dotted notation to find the type of an element represented by an xpath expression as far as I know.

        Show
        Ashish Thusoo added a comment - yes that is a possibility. One problem from a usability angle though is that even if I know the path to the field I would have to go though multiple describe types to get to the member that I am interested in finding the type of. e.g. {int a; lObj b; } Obj { int c; } In order to find the type for c I would have to describe T; which would give me int a Obj b; then I would have to describe type Obj; which would give me int c; instead of just saying describe T.b Having a TYPE keyword though is a good idea, though Oracle and other OORDBMS do not make this explict. Check out the Oracle type syntax at http://download.oracle.com/docs/cd/B19306_01/server.102/b14357/ch12019.htm though Oracle itself does not support the dotted notation to find the type of an element represented by an xpath expression as far as I know.
        Hide
        Zheng Shao added a comment -

        Another way is to allow describe on types but not columns.

        Basically if "describe table" says the column is of type x. Then we can do "describe type x", and it tells us the internal structure of x, either "map<y,z>", "list<e>", "struct

        { ... }

        ", "primitive" or whatever.

        This "describe type" grammar looks consistent to "create type".

        Show
        Zheng Shao added a comment - Another way is to allow describe on types but not columns. Basically if "describe table" says the column is of type x. Then we can do "describe type x", and it tells us the internal structure of x, either "map<y,z>", "list<e>", "struct { ... } ", "primitive" or whatever. This "describe type" grammar looks consistent to "create type".
        Hide
        Ashish Thusoo added a comment -

        This becomes a bit complex in case a partition of a table is being described.

        today we can describe a partition simply by

        describe tab partspec...

        we could still do that by doing

        describe typeexpr partspec...

        Show
        Ashish Thusoo added a comment - This becomes a bit complex in case a partition of a table is being described. today we can describe a partition simply by describe tab partspec... we could still do that by doing describe typeexpr partspec...
        Hide
        Ashish Thusoo added a comment -

        In hive we can do a describe on the table to get the top level columns of the table. However, there is no way of describing the types within the column in the case when the column is an object.

        e.g.

        one cannot do

        describe tab.c1.f1

        to get field f1 of an object columns c1 of table tab.

        the syntax is fairly straightforward for nested objects but becomes more complex for lists and maps.

        So in order to inspect within the list we can do

        describe tab.c1.f1.__elem.f2 where in __elem means that an element of the list f1 is being inspected.

        similarly

        describe tab.c1.f1.__key.f2
        and

        describe tab.c1.f1.__value.f2

        would be the corresponding things for the key and values parts of a map.

        Thoughts?

        Show
        Ashish Thusoo added a comment - In hive we can do a describe on the table to get the top level columns of the table. However, there is no way of describing the types within the column in the case when the column is an object. e.g. one cannot do describe tab.c1.f1 to get field f1 of an object columns c1 of table tab. the syntax is fairly straightforward for nested objects but becomes more complex for lists and maps. So in order to inspect within the list we can do describe tab.c1.f1.__elem.f2 where in __elem means that an element of the list f1 is being inspected. similarly describe tab.c1.f1.__key.f2 and describe tab.c1.f1.__value.f2 would be the corresponding things for the key and values parts of a map. Thoughts?
        Ashish Thusoo created issue -

          People

          • Assignee:
            Ashish Thusoo
            Reporter:
            Ashish Thusoo
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development