Hive
  1. Hive
  2. HIVE-2215

Add api for marking / querying set of partitions for events

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: Metastore
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. hive_2215.patch
      230 kB
      Ashutosh Chauhan
    2. hive-2215_full-1.patch
      313 kB
      Ashutosh Chauhan

      Issue Links

        Activity

        Hide
        Ashutosh Chauhan added a comment -

        Patch including generated code. Will post on RB without generated code. Incorporates feedback from John on HIVE-2147

        Show
        Ashutosh Chauhan added a comment - Patch including generated code. Will post on RB without generated code. Incorporates feedback from John on HIVE-2147
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/883/
        -----------------------------------------------------------

        Review request for hive and John Sichi.

        Summary
        -------

        Follow-up for HIVE-2147.

        This addresses bug HIVE-2215.
        https://issues.apache.org/jira/browse/HIVE-2215

        Diffs


        trunk/metastore/if/hive_metastore.thrift 1134443
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1134443
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1134443
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1134443
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1134443
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1134443
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1134443
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/MarkPartitionEvent.java PRE-CREATION
        trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java PRE-CREATION
        trunk/metastore/src/model/package.jdo 1134443
        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1134443
        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionSet.java PRE-CREATION
        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1134443

        Diff: https://reviews.apache.org/r/883/diff

        Testing
        -------

        Added test cases for new api.

        Thanks,

        Ashutosh

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/883/ ----------------------------------------------------------- Review request for hive and John Sichi. Summary ------- Follow-up for HIVE-2147 . This addresses bug HIVE-2215 . https://issues.apache.org/jira/browse/HIVE-2215 Diffs trunk/metastore/if/hive_metastore.thrift 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/MarkPartitionEvent.java PRE-CREATION trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java PRE-CREATION trunk/metastore/src/model/package.jdo 1134443 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1134443 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionSet.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1134443 Diff: https://reviews.apache.org/r/883/diff Testing ------- Added test cases for new api. Thanks, Ashutosh
        Hide
        Ashutosh Chauhan added a comment -

        This patch is ready for review.

        Show
        Ashutosh Chauhan added a comment - This patch is ready for review.
        Hide
        Ashutosh Chauhan added a comment -

        Replying to Paul's comments since I closed HIVE-2147 :

        I agree with John's suggestion for PARTITION_EVENTS. For this event table, when will rows be dropped?

        This also needs to be considered. I will prefer to do it in a followup jira to keep this one manageable.

        Also, for when partitions are represented using a string, we've followed the convention that they are called partition names. Can we use that for MPartitionSet?

        Yup, I can rename that.

        Since MPartitionSet.partVals is a string, we should make it indexed, much like partitionName for the PARTITION table.

        In the latest patch, I have made it indexed.

        If you can take a look at the latest patch, that will be great.

        Show
        Ashutosh Chauhan added a comment - Replying to Paul's comments since I closed HIVE-2147 : I agree with John's suggestion for PARTITION_EVENTS. For this event table, when will rows be dropped? This also needs to be considered. I will prefer to do it in a followup jira to keep this one manageable. Also, for when partitions are represented using a string, we've followed the convention that they are called partition names. Can we use that for MPartitionSet? Yup, I can rename that. Since MPartitionSet.partVals is a string, we should make it indexed, much like partitionName for the PARTITION table. In the latest patch, I have made it indexed. If you can take a look at the latest patch, that will be great.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/883/#review822
        -----------------------------------------------------------

        trunk/metastore/src/model/package.jdo
        <https://reviews.apache.org/r/883/#comment1782>

        Does indexing actually work on a LONGVARCHAR field across all DB's of interest?

        • John

        On 2011-06-10 21:24:13, Ashutosh Chauhan wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/883/

        -----------------------------------------------------------

        (Updated 2011-06-10 21:24:13)

        Review request for hive and John Sichi.

        Summary

        -------

        Follow-up for HIVE-2147.

        This addresses bug HIVE-2215.

        https://issues.apache.org/jira/browse/HIVE-2215

        Diffs

        -----

        trunk/metastore/if/hive_metastore.thrift 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/MarkPartitionEvent.java PRE-CREATION

        trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java PRE-CREATION

        trunk/metastore/src/model/package.jdo 1134443

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1134443

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionSet.java PRE-CREATION

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1134443

        Diff: https://reviews.apache.org/r/883/diff

        Testing

        -------

        Added test cases for new api.

        Thanks,

        Ashutosh

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/883/#review822 ----------------------------------------------------------- trunk/metastore/src/model/package.jdo < https://reviews.apache.org/r/883/#comment1782 > Does indexing actually work on a LONGVARCHAR field across all DB's of interest? John On 2011-06-10 21:24:13, Ashutosh Chauhan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/883/ ----------------------------------------------------------- (Updated 2011-06-10 21:24:13) Review request for hive and John Sichi. Summary ------- Follow-up for HIVE-2147 . This addresses bug HIVE-2215 . https://issues.apache.org/jira/browse/HIVE-2215 Diffs ----- trunk/metastore/if/hive_metastore.thrift 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/MarkPartitionEvent.java PRE-CREATION trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java PRE-CREATION trunk/metastore/src/model/package.jdo 1134443 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1134443 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionSet.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1134443 Diff: https://reviews.apache.org/r/883/diff Testing ------- Added test cases for new api. Thanks, Ashutosh
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-06-13 21:47:25, John Sichi wrote:

        > trunk/metastore/src/model/package.jdo, line 670

        > <https://reviews.apache.org/r/883/diff/1/?file=20978#file20978line670>

        >

        > Does indexing actually work on a LONGVARCHAR field across all DB's of interest?

        No, it doesn't. So, I reverted it back to VARCHAR.

        If the rest of the patch looks alright, I will attach a new patch with this change.

        • Ashutosh

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/883/#review822
        -----------------------------------------------------------

        On 2011-06-10 21:24:13, Ashutosh Chauhan wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/883/

        -----------------------------------------------------------

        (Updated 2011-06-10 21:24:13)

        Review request for hive and John Sichi.

        Summary

        -------

        Follow-up for HIVE-2147.

        This addresses bug HIVE-2215.

        https://issues.apache.org/jira/browse/HIVE-2215

        Diffs

        -----

        trunk/metastore/if/hive_metastore.thrift 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/MarkPartitionEvent.java PRE-CREATION

        trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java PRE-CREATION

        trunk/metastore/src/model/package.jdo 1134443

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1134443

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionSet.java PRE-CREATION

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1134443

        Diff: https://reviews.apache.org/r/883/diff

        Testing

        -------

        Added test cases for new api.

        Thanks,

        Ashutosh

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-06-13 21:47:25, John Sichi wrote: > trunk/metastore/src/model/package.jdo, line 670 > < https://reviews.apache.org/r/883/diff/1/?file=20978#file20978line670 > > > Does indexing actually work on a LONGVARCHAR field across all DB's of interest? No, it doesn't. So, I reverted it back to VARCHAR. If the rest of the patch looks alright, I will attach a new patch with this change. Ashutosh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/883/#review822 ----------------------------------------------------------- On 2011-06-10 21:24:13, Ashutosh Chauhan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/883/ ----------------------------------------------------------- (Updated 2011-06-10 21:24:13) Review request for hive and John Sichi. Summary ------- Follow-up for HIVE-2147 . This addresses bug HIVE-2215 . https://issues.apache.org/jira/browse/HIVE-2215 Diffs ----- trunk/metastore/if/hive_metastore.thrift 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/MarkPartitionEvent.java PRE-CREATION trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java PRE-CREATION trunk/metastore/src/model/package.jdo 1134443 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1134443 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionSet.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1134443 Diff: https://reviews.apache.org/r/883/diff Testing ------- Added test cases for new api. Thanks, Ashutosh
        Hide
        John Sichi added a comment -

        Carl, any comments on the latest?

        Show
        John Sichi added a comment - Carl, any comments on the latest?
        Hide
        Carl Steinbach added a comment -

        I'm working on some comments right now. I'll have them posted in an hour.

        Show
        Carl Steinbach added a comment - I'm working on some comments right now. I'll have them posted in an hour.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/883/#review824
        -----------------------------------------------------------

        trunk/metastore/if/hive_metastore.thrift
        <https://reviews.apache.org/r/883/#comment1784>

        I think this should be changed to "PartitionEventType" in order to make it clear that this applies to partitions only. If in the future we need to introduce event types for tables, indexes, etc, then we should add new enums for those event types as well.

        trunk/metastore/if/hive_metastore.thrift
        <https://reviews.apache.org/r/883/#comment1785>

        This should also throw UnknownDBException and UnknownTableException. The same goes for isPartitionMarkedForEvent.

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        <https://reviews.apache.org/r/883/#comment1786>

        Collections aren't required to satisfy an ordering property, so we have to assume the output of this logging statement is ambiguous, e.g. "[a, b]" versus "[b, a]". We should disambiguate this by passing in the part_vals map and logging the key/value pairs instead of just the values.

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        <https://reviews.apache.org/r/883/#comment1789>

        Missing exceptions: UnknownDbException and UnknownTableException.

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        <https://reviews.apache.org/r/883/#comment1787>

        Checking to see if the DB and Table exist should be done in the same database transaction as the rest of the operation. If you do it here there's no guarantee that the db/table will still exist when ms.markPartitionForEvent() is called.

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        <https://reviews.apache.org/r/883/#comment1791>

        Should we add an InvalidPartitionException and UnknownPartitionException? Seems like those are both valid exceptions in this situation.

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        <https://reviews.apache.org/r/883/#comment1788>

        Same issue here as before. These checks need to get pushed into ms.isPartitionMarkedForEvent().

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java
        <https://reviews.apache.org/r/883/#comment1798>

        I think the name of this method is misleading. You're marking a single partition done, not a set of partitions, right?

        Also, in this context being "done" means that the load operation on that partition has completed, so it would be good to include "load" in the name of the method and event class, e.g. "LoadPartitionDoneEvent" and "onLoadPartitionDone".

        trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java
        <https://reviews.apache.org/r/883/#comment1800>

        Is it possible to use org.apache.hadoop.hive.metastore.api.EventType instead of int?

        Another approach is to create an MPartitionEvent baseclass, and then subclass that with MPartitionLoadDoneEvent, etc, and use eventType as the internal type discriminator for JDO.

        trunk/metastore/src/model/package.jdo
        <https://reviews.apache.org/r/883/#comment1793>

        You need to supply schema upgrade scripts for Derby and MySQL. Please either do that in this ticket or open a followup ticket and assign it to yourself.

        trunk/metastore/src/model/package.jdo
        <https://reviews.apache.org/r/883/#comment1799>

        It looks like it's possible for this table to hold more than one "MarkPartitionDone" event for the same partition, but is that a legal state? If it is, how do you know when the load operation for a partition is still in progress?

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionSet.java
        <https://reviews.apache.org/r/883/#comment1801>

        Can you subclass this with a remote and embedded version?

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java
        <https://reviews.apache.org/r/883/#comment1797>

        Any reason in particular why you switched to always running this test in local mode? If we can only test one scenario, then I think there's more value in focusing on the standalone client/server setup.

        • Carl

        On 2011-06-10 21:24:13, Ashutosh Chauhan wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/883/

        -----------------------------------------------------------

        (Updated 2011-06-10 21:24:13)

        Review request for hive and John Sichi.

        Summary

        -------

        Follow-up for HIVE-2147.

        This addresses bug HIVE-2215.

        https://issues.apache.org/jira/browse/HIVE-2215

        Diffs

        -----

        trunk/metastore/if/hive_metastore.thrift 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1134443

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/MarkPartitionEvent.java PRE-CREATION

        trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java PRE-CREATION

        trunk/metastore/src/model/package.jdo 1134443

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1134443

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionSet.java PRE-CREATION

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1134443

        Diff: https://reviews.apache.org/r/883/diff

        Testing

        -------

        Added test cases for new api.

        Thanks,

        Ashutosh

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/883/#review824 ----------------------------------------------------------- trunk/metastore/if/hive_metastore.thrift < https://reviews.apache.org/r/883/#comment1784 > I think this should be changed to "PartitionEventType" in order to make it clear that this applies to partitions only. If in the future we need to introduce event types for tables, indexes, etc, then we should add new enums for those event types as well. trunk/metastore/if/hive_metastore.thrift < https://reviews.apache.org/r/883/#comment1785 > This should also throw UnknownDBException and UnknownTableException. The same goes for isPartitionMarkedForEvent. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java < https://reviews.apache.org/r/883/#comment1786 > Collections aren't required to satisfy an ordering property, so we have to assume the output of this logging statement is ambiguous, e.g. " [a, b] " versus " [b, a] ". We should disambiguate this by passing in the part_vals map and logging the key/value pairs instead of just the values. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java < https://reviews.apache.org/r/883/#comment1789 > Missing exceptions: UnknownDbException and UnknownTableException. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java < https://reviews.apache.org/r/883/#comment1787 > Checking to see if the DB and Table exist should be done in the same database transaction as the rest of the operation. If you do it here there's no guarantee that the db/table will still exist when ms.markPartitionForEvent() is called. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java < https://reviews.apache.org/r/883/#comment1791 > Should we add an InvalidPartitionException and UnknownPartitionException? Seems like those are both valid exceptions in this situation. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java < https://reviews.apache.org/r/883/#comment1788 > Same issue here as before. These checks need to get pushed into ms.isPartitionMarkedForEvent(). trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java < https://reviews.apache.org/r/883/#comment1798 > I think the name of this method is misleading. You're marking a single partition done, not a set of partitions, right? Also, in this context being "done" means that the load operation on that partition has completed, so it would be good to include "load" in the name of the method and event class, e.g. "LoadPartitionDoneEvent" and "onLoadPartitionDone". trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java < https://reviews.apache.org/r/883/#comment1800 > Is it possible to use org.apache.hadoop.hive.metastore.api.EventType instead of int? Another approach is to create an MPartitionEvent baseclass, and then subclass that with MPartitionLoadDoneEvent, etc, and use eventType as the internal type discriminator for JDO. trunk/metastore/src/model/package.jdo < https://reviews.apache.org/r/883/#comment1793 > You need to supply schema upgrade scripts for Derby and MySQL. Please either do that in this ticket or open a followup ticket and assign it to yourself. trunk/metastore/src/model/package.jdo < https://reviews.apache.org/r/883/#comment1799 > It looks like it's possible for this table to hold more than one "MarkPartitionDone" event for the same partition, but is that a legal state? If it is, how do you know when the load operation for a partition is still in progress? trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionSet.java < https://reviews.apache.org/r/883/#comment1801 > Can you subclass this with a remote and embedded version? trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java < https://reviews.apache.org/r/883/#comment1797 > Any reason in particular why you switched to always running this test in local mode? If we can only test one scenario, then I think there's more value in focusing on the standalone client/server setup. Carl On 2011-06-10 21:24:13, Ashutosh Chauhan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/883/ ----------------------------------------------------------- (Updated 2011-06-10 21:24:13) Review request for hive and John Sichi. Summary ------- Follow-up for HIVE-2147 . This addresses bug HIVE-2215 . https://issues.apache.org/jira/browse/HIVE-2215 Diffs ----- trunk/metastore/if/hive_metastore.thrift 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1134443 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/MarkPartitionEvent.java PRE-CREATION trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java PRE-CREATION trunk/metastore/src/model/package.jdo 1134443 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1134443 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionSet.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1134443 Diff: https://reviews.apache.org/r/883/diff Testing ------- Added test cases for new api. Thanks, Ashutosh
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/883/
        -----------------------------------------------------------

        (Updated 2011-06-14 20:51:53.179968)

        Review request for hive and John Sichi.

        Changes
        -------

        Updated patch with Carl's comments.
        Carl, can you take a look?

        Summary
        -------

        Follow-up for HIVE-2147.

        This addresses bug HIVE-2215.
        https://issues.apache.org/jira/browse/HIVE-2215

        Diffs (updated)


        trunk/metastore/if/hive_metastore.thrift 1135779
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1135779
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1135779
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1135779
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1135779
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1135779
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1135779
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/LoadPartitionDoneEvent.java PRE-CREATION
        trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java PRE-CREATION
        trunk/metastore/src/model/package.jdo 1135779
        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1135779
        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java PRE-CREATION
        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java PRE-CREATION
        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1135779

        Diff: https://reviews.apache.org/r/883/diff

        Testing
        -------

        Added test cases for new api.

        Thanks,

        Ashutosh

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/883/ ----------------------------------------------------------- (Updated 2011-06-14 20:51:53.179968) Review request for hive and John Sichi. Changes ------- Updated patch with Carl's comments. Carl, can you take a look? Summary ------- Follow-up for HIVE-2147 . This addresses bug HIVE-2215 . https://issues.apache.org/jira/browse/HIVE-2215 Diffs (updated) trunk/metastore/if/hive_metastore.thrift 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/LoadPartitionDoneEvent.java PRE-CREATION trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java PRE-CREATION trunk/metastore/src/model/package.jdo 1135779 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1135779 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1135779 Diff: https://reviews.apache.org/r/883/diff Testing ------- Added test cases for new api. Thanks, Ashutosh
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-06-14 01:02:20, Carl Steinbach wrote:

        > trunk/metastore/if/hive_metastore.thrift, line 46

        > <https://reviews.apache.org/r/883/diff/1/?file=20969#file20969line46>

        >

        > I think this should be changed to "PartitionEventType" in order to make it clear that this applies to partitions only. If in the future we need to introduce event types for tables, indexes, etc, then we should add new enums for those event types as well.

        Done.

        On 2011-06-14 01:02:20, Carl Steinbach wrote:

        > trunk/metastore/if/hive_metastore.thrift, line 338

        > <https://reviews.apache.org/r/883/diff/1/?file=20969#file20969line338>

        >

        > This should also throw UnknownDBException and UnknownTableException. The same goes for isPartitionMarkedForEvent.

        Done.

        On 2011-06-14 01:02:20, Carl Steinbach wrote:

        > trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 528

        > <https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line528>

        >

        > Collections aren't required to satisfy an ordering property, so we have to assume the output of this logging statement is ambiguous, e.g. "[a, b]" versus "[b, a]". We should disambiguate this by passing in the part_vals map and logging the key/value pairs instead of just the values.

        Done.

        On 2011-06-14 01:02:20, Carl Steinbach wrote:

        > trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3182

        > <https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line3182>

        >

        > Missing exceptions: UnknownDbException and UnknownTableException.

        Done.

        On 2011-06-14 01:02:20, Carl Steinbach wrote:

        > trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3186

        > <https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line3186>

        >

        > Checking to see if the DB and Table exist should be done in the same database transaction as the rest of the operation. If you do it here there's no guarantee that the db/table will still exist when ms.markPartitionForEvent() is called.

        Done.

        On 2011-06-14 01:02:20, Carl Steinbach wrote:

        > trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3188

        > <https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line3188>

        >

        > Should we add an InvalidPartitionException and UnknownPartitionException? Seems like those are both valid exceptions in this situation.

        Done.

        On 2011-06-14 01:02:20, Carl Steinbach wrote:

        > trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3224

        > <https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line3224>

        >

        > Same issue here as before. These checks need to get pushed into ms.isPartitionMarkedForEvent().

        Done.

        On 2011-06-14 01:02:20, Carl Steinbach wrote:

        > trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java, line 82

        > <https://reviews.apache.org/r/883/diff/1/?file=20973#file20973line82>

        >

        > I think the name of this method is misleading. You're marking a single partition done, not a set of partitions, right?

        >

        > Also, in this context being "done" means that the load operation on that partition has completed, so it would be good to include "load" in the name of the method and event class, e.g. "LoadPartitionDoneEvent" and "onLoadPartitionDone".

        >

        Done.

        On 2011-06-14 01:02:20, Carl Steinbach wrote:

        > trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java, line 34

        > <https://reviews.apache.org/r/883/diff/1/?file=20977#file20977line34>

        >

        > Is it possible to use org.apache.hadoop.hive.metastore.api.EventType instead of int?

        >

        > Another approach is to create an MPartitionEvent baseclass, and then subclass that with MPartitionLoadDoneEvent, etc, and use eventType as the internal type discriminator for JDO.

        No.

        I don't see any benefit of it. Nor I cant see how will this work.

        On 2011-06-14 01:02:20, Carl Steinbach wrote:

        > trunk/metastore/src/model/package.jdo, line 668

        > <https://reviews.apache.org/r/883/diff/1/?file=20978#file20978line668>

        >

        > You need to supply schema upgrade scripts for Derby and MySQL. Please either do that in this ticket or open a followup ticket and assign it to yourself.

        Will do.

        On 2011-06-14 01:02:20, Carl Steinbach wrote:

        > trunk/metastore/src/model/package.jdo, line 683

        > <https://reviews.apache.org/r/883/diff/1/?file=20978#file20978line683>

        >

        > It looks like it's possible for this table to hold more than one "MarkPartitionDone" event for the same partition, but is that a legal state? If it is, how do you know when the load operation for a partition is still in progress?

        This is not for when load operation is in progress. As suggested from name its "LoadPartitionDone". So, marking partition load done is idempotent. Client can mark it multiple times. So, metastore will return true if it finds one or more such partitioned marked in the table.

        On 2011-06-14 01:02:20, Carl Steinbach wrote:

        > trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionSet.java, line 36

        > <https://reviews.apache.org/r/883/diff/1/?file=20980#file20980line36>

        >

        > Can you subclass this with a remote and embedded version?

        Done.

        On 2011-06-14 01:02:20, Carl Steinbach wrote:

        > trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java, line 80

        > <https://reviews.apache.org/r/883/diff/1/?file=20981#file20981line80>

        >

        > Any reason in particular why you switched to always running this test in local mode? If we can only test one scenario, then I think there's more value in focusing on the standalone client/server setup.

        I reverted those changes.

        • Ashutosh

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/883/#review824
        -----------------------------------------------------------

        On 2011-06-14 20:51:53, Ashutosh Chauhan wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/883/

        -----------------------------------------------------------

        (Updated 2011-06-14 20:51:53)

        Review request for hive and John Sichi.

        Summary

        -------

        Follow-up for HIVE-2147.

        This addresses bug HIVE-2215.

        https://issues.apache.org/jira/browse/HIVE-2215

        Diffs

        -----

        trunk/metastore/if/hive_metastore.thrift 1135779

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1135779

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1135779

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1135779

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1135779

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1135779

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1135779

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/LoadPartitionDoneEvent.java PRE-CREATION

        trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java PRE-CREATION

        trunk/metastore/src/model/package.jdo 1135779

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1135779

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java PRE-CREATION

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java PRE-CREATION

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1135779

        Diff: https://reviews.apache.org/r/883/diff

        Testing

        -------

        Added test cases for new api.

        Thanks,

        Ashutosh

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-06-14 01:02:20, Carl Steinbach wrote: > trunk/metastore/if/hive_metastore.thrift, line 46 > < https://reviews.apache.org/r/883/diff/1/?file=20969#file20969line46 > > > I think this should be changed to "PartitionEventType" in order to make it clear that this applies to partitions only. If in the future we need to introduce event types for tables, indexes, etc, then we should add new enums for those event types as well. Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: > trunk/metastore/if/hive_metastore.thrift, line 338 > < https://reviews.apache.org/r/883/diff/1/?file=20969#file20969line338 > > > This should also throw UnknownDBException and UnknownTableException. The same goes for isPartitionMarkedForEvent. Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: > trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 528 > < https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line528 > > > Collections aren't required to satisfy an ordering property, so we have to assume the output of this logging statement is ambiguous, e.g. " [a, b] " versus " [b, a] ". We should disambiguate this by passing in the part_vals map and logging the key/value pairs instead of just the values. Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: > trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3182 > < https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line3182 > > > Missing exceptions: UnknownDbException and UnknownTableException. Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: > trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3186 > < https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line3186 > > > Checking to see if the DB and Table exist should be done in the same database transaction as the rest of the operation. If you do it here there's no guarantee that the db/table will still exist when ms.markPartitionForEvent() is called. Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: > trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3188 > < https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line3188 > > > Should we add an InvalidPartitionException and UnknownPartitionException? Seems like those are both valid exceptions in this situation. Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: > trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3224 > < https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line3224 > > > Same issue here as before. These checks need to get pushed into ms.isPartitionMarkedForEvent(). Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: > trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java, line 82 > < https://reviews.apache.org/r/883/diff/1/?file=20973#file20973line82 > > > I think the name of this method is misleading. You're marking a single partition done, not a set of partitions, right? > > Also, in this context being "done" means that the load operation on that partition has completed, so it would be good to include "load" in the name of the method and event class, e.g. "LoadPartitionDoneEvent" and "onLoadPartitionDone". > Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: > trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java, line 34 > < https://reviews.apache.org/r/883/diff/1/?file=20977#file20977line34 > > > Is it possible to use org.apache.hadoop.hive.metastore.api.EventType instead of int? > > Another approach is to create an MPartitionEvent baseclass, and then subclass that with MPartitionLoadDoneEvent, etc, and use eventType as the internal type discriminator for JDO. No. I don't see any benefit of it. Nor I cant see how will this work. On 2011-06-14 01:02:20, Carl Steinbach wrote: > trunk/metastore/src/model/package.jdo, line 668 > < https://reviews.apache.org/r/883/diff/1/?file=20978#file20978line668 > > > You need to supply schema upgrade scripts for Derby and MySQL. Please either do that in this ticket or open a followup ticket and assign it to yourself. Will do. On 2011-06-14 01:02:20, Carl Steinbach wrote: > trunk/metastore/src/model/package.jdo, line 683 > < https://reviews.apache.org/r/883/diff/1/?file=20978#file20978line683 > > > It looks like it's possible for this table to hold more than one "MarkPartitionDone" event for the same partition, but is that a legal state? If it is, how do you know when the load operation for a partition is still in progress? This is not for when load operation is in progress. As suggested from name its "LoadPartitionDone". So, marking partition load done is idempotent. Client can mark it multiple times. So, metastore will return true if it finds one or more such partitioned marked in the table. On 2011-06-14 01:02:20, Carl Steinbach wrote: > trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionSet.java, line 36 > < https://reviews.apache.org/r/883/diff/1/?file=20980#file20980line36 > > > Can you subclass this with a remote and embedded version? Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: > trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java, line 80 > < https://reviews.apache.org/r/883/diff/1/?file=20981#file20981line80 > > > Any reason in particular why you switched to always running this test in local mode? If we can only test one scenario, then I think there's more value in focusing on the standalone client/server setup. I reverted those changes. Ashutosh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/883/#review824 ----------------------------------------------------------- On 2011-06-14 20:51:53, Ashutosh Chauhan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/883/ ----------------------------------------------------------- (Updated 2011-06-14 20:51:53) Review request for hive and John Sichi. Summary ------- Follow-up for HIVE-2147 . This addresses bug HIVE-2215 . https://issues.apache.org/jira/browse/HIVE-2215 Diffs ----- trunk/metastore/if/hive_metastore.thrift 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/LoadPartitionDoneEvent.java PRE-CREATION trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java PRE-CREATION trunk/metastore/src/model/package.jdo 1135779 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1135779 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1135779 Diff: https://reviews.apache.org/r/883/diff Testing ------- Added test cases for new api. Thanks, Ashutosh
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/883/#review843
        -----------------------------------------------------------

        Ship it!

        Looks good to me. Thanks for making the changes.

        • Carl

        On 2011-06-14 20:51:53, Ashutosh Chauhan wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/883/

        -----------------------------------------------------------

        (Updated 2011-06-14 20:51:53)

        Review request for hive and John Sichi.

        Summary

        -------

        Follow-up for HIVE-2147.

        This addresses bug HIVE-2215.

        https://issues.apache.org/jira/browse/HIVE-2215

        Diffs

        -----

        trunk/metastore/if/hive_metastore.thrift 1135779

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1135779

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1135779

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1135779

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1135779

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1135779

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1135779

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/LoadPartitionDoneEvent.java PRE-CREATION

        trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java PRE-CREATION

        trunk/metastore/src/model/package.jdo 1135779

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1135779

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java PRE-CREATION

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java PRE-CREATION

        trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1135779

        Diff: https://reviews.apache.org/r/883/diff

        Testing

        -------

        Added test cases for new api.

        Thanks,

        Ashutosh

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/883/#review843 ----------------------------------------------------------- Ship it! Looks good to me. Thanks for making the changes. Carl On 2011-06-14 20:51:53, Ashutosh Chauhan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/883/ ----------------------------------------------------------- (Updated 2011-06-14 20:51:53) Review request for hive and John Sichi. Summary ------- Follow-up for HIVE-2147 . This addresses bug HIVE-2215 . https://issues.apache.org/jira/browse/HIVE-2215 Diffs ----- trunk/metastore/if/hive_metastore.thrift 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/LoadPartitionDoneEvent.java PRE-CREATION trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java PRE-CREATION trunk/metastore/src/model/package.jdo 1135779 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1135779 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1135779 Diff: https://reviews.apache.org/r/883/diff Testing ------- Added test cases for new api. Thanks, Ashutosh
        Hide
        Ashutosh Chauhan added a comment -

        Thanks, Carl for quick reply. Here is the full patch with the generated code. Can someone please commit this?

        Show
        Ashutosh Chauhan added a comment - Thanks, Carl for quick reply. Here is the full patch with the generated code. Can someone please commit this?
        Hide
        Ashutosh Chauhan added a comment -

        Anyone care to commit this one?

        Show
        Ashutosh Chauhan added a comment - Anyone care to commit this one?
        Hide
        Carl Steinbach added a comment -

        +1. Will commit when tests pass.

        Show
        Carl Steinbach added a comment - +1. Will commit when tests pass.
        Hide
        Carl Steinbach added a comment -

        Committed to trunk. Thanks Ashutosh!

        Show
        Carl Steinbach added a comment - Committed to trunk. Thanks Ashutosh!
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #790 (See https://builds.apache.org/job/Hive-trunk-h0.21/790/)

        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #790 (See https://builds.apache.org/job/Hive-trunk-h0.21/790/ )
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #951 (See https://builds.apache.org/job/Hive-trunk-h0.21/951/)
        HIVE-2441. Metastore upgrade scripts for schema change introduced in HIVE-2215 (Ashutosh Chauhan via cws)

        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170344
        Files :

        • /hive/trunk/metastore/scripts/upgrade/derby/009-HIVE-2215.derby.sql
        • /hive/trunk/metastore/scripts/upgrade/mysql/009-HIVE-2215.mysql.sql
        • /hive/trunk/metastore/scripts/upgrade/postgres/009-HIVE-2215.postgres.sql
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #951 (See https://builds.apache.org/job/Hive-trunk-h0.21/951/ ) HIVE-2441 . Metastore upgrade scripts for schema change introduced in HIVE-2215 (Ashutosh Chauhan via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170344 Files : /hive/trunk/metastore/scripts/upgrade/derby/009- HIVE-2215 .derby.sql /hive/trunk/metastore/scripts/upgrade/mysql/009- HIVE-2215 .mysql.sql /hive/trunk/metastore/scripts/upgrade/postgres/009- HIVE-2215 .postgres.sql
        Hide
        Hudson added a comment -

        Integrated in Hive-0.8.0-SNAPSHOT-h0.21 #20 (See https://builds.apache.org/job/Hive-0.8.0-SNAPSHOT-h0.21/20/)
        HIVE-2441. Metastore upgrade scripts for schema change introduced in HIVE-2215 (Ashutosh Chauhan via cws)

        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170346
        Files :

        • /hive/branches/branch-0.8/metastore/scripts/upgrade/derby/009-HIVE-2215.derby.sql
        • /hive/branches/branch-0.8/metastore/scripts/upgrade/mysql/009-HIVE-2215.mysql.sql
        • /hive/branches/branch-0.8/metastore/scripts/upgrade/postgres/009-HIVE-2215.postgres.sql
        Show
        Hudson added a comment - Integrated in Hive-0.8.0-SNAPSHOT-h0.21 #20 (See https://builds.apache.org/job/Hive-0.8.0-SNAPSHOT-h0.21/20/ ) HIVE-2441 . Metastore upgrade scripts for schema change introduced in HIVE-2215 (Ashutosh Chauhan via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170346 Files : /hive/branches/branch-0.8/metastore/scripts/upgrade/derby/009- HIVE-2215 .derby.sql /hive/branches/branch-0.8/metastore/scripts/upgrade/mysql/009- HIVE-2215 .mysql.sql /hive/branches/branch-0.8/metastore/scripts/upgrade/postgres/009- HIVE-2215 .postgres.sql

          People

          • Assignee:
            Ashutosh Chauhan
            Reporter:
            Ashutosh Chauhan
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development