Integrated in Hive-trunk-h0.21 #801 (See https://builds.apache.org/job/Hive-trunk-h0.21/801/)
HIVE-2225. Purge expired metastore events (Ashutosh Chauhan via cws)

cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1141430
Files :

• /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
• /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/EventCleanerTask.java
• /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
• /hive/trunk/conf/hive-default.xml
• /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java
• /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
• /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java

Committed to trunk. Thanks Ashutosh!

Sorry.. RB-Jira sync. Had the patch on RB. Reattaching it here.

@Ashutosh: can you please attach the most recent version of the patch? Thanks.

+1. Will commit if tests pass.

Looks good to me.

Looking at it now.

Carl, any comments on this one?

On 2011-06-22 23:12:49, John Sichi wrote:

> trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 222

> <https://reviews.apache.org/r/940/diff/1/?file=21411#file21411line222>

>

> Existing property names are "hive.metastore..." instead of just "metastore..."

done

• Ashutosh

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/940/#review890
-----------------------------------------------------------

On 2011-06-23 02:55:08, Ashutosh Chauhan wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/940/

-----------------------------------------------------------

(Updated 2011-06-23 02:55:08)

Review request for hive, Carl Steinbach and John Sichi.

Summary

-------

This addresses HIVE-2225

This addresses bug HIVE-2225.

https://issues.apache.org/jira/browse/HIVE-2225

Diffs

-----

trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1138719

trunk/conf/hive-default.xml 1138719

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1138719

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1138719

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1138719

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/EventCleanerTask.java PRE-CREATION

trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java 1138719

Diff: https://reviews.apache.org/r/940/diff

Testing

-------

updated a test case which exercises this code path.

Thanks,

Ashutosh

On 2011-06-22 23:07:46, John Sichi wrote:

> trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 221

> <https://reviews.apache.org/r/940/diff/1/?file=21411#file21411line221>

>

> If you agree about making this disabled by default, we could use a special value such as 0 for the frequency to indicate disabled.

>

Done. Timer is now created only if this property has non-zero value.

• Ashutosh

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/940/#review889
-----------------------------------------------------------

On 2011-06-23 02:55:08, Ashutosh Chauhan wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/940/

-----------------------------------------------------------

(Updated 2011-06-23 02:55:08)

Review request for hive, Carl Steinbach and John Sichi.

Summary

-------

This addresses HIVE-2225

This addresses bug HIVE-2225.

https://issues.apache.org/jira/browse/HIVE-2225

Diffs

-----

trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1138719

trunk/conf/hive-default.xml 1138719

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1138719

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1138719

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1138719

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/EventCleanerTask.java PRE-CREATION

trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java 1138719

Diff: https://reviews.apache.org/r/940/diff

Testing

-------

updated a test case which exercises this code path.

Thanks,

Ashutosh

On 2011-06-22 23:18:43, John Sichi wrote:

> trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 259

> <https://reviews.apache.org/r/940/diff/1/?file=21412#file21412line259>

>

> Why is this using a Thread instead of a Timer?

Agreed timer is better suited here then Thread. Changed to timer.

On 2011-06-22 23:18:43, John Sichi wrote:

> trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/EventCleanerThread.java, line 33

> <https://reviews.apache.org/r/940/diff/1/?file=21415#file21415line33>

>

> 6 hrs is actually configurable, right?

Yup, it is.

• Ashutosh

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/940/#review891
-----------------------------------------------------------

On 2011-06-23 02:55:08, Ashutosh Chauhan wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/940/

-----------------------------------------------------------

(Updated 2011-06-23 02:55:08)

Review request for hive, Carl Steinbach and John Sichi.

Summary

-------

This addresses HIVE-2225

This addresses bug HIVE-2225.

https://issues.apache.org/jira/browse/HIVE-2225

Diffs

-----

trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1138719

trunk/conf/hive-default.xml 1138719

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1138719

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1138719

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1138719

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/EventCleanerTask.java PRE-CREATION

trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java 1138719

Diff: https://reviews.apache.org/r/940/diff

Testing

-------

updated a test case which exercises this code path.

Thanks,

Ashutosh

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/940/
-----------------------------------------------------------

(Updated 2011-06-23 02:55:08.540561)

Review request for hive, Carl Steinbach and John Sichi.

Changes
-------

Updated the patch per John's comments.

Summary
-------

This addresses HIVE-2225

This addresses bug HIVE-2225.
https://issues.apache.org/jira/browse/HIVE-2225

Diffs (updated)

trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1138719
trunk/conf/hive-default.xml 1138719
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1138719
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1138719
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1138719
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/EventCleanerTask.java PRE-CREATION
trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java 1138719

Diff: https://reviews.apache.org/r/940/diff

Testing
-------

updated a test case which exercises this code path.

Thanks,

Ashutosh

On 2011-06-22 23:07:05, John Sichi wrote:

> trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/EventCleanerThread.java, line 1

> <https://reviews.apache.org/r/940/diff/1/?file=21415#file21415line1>

>

> New files need Apache headers

Added.

• Ashutosh

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/940/#review888
-----------------------------------------------------------

On 2011-06-21 17:34:28, Ashutosh Chauhan wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/940/

-----------------------------------------------------------

(Updated 2011-06-21 17:34:28)

Review request for hive, Carl Steinbach and John Sichi.

Summary

-------

This addresses HIVE-2225

This addresses bug HIVE-2225.

https://issues.apache.org/jira/browse/HIVE-2225

Diffs

-----

trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/EventCleanerThread.java PRE-CREATION

trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java 1138099

Diff: https://reviews.apache.org/r/940/diff

Testing

-------

updated a test case which exercises this code path.

Thanks,

Ashutosh

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/940/#review891
-----------------------------------------------------------

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
<https://reviews.apache.org/r/940/#comment1932>

Why is this using a Thread instead of a Timer?

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/EventCleanerThread.java
<https://reviews.apache.org/r/940/#comment1931>

6 hrs is actually configurable, right?

• John

On 2011-06-21 17:34:28, Ashutosh Chauhan wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/940/

-----------------------------------------------------------

(Updated 2011-06-21 17:34:28)

Review request for hive, Carl Steinbach and John Sichi.

Summary

-------

This addresses HIVE-2225

This addresses bug HIVE-2225.

https://issues.apache.org/jira/browse/HIVE-2225

Diffs

-----

trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/EventCleanerThread.java PRE-CREATION

trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java 1138099

Diff: https://reviews.apache.org/r/940/diff

Testing

-------

updated a test case which exercises this code path.

Thanks,

Ashutosh

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/940/#review890
-----------------------------------------------------------

trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
<https://reviews.apache.org/r/940/#comment1930>

Existing property names are "hive.metastore..." instead of just "metastore..."

• John

On 2011-06-21 17:34:28, Ashutosh Chauhan wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/940/

-----------------------------------------------------------

(Updated 2011-06-21 17:34:28)

Review request for hive, Carl Steinbach and John Sichi.

Summary

-------

This addresses HIVE-2225

This addresses bug HIVE-2225.

https://issues.apache.org/jira/browse/HIVE-2225

Diffs

-----

trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/EventCleanerThread.java PRE-CREATION

trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java 1138099

Diff: https://reviews.apache.org/r/940/diff

Testing

-------

updated a test case which exercises this code path.

Thanks,

Ashutosh

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/940/#review888
-----------------------------------------------------------

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/EventCleanerThread.java
<https://reviews.apache.org/r/940/#comment1928>

New files need Apache headers

• John

On 2011-06-21 17:34:28, Ashutosh Chauhan wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/940/

-----------------------------------------------------------

(Updated 2011-06-21 17:34:28)

Review request for hive, Carl Steinbach and John Sichi.

Summary

-------

This addresses HIVE-2225

This addresses bug HIVE-2225.

https://issues.apache.org/jira/browse/HIVE-2225

Diffs

-----

trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/EventCleanerThread.java PRE-CREATION

trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java 1138099

Diff: https://reviews.apache.org/r/940/diff

Testing

-------

updated a test case which exercises this code path.

Thanks,

Ashutosh

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/940/#review889
-----------------------------------------------------------

trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
<https://reviews.apache.org/r/940/#comment1929>

If you agree about making this disabled by default, we could use a special value such as 0 for the frequency to indicate disabled.

• John

On 2011-06-21 17:34:28, Ashutosh Chauhan wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/940/

-----------------------------------------------------------

(Updated 2011-06-21 17:34:28)

Review request for hive, Carl Steinbach and John Sichi.

Summary

-------

This addresses HIVE-2225

This addresses bug HIVE-2225.

https://issues.apache.org/jira/browse/HIVE-2225

Diffs

-----

trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1138099

trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/EventCleanerThread.java PRE-CREATION

trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java 1138099

Diff: https://reviews.apache.org/r/940/diff

Testing

-------

updated a test case which exercises this code path.

Thanks,

Ashutosh

Let's make it possible to disable the thread altogether, and make that the default for out-of-the-box Hive.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/940/
-----------------------------------------------------------

Review request for hive, Carl Steinbach and John Sichi.

Summary
-------

This addresses HIVE-2225

This addresses bug HIVE-2225.
https://issues.apache.org/jira/browse/HIVE-2225

Diffs

trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1138099
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1138099
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1138099
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1138099
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/EventCleanerThread.java PRE-CREATION
trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java 1138099

Diff: https://reviews.apache.org/r/940/diff

Testing
-------

updated a test case which exercises this code path.

Thanks,

Ashutosh

This patch is on top of HIVE-2215. It wakes up after every 6 hours (configurable) and deletes all the events which are older then 7 days(configurable).