Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
HideThis patch removed support for the 'hive.metastore.local' configuration property. Going forward Hive will assume that the metastore is operating in local mode if the configuration property 'hive.metastore.uris' is unset, and will assume remote mode otherwise.ShowThis patch removed support for the 'hive.metastore.local' configuration property. Going forward Hive will assume that the metastore is operating in local mode if the configuration property 'hive.metastore.uris' is unset, and will assume remote mode otherwise.
Description
We should just have hive.metastore.uris. If it is empty, we shall assume local mode, if non-empty we shall use that string to connect to remote metastore. Having two different keys for same information is confusing.
Attachments
Attachments
- ASF.LICENSE.NOT.GRANTED--HIVE-2585.D2559.1.patch
- 7 kB
- Phabricator
- ASF.LICENSE.NOT.GRANTED--HIVE-2585.D2559.2.patch
- 8 kB
- Phabricator
- hive-2585_3.patch
- 8 kB
- Ashutosh Chauhan
Issue Links
- relates to
-
HIVE-3560 Hive always prints a warning message when using remote metastore
- Closed
Activity
BUILD SUCCESSFUL Total time: 339 minutes 51 seconds
All the tests passed with this patch.
cwsteinbach has requested changes to the revision "HIVE-2585 [jira] Collapse hive.metastore.uris and hive.metastore.local".
INLINE COMMENTS
conf/hive-default.xml.template:102 We should throw an error at startup if both hive.metastore.uris and javax.jdo.option.ConnectionURL are set. Giving silent preference to one of these properties is going to cause a lot of confusion for people who are using hive.metastore.local to quickly swap between local and remote metastores.
data/conf/hive-site.xml:71 It would be nice to add a deprecation warning at startup if this is set.
REVISION DETAIL
https://reviews.facebook.net/D2559
BRANCH
svn
ashutoshc has commented on the revision "HIVE-2585 [jira] Collapse hive.metastore.uris and hive.metastore.local".
INLINE COMMENTS
conf/hive-default.xml.template:102 Good idea, but not sure where should this check be made? In metastore client, metastore::main(), HMSHandler init, HiveConf::init() ?
data/conf/hive-site.xml:71 Same as above. not sure where should this check be made? In metastore client, metastore::main(), HMSHandler init, HiveConf::init() ?
REVISION DETAIL
https://reviews.facebook.net/D2559
BRANCH
svn
cwsteinbach has commented on the revision "HIVE-2585 [jira] Collapse hive.metastore.uris and hive.metastore.local".
INLINE COMMENTS
conf/hive-default.xml.template:102 HiveConf.init() seems like the right place to make these checks.
REVISION DETAIL
https://reviews.facebook.net/D2559
BRANCH
svn
thw has commented on the revision "HIVE-2585 [jira] Collapse hive.metastore.uris and hive.metastore.local".
INLINE COMMENTS
conf/hive-default.xml.template:102 A scenario where this would lead to change of existing behavior is when client and metastore server point to the same conf, with client using the thrift URI and server javax.jdo.option.ConnectionURL (should only happen in dev environments).
REVISION DETAIL
https://reviews.facebook.net/D2559
BRANCH
svn
ashutoshc has commented on the revision "HIVE-2585 [jira] Collapse hive.metastore.uris and hive.metastore.local".
INLINE COMMENTS
conf/hive-default.xml.template:102 Valid point. Then, instead of throwing exception, we can log a message then.
REVISION DETAIL
https://reviews.facebook.net/D2559
BRANCH
svn
ashutoshc updated the revision "HIVE-2585 [jira] Collapse hive.metastore.uris and hive.metastore.local".
Reviewers: JIRA, cwsteinbach
Incorporated Thomas and Carl's comments.
REVISION DETAIL
https://reviews.facebook.net/D2559
AFFECTED FILES
shims/src/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java
conf/hive-default.xml.template
metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java
metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java
metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreAuthorization.java
metastore/src/test/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStoreIpAddress.java
metastore/src/test/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStore.java
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
data/conf/hive-site.xml
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Bump. Got hit by this today again. Carl, do you have any further suggestions. Else, I think its ready to go in?
+1
On changes can you do one more rebase and will commit.
[edward@tablitha trunk]$ patch -p0 --dry-run < HIVE-2585.D2559.2.patch patching file metastore/src/test/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStoreIpAddress.java Hunk #1 FAILED at 91. 1 out of 1 hunk FAILED -- saving rejects to file metastore/src/test/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStoreIpAddress.java.rej
Integrated in Hive-trunk-h0.21 #1501 (See https://builds.apache.org/job/Hive-trunk-h0.21/1501/)
HIVE-2585 Collapse hive.metastore.uris and hive.metastore.local (Ashutosh Chauhan via egc) (Revision 1352333)
Result = SUCCESS
ecapriolo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352333
Files :
- /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
- /hive/trunk/conf/hive-default.xml.template
- /hive/trunk/data/conf/hive-site.xml
- /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
- /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java
- /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreAuthorization.java
- /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java
- /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStore.java
- /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStoreIpAddress.java
- /hive/trunk/shims/src/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java
I started seeing this error message in my logs, and when investigating believe there's an issue here. AFAICT, if you set hive.metastore.uris you will ALWAYS see this error.
The reason is javax.jdo.option.ConnectionURL has a default value and will never be null. I set this property in hive-site.xml and walked through the configuration loading in a debugger. If the value is not empty it takes effect, and is ignored if empty.
Since javax.jdo.option.ConnectionURL has a default and cannot be unset, this warning will always be printed if someone sets hive.metastore.uris.
Before diving into possible solutions, Ashutosh Chauhan & Edward Capriolo can you confirm this is an issue, or clarify how to correctly configure this? If this is a user error I can post a patch clarifying the error message so others that see this message will know what to do.
<property> <name>javax.jdo.option.ConnectionURL</name> <value></value> </property>
+ if (null != this.get(ConfVars.METASTOREURIS.varname, null) && + null != this.get(ConfVars.METASTORECONNECTURLKEY.varname, null)) { + l4j.error("Found both " + ConfVars.METASTOREURIS.varname + " and " + + ConfVars.METASTORECONNECTURLKEY + " Recommended to have exactly one of those config key" + + "in configuration"); + }
Is it just that you are seeing a log message, which you should not have been seeing or is it affecting some functionality as well.
Hive clients using a remote metastore via hive.metastore.uris works correctly, however, this error message is always printed because AFAICT its not possible null out javax.jdo.option.ConnectionURL from hive-site.xml.
Looking through the comments on this patch, the error message was intended to reduce user confusion, a goal I totally support. In HiveMetaStoreClient.java we see a very clear info message printed saying that a remote metastore is used.
LOG.info("Trying to connect to metastore with URI " + store); ... LOG.info("Connected to metastore.");
Since we clearly communicate to the user that a remote metastore at the given URI is being used, is this warning needed? I'd be happy to submit a patch that removes the above warning message, and makes the following HiveMetaStoreClient logging change:
LOG.debug("Trying to connect to remote HiveMetaStore: " + store); ... LOG.info("Connected to remote HiveMetaStore: " + store);
The change is at debug level we print connection attempts, and always print which remote HiveMetaStore we actually connected to.
Thoughts? If this sounds good I'll file a related issue and submit the patch.
Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
HIVE-2585 Collapse hive.metastore.uris and hive.metastore.local (Ashutosh Chauhan via egc) (Revision 1352333)
Result = ABORTED
ecapriolo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352333
Files :
- /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
- /hive/trunk/conf/hive-default.xml.template
- /hive/trunk/data/conf/hive-site.xml
- /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
- /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java
- /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreAuthorization.java
- /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java
- /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStore.java
- /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStoreIpAddress.java
- /hive/trunk/shims/src/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java
This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
ashutoshc requested code review of "
HIVE-2585[jira] Collapse hive.metastore.uris and hive.metastore.local".Reviewers: JIRA
https://issues.apache.org/jira/browse/HIVE-2585
Get rid of hive.metastore.local. By default, it was assumed to be true as a result hive will try to connect to local metastore. That behavior is preserved since hive.metastore.uris is empty by default.
We should just have hive.metastore.uris. If it is empty, we shall assume local mode, if non-empty we shall use that string to connect to remote metastore. Having two different keys for same information is confusing.
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D2559
AFFECTED FILES
shims/src/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java
conf/hive-default.xml.template
metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java
metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java
metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreAuthorization.java
metastore/src/test/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStoreIpAddress.java
metastore/src/test/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStore.java
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
data/conf/hive-site.xml
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/5811/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.