Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-3056

Create a new metastore tool to bulk update location field in Db/Table/Partition records

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Metastore
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      This patch added bin/metatool
    1. HIVE-3056.2.patch.txt
      32 kB
      Shreepadma Venugopalan
    2. HIVE-3056.3.patch.txt
      38 kB
      Shreepadma Venugopalan
    3. HIVE-3056.4.patch.txt
      26 kB
      Shreepadma Venugopalan
    4. HIVE-3056.5.patch.txt
      32 kB
      Shreepadma Venugopalan
    5. HIVE-3056.7.patch.txt
      45 kB
      Shreepadma Venugopalan
    6. HIVE-3056.patch
      24 kB
      Shreepadma Venugopalan

      Issue Links

        Activity

        Hide
        cwsteinbach Carl Steinbach added a comment -

        Each Db, Table, and Partition record in the MetaStore includes a location field. We need to develop a mechanism that allows us to do bulk updates on this field. The motivating use case is handling the transition from pre-HA to HA HDFS.

        Show
        cwsteinbach Carl Steinbach added a comment - Each Db, Table, and Partition record in the MetaStore includes a location field. We need to develop a mechanism that allows us to do bulk updates on this field. The motivating use case is handling the transition from pre-HA to HA HDFS.
        Hide
        namit Namit Jain added a comment -

        @Shreepadma, can you add a phabricator/reviewBoard review ?

        Show
        namit Namit Jain added a comment - @Shreepadma, can you add a phabricator/reviewBoard review ?
        Hide
        shreepadma Shreepadma Venugopalan added a comment -

        @Namit: I've also posted the review on Apache reviewboard.

        Show
        shreepadma Shreepadma Venugopalan added a comment - @Namit: I've also posted the review on Apache reviewboard.
        Hide
        cwsteinbach Carl Steinbach added a comment -

        @Shreepadma: Can you please provide a link to the reviewboard request? Thanks!

        Show
        cwsteinbach Carl Steinbach added a comment - @Shreepadma: Can you please provide a link to the reviewboard request? Thanks!
        Hide
        shreepadma Shreepadma Venugopalan added a comment -

        Link to review board review request: https://reviews.apache.org/r/6650/

        Show
        shreepadma Shreepadma Venugopalan added a comment - Link to review board review request: https://reviews.apache.org/r/6650/
        Hide
        namit Namit Jain added a comment -

        comments on review board

        Show
        namit Namit Jain added a comment - comments on review board
        Hide
        shreepadma Shreepadma Venugopalan added a comment -

        @Namit: Responded to your comments on review board.

        Show
        shreepadma Shreepadma Venugopalan added a comment - @Namit: Responded to your comments on review board.
        Hide
        shreepadma Shreepadma Venugopalan added a comment -

        Updated patch is available both on this page as well as https://reviews.apache.org/r/6650/diff/

        Show
        shreepadma Shreepadma Venugopalan added a comment - Updated patch is available both on this page as well as https://reviews.apache.org/r/6650/diff/
        Hide
        cwsteinbach Carl Steinbach added a comment -

        @Shreepadma: Please see my comments on reviewboard. Thanks!

        Show
        cwsteinbach Carl Steinbach added a comment - @Shreepadma: Please see my comments on reviewboard. Thanks!
        Hide
        shreepadma Shreepadma Venugopalan added a comment -

        @Carl: I've addressed your comments in the new patch. Its on review board at : https://reviews.apache.org/r/6650/diff/ Thanks!

        Show
        shreepadma Shreepadma Venugopalan added a comment - @Carl: I've addressed your comments in the new patch. Its on review board at : https://reviews.apache.org/r/6650/diff/ Thanks!
        Hide
        cwsteinbach Carl Steinbach added a comment -

        @Shreepadma: please remember to set the status to "patch available" after resubmitting a patch. Thanks.

        Show
        cwsteinbach Carl Steinbach added a comment - @Shreepadma: please remember to set the status to "patch available" after resubmitting a patch. Thanks.
        Hide
        cwsteinbach Carl Steinbach added a comment -

        @Shreepadma: Please see my comments on reviewboard. Thanks.

        Show
        cwsteinbach Carl Steinbach added a comment - @Shreepadma: Please see my comments on reviewboard. Thanks.
        Hide
        cwsteinbach Carl Steinbach added a comment -

        @Shreepadma: Please update the diff on reviewboard. Thanks.

        Show
        cwsteinbach Carl Steinbach added a comment - @Shreepadma: Please update the diff on reviewboard. Thanks.
        Hide
        cwsteinbach Carl Steinbach added a comment -

        +1. Will commit if tests pass.

        Show
        cwsteinbach Carl Steinbach added a comment - +1. Will commit if tests pass.
        Hide
        cwsteinbach Carl Steinbach added a comment -

        Committed to trunk. Thanks Shreepadma!

        Show
        cwsteinbach Carl Steinbach added a comment - Committed to trunk. Thanks Shreepadma!
        Hide
        hudson Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1646 (See https://builds.apache.org/job/Hive-trunk-h0.21/1646/)
        HIVE-3056. Ability to bulk update location field in Db/Table/Partition records (Shreepadma Venugopalan via cws) (Revision 1380500)

        Result = FAILURE
        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380500
        Files :

        • /hive/trunk/bin/ext/metatool.sh
        • /hive/trunk/bin/metatool
        • /hive/trunk/build.xml
        • /hive/trunk/eclipse-templates/TestHiveMetaTool.launchtemplate
        • /hive/trunk/metastore/ivy.xml
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/tools
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaTool.java
        Show
        hudson Hudson added a comment - Integrated in Hive-trunk-h0.21 #1646 (See https://builds.apache.org/job/Hive-trunk-h0.21/1646/ ) HIVE-3056 . Ability to bulk update location field in Db/Table/Partition records (Shreepadma Venugopalan via cws) (Revision 1380500) Result = FAILURE cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380500 Files : /hive/trunk/bin/ext/metatool.sh /hive/trunk/bin/metatool /hive/trunk/build.xml /hive/trunk/eclipse-templates/TestHiveMetaTool.launchtemplate /hive/trunk/metastore/ivy.xml /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/tools /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaTool.java
        Hide
        shreepadma Shreepadma Venugopalan added a comment -

        Metatool provides the following functionality,

        • Ability to search and replace the HDFS NN location in metastore records that reference the NN. Note that one use is the ability to transition an existing Hive deployment to use HDFS HA NN.
        • A command line tool to execute JDOQL against the metastore. We believe the ability to execute JDOQL against the metastore will be a useful debugging tool for both users and Hive developers.
        Show
        shreepadma Shreepadma Venugopalan added a comment - Metatool provides the following functionality, Ability to search and replace the HDFS NN location in metastore records that reference the NN. Note that one use is the ability to transition an existing Hive deployment to use HDFS HA NN. A command line tool to execute JDOQL against the metastore. We believe the ability to execute JDOQL against the metastore will be a useful debugging tool for both users and Hive developers.
        Hide
        shreepadma Shreepadma Venugopalan added a comment -

        Command Notes
        metatool -listFSRoot Obtain the NameNode location. The value is prefixed with hdfs:// scheme
        metatool -updateLocation [-dryRun] <new-HA-NN-loc> <pre-NN-loc> Update records in the Hive metastore to point to the new NameNode location
        metatool -updateLocation [-dryRun] [-tablePropKey <table-prop-key>] [-serdePropKey <serde-prop-key>] <new-HA-NN-loc> <pre-NN-loc> Updates metastore records, including the ones that reference the Avro SerDe schema URL, to point to the new HDFS NameNode location.
        metatool -listFSRoot Verify that the update was executed successfully; it should now return the new location that matches the value of the dfs.nameservices property

        Show
        shreepadma Shreepadma Venugopalan added a comment - Command Notes metatool -listFSRoot Obtain the NameNode location. The value is prefixed with hdfs:// scheme metatool -updateLocation [-dryRun] <new-HA-NN-loc> <pre-NN-loc> Update records in the Hive metastore to point to the new NameNode location metatool -updateLocation [-dryRun] [-tablePropKey <table-prop-key>] [-serdePropKey <serde-prop-key>] <new-HA-NN-loc> <pre-NN-loc> Updates metastore records, including the ones that reference the Avro SerDe schema URL, to point to the new HDFS NameNode location. metatool -listFSRoot Verify that the update was executed successfully; it should now return the new location that matches the value of the dfs.nameservices property
        Hide
        shreepadma Shreepadma Venugopalan added a comment -

        The options used for the listFSRoot option are:

        <new-HA-NN-loc> – specifies the new location and should match the value of the dfs.nameservices property.
        <pre-NN-loc> – return value from the listFSRoot command that references the NameNode location.
        dryRun – outputs the location to the console without making persistent changes.
        tablePropKey <table-prop-key> – allows users to specify a table property key whose value field may reference the HDFS NameNode location and hence may require an update. To update the Avro SerDe schema URL, specify avro.schema.url for this argument.
        serdePropKey <serde-prop-key> – allows users to specify a SerDe property key whose value field may reference the HDFS NameNode location and hence may require an update. To update the Haivvero schema URL, specify schema.url for this argument.
        If you are unsure which version of Avro SerDe is used, pass both schemaPropKey and tablePropKey arguments with their respective values for the keys to updateLocation

        Show
        shreepadma Shreepadma Venugopalan added a comment - The options used for the listFSRoot option are: <new-HA-NN-loc> – specifies the new location and should match the value of the dfs.nameservices property. <pre-NN-loc> – return value from the listFSRoot command that references the NameNode location. dryRun – outputs the location to the console without making persistent changes. tablePropKey <table-prop-key> – allows users to specify a table property key whose value field may reference the HDFS NameNode location and hence may require an update. To update the Avro SerDe schema URL, specify avro.schema.url for this argument. serdePropKey <serde-prop-key> – allows users to specify a SerDe property key whose value field may reference the HDFS NameNode location and hence may require an update. To update the Haivvero schema URL, specify schema.url for this argument. If you are unsure which version of Avro SerDe is used, pass both schemaPropKey and tablePropKey arguments with their respective values for the keys to updateLocation
        Hide
        hudson Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-3056. Ability to bulk update location field in Db/Table/Partition records (Shreepadma Venugopalan via cws) (Revision 1380500)

        Result = ABORTED
        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380500
        Files :

        • /hive/trunk/bin/ext/metatool.sh
        • /hive/trunk/bin/metatool
        • /hive/trunk/build.xml
        • /hive/trunk/eclipse-templates/TestHiveMetaTool.launchtemplate
        • /hive/trunk/metastore/ivy.xml
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/tools
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaTool.java
        Show
        hudson Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3056 . Ability to bulk update location field in Db/Table/Partition records (Shreepadma Venugopalan via cws) (Revision 1380500) Result = ABORTED cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380500 Files : /hive/trunk/bin/ext/metatool.sh /hive/trunk/bin/metatool /hive/trunk/build.xml /hive/trunk/eclipse-templates/TestHiveMetaTool.launchtemplate /hive/trunk/metastore/ivy.xml /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/tools /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaTool.java
        Hide
        ashutoshc Ashutosh Chauhan added a comment -

        This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

        Show
        ashutoshc Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          People

          • Assignee:
            shreepadma Shreepadma Venugopalan
            Reporter:
            cwsteinbach Carl Steinbach
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development