Hive
  1. Hive
  2. HIVE-3068

Add ability to export table metadata as JSON on table drop

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Committed. Thanks Andrew.

      Description

      When a table is dropped, the contents go to the users trash but the metadata is lost. It would be super neat to be able to save the metadata as well so that tables could be trivially re-instantiated via thrift.

      1. HIVE-3068.2.patch.txt
        4 kB
        Andrew Chalfant
      2. hive-3068.patch.txt
        9 kB
        Edward Capriolo

        Issue Links

          Activity

          Hide
          Andrew Chalfant added a comment -
          Show
          Andrew Chalfant added a comment - Diff available here: https://reviews.facebook.net/D3795
          Hide
          Edward Capriolo added a comment -

          +1 I like this. Add a unit test and we are good to go.

          Show
          Edward Capriolo added a comment - +1 I like this. Add a unit test and we are good to go.
          Hide
          Edward Capriolo added a comment -

          We have done another issue for inport and export functions. Maybe you can borrow that code to dump the metadata instead of the json export you have done.

          Show
          Edward Capriolo added a comment - We have done another issue for inport and export functions. Maybe you can borrow that code to dump the metadata instead of the json export you have done.
          Hide
          Andrew Chalfant added a comment -

          Can you point me to a diff, issue number, or where the code is located?

          Show
          Andrew Chalfant added a comment - Can you point me to a diff, issue number, or where the code is located?
          Hide
          Kevin Wilfong added a comment -
          Show
          Kevin Wilfong added a comment - I think he's referring to https://issues.apache.org/jira/browse/HIVE-1918
          Hide
          Andrew Chalfant added a comment -

          Updated patch

          Show
          Andrew Chalfant added a comment - Updated patch
          Hide
          Andrew Chalfant added a comment -

          New revision is here: https://reviews.facebook.net/D4341

          Show
          Andrew Chalfant added a comment - New revision is here: https://reviews.facebook.net/D4341
          Hide
          Edward Capriolo added a comment -

          I will look this over tonight.

          Show
          Edward Capriolo added a comment - I will look this over tonight.
          Hide
          Edward Capriolo added a comment -

          See comments in phabricator.

          Show
          Edward Capriolo added a comment - See comments in phabricator.
          Hide
          Andrew Chalfant added a comment -

          Hi Edward, is it possible that you didn't commit your comments? I don't see anything.

          Show
          Andrew Chalfant added a comment - Hi Edward, is it possible that you didn't commit your comments? I don't see anything.
          Hide
          Edward Capriolo added a comment -

          My comments are inline in the patch

          https://reviews.facebook.net/D4341

          Show
          Edward Capriolo added a comment - My comments are inline in the patch https://reviews.facebook.net/D4341
          Hide
          Kevin Wilfong added a comment -

          Did you forget to hit Clowncopterize? I can't see them either.

          Show
          Kevin Wilfong added a comment - Did you forget to hit Clowncopterize? I can't see them either.
          Hide
          Edward Capriolo added a comment -

          I see I had no idea that the comments were not added until you touched a button named clowncopterize. I do not know why I did not figure this out, it is so intuitive.

          Show
          Edward Capriolo added a comment - I see I had no idea that the comments were not added until you touched a button named clowncopterize. I do not know why I did not figure this out, it is so intuitive.
          Hide
          Andrew Chalfant added a comment -

          Okay, I have updated the diff to take parameterized locations and whether or not the file is deleted, so that we can test the existence of the subfolder int he export location with dfs -rmr (which throws and error if the path doesn't exist). This should resolve the issues you brought up last time.

          Show
          Andrew Chalfant added a comment - Okay, I have updated the diff to take parameterized locations and whether or not the file is deleted, so that we can test the existence of the subfolder int he export location with dfs -rmr (which throws and error if the path doesn't exist). This should resolve the issues you brought up last time.
          Hide
          Andrew Chalfant added a comment -

          ping

          Show
          Andrew Chalfant added a comment - ping
          Hide
          Andrew Chalfant added a comment -

          pingping

          Show
          Andrew Chalfant added a comment - pingping
          Hide
          Edward Capriolo added a comment -

          Dude don't do that. The average patch sits on the queue for some time and many committers volunteer time ill review ASAP.

          Show
          Edward Capriolo added a comment - Dude don't do that. The average patch sits on the queue for some time and many committers volunteer time ill review ASAP.
          Hide
          Andrew Chalfant added a comment -

          Sorry about that, did not realize it was a faux pas, won't happen again.

          Show
          Andrew Chalfant added a comment - Sorry about that, did not realize it was a faux pas, won't happen again.
          Hide
          Edward Capriolo added a comment -

          Andrew. It is ok this patch took a little longer then it should have to get reviewed. I applied arc diff
          https://reviews.facebook.net/D4341

          You are missing the changes to the HiveConf.java and to the hive-site.xml.template make these changes and I will review.

          Show
          Edward Capriolo added a comment - Andrew. It is ok this patch took a little longer then it should have to get reviewed. I applied arc diff https://reviews.facebook.net/D4341 You are missing the changes to the HiveConf.java and to the hive-site.xml.template make these changes and I will review.
          Hide
          Andrew Chalfant added a comment -

          Added, thanks Edward!

          Show
          Andrew Chalfant added a comment - Added, thanks Edward!
          Hide
          Edward Capriolo added a comment -

          Minor correction to patch

          Show
          Edward Capriolo added a comment - Minor correction to patch
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1640 (See https://builds.apache.org/job/Hive-trunk-h0.21/1640/)
          HIVE-3068 Export table metadata as JSON on table drop (Andrew Chalfant via egc) (Revision 1379683)

          Result = FAILURE
          ecapriolo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379683
          Files :

          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/MetaDataExportListener.java
          • /hive/trunk/ql/src/test/queries/clientpositive/metadata_export_drop.q
          • /hive/trunk/ql/src/test/results/clientpositive/metadata_export_drop.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1640 (See https://builds.apache.org/job/Hive-trunk-h0.21/1640/ ) HIVE-3068 Export table metadata as JSON on table drop (Andrew Chalfant via egc) (Revision 1379683) Result = FAILURE ecapriolo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379683 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/MetaDataExportListener.java /hive/trunk/ql/src/test/queries/clientpositive/metadata_export_drop.q /hive/trunk/ql/src/test/results/clientpositive/metadata_export_drop.q.out
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-3068 Export table metadata as JSON on table drop (Andrew Chalfant via egc) (Revision 1379683)

          Result = ABORTED
          ecapriolo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379683
          Files :

          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/MetaDataExportListener.java
          • /hive/trunk/ql/src/test/queries/clientpositive/metadata_export_drop.q
          • /hive/trunk/ql/src/test/results/clientpositive/metadata_export_drop.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3068 Export table metadata as JSON on table drop (Andrew Chalfant via egc) (Revision 1379683) Result = ABORTED ecapriolo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379683 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/MetaDataExportListener.java /hive/trunk/ql/src/test/queries/clientpositive/metadata_export_drop.q /hive/trunk/ql/src/test/results/clientpositive/metadata_export_drop.q.out
          Hide
          Ashutosh Chauhan added a comment -

          This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          Show
          Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

            People

            • Assignee:
              Andrew Chalfant
              Reporter:
              Andrew Chalfant
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development