Hive
  1. Hive
  2. HIVE-3231

msck repair should find partitions already containing data files

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.1, 0.10.0
    • Fix Version/s: 0.11.0
    • Component/s: Metastore
    • Labels:

      Description

      msck repair currently will only discover partition directories if they are empty.

      It seems a more apt use case to copy data files into a table, creating the partition directories as you go, rather than creating a bunch of empty partition directories, then running msck repair to dynamically add them, then inserting your actual data files.

      1. HIVE-3231.1.patch.txt
        1 kB
        Keegan Mosley
      2. HIVE-3231.2.patch.txt
        2 kB
        Keegan Mosley

        Activity

        Hide
        Carl Steinbach added a comment -

        @Keegan: This patch needs to be rebased on trunk. Also, repair.q has been split into repair.q and repair_hadoop23.q, so both files probably need to be updated. Finally, would you mind submitting a review request for this on either phabricator or reviewboard? Thanks.

        Show
        Carl Steinbach added a comment - @Keegan: This patch needs to be rebased on trunk. Also, repair.q has been split into repair.q and repair_hadoop23.q, so both files probably need to be updated. Finally, would you mind submitting a review request for this on either phabricator or reviewboard? Thanks.
        Show
        Keegan Mosley added a comment - https://reviews.apache.org/r/7649/
        Hide
        Ashutosh Chauhan added a comment -

        +1 will commit if tests pass.

        Show
        Ashutosh Chauhan added a comment - +1 will commit if tests pass.
        Hide
        Ashutosh Chauhan added a comment -

        Committed to trunk. Thanks, Keegan!

        Show
        Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Keegan!
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1845 (See https://builds.apache.org/job/Hive-trunk-h0.21/1845/)
        HIVE-3231 : msck repair should find partitions already containing data files (Keegan Mosley via Ashutosh Chauhan) (Revision 1418863)

        Result = FAILURE
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1418863
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java
        • /hive/trunk/ql/src/test/queries/clientpositive/repair.q
        • /hive/trunk/ql/src/test/queries/clientpositive/repair_hadoop23.q
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1845 (See https://builds.apache.org/job/Hive-trunk-h0.21/1845/ ) HIVE-3231 : msck repair should find partitions already containing data files (Keegan Mosley via Ashutosh Chauhan) (Revision 1418863) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1418863 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java /hive/trunk/ql/src/test/queries/clientpositive/repair.q /hive/trunk/ql/src/test/queries/clientpositive/repair_hadoop23.q
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-3231 : msck repair should find partitions already containing data files (Keegan Mosley via Ashutosh Chauhan) (Revision 1418863)

        Result = ABORTED
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1418863
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java
        • /hive/trunk/ql/src/test/queries/clientpositive/repair.q
        • /hive/trunk/ql/src/test/queries/clientpositive/repair_hadoop23.q
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3231 : msck repair should find partitions already containing data files (Keegan Mosley via Ashutosh Chauhan) (Revision 1418863) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1418863 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java /hive/trunk/ql/src/test/queries/clientpositive/repair.q /hive/trunk/ql/src/test/queries/clientpositive/repair_hadoop23.q

          People

          • Assignee:
            Carl Steinbach
            Reporter:
            Keegan Mosley
          • Votes:
            1 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development