Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.10.0
    • Component/s: Shims
    • Labels:
      None

      Description

      Hadoop archiving through hive is broken for 1.0. The hadoop archive command line and paths have changed for 1.0 and were not addressed in the shim layer.

      1. HIVE-3338_2.patch
        15 kB
        Vikram Dixit K
      2. HIVE-3338_3.patch
        65 kB
        Vikram Dixit K
      3. HIVE-3338_4.patch
        64 kB
        Vikram Dixit K
      4. HIVE-3338_5.patch
        70 kB
        Vikram Dixit K
      5. HIVE-3338.patch
        12 kB
        Vikram Dixit K

        Issue Links

          Activity

          Show
          Vikram Dixit K added a comment - <a href= https://issues.apache.org/jira/browse/HIVE-3025?focusedCommentId=13422589&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13422589 >Link</a> to comment
          Hide
          Vikram Dixit K added a comment -

          There is one issue that is not resolved. For the archive.q test, the golden results with 0.20 hadoop seem incorrect.

          The last queries are:
          SELECT SUM(hash(col)) FROM (SELECT transform using 'tr "\t" "_"' AS col
          FROM (SELECT * FROM old_name WHERE ds='1') subq1) subq2;
          ALTER TABLE old_name RENAME TO new_name;
          SELECT SUM(hash(col)) FROM (SELECT transform using 'tr "\t" "_"' AS col
          FROM (SELECT * FROM new_name WHERE ds='1') subq1) subq2;

          The select queries seem to produce different results in 20. The first one produces 48656137. But, after rename of the table, it produces NULL. This doesn't make sense to me. With my changes for 1.0, I see the proper results of 48656137 in both cases. It would be great if someone can explain to me why we expect NULL in the case of 20.

          Thanks,
          Vikram

          Show
          Vikram Dixit K added a comment - There is one issue that is not resolved. For the archive.q test, the golden results with 0.20 hadoop seem incorrect. The last queries are: SELECT SUM(hash(col)) FROM (SELECT transform using 'tr "\t" "_"' AS col FROM (SELECT * FROM old_name WHERE ds='1') subq1) subq2; ALTER TABLE old_name RENAME TO new_name; SELECT SUM(hash(col)) FROM (SELECT transform using 'tr "\t" "_"' AS col FROM (SELECT * FROM new_name WHERE ds='1') subq1) subq2; The select queries seem to produce different results in 20. The first one produces 48656137. But, after rename of the table, it produces NULL. This doesn't make sense to me. With my changes for 1.0, I see the proper results of 48656137 in both cases. It would be great if someone can explain to me why we expect NULL in the case of 20. Thanks, Vikram
          Hide
          Vikram Dixit K added a comment -

          This resolves the issues in 1.0. The renaming of an archived table does not work in case of 20. This patch does not address that issue.

          Show
          Vikram Dixit K added a comment - This resolves the issues in 1.0. The renaming of an archived table does not work in case of 20. This patch does not address that issue.
          Hide
          Vikram Dixit K added a comment -

          Missed the .out file for the new test.

          Show
          Vikram Dixit K added a comment - Missed the .out file for the new test.
          Hide
          Vikram Dixit K added a comment -

          build file change reverted.

          Show
          Vikram Dixit K added a comment - build file change reverted.
          Hide
          Ashutosh Chauhan added a comment -

          @Vikram,
          Can you create either a phabricator or review board entry of your patch ?

          Show
          Ashutosh Chauhan added a comment - @Vikram, Can you create either a phabricator or review board entry of your patch ?
          Hide
          Ashutosh Chauhan added a comment -

          Vikram,
          I left some comments on https://reviews.apache.org/r/6533/
          Also patch needs to be rebased.

          Show
          Ashutosh Chauhan added a comment - Vikram, I left some comments on https://reviews.apache.org/r/6533/ Also patch needs to be rebased.
          Hide
          Vikram Dixit K added a comment -

          Addressed Ashutosh's comments.

          Show
          Vikram Dixit K added a comment - Addressed Ashutosh's comments.
          Hide
          Ashutosh Chauhan added a comment -

          +1 will commit if tests pass.

          Show
          Ashutosh Chauhan added a comment - +1 will commit if tests pass.
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Vikram!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Vikram!
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1642 (See https://builds.apache.org/job/Hive-trunk-h0.21/1642/)
          HIVE-3338 : Archives broken for hadoop 1.0 (Vikram Dixit via Ashutosh Chauhan) (Revision 1379908)

          Result = FAILURE
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379908
          Files :

          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
          • /hive/trunk/ql/src/test/queries/clientpositive/archive.q
          • /hive/trunk/ql/src/test/queries/clientpositive/archive_excludeHadoop20.q
          • /hive/trunk/ql/src/test/results/clientpositive/archive.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/archive_excludeHadoop20.q.out
          • /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
          • /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
          • /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1642 (See https://builds.apache.org/job/Hive-trunk-h0.21/1642/ ) HIVE-3338 : Archives broken for hadoop 1.0 (Vikram Dixit via Ashutosh Chauhan) (Revision 1379908) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379908 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java /hive/trunk/ql/src/test/queries/clientpositive/archive.q /hive/trunk/ql/src/test/queries/clientpositive/archive_excludeHadoop20.q /hive/trunk/ql/src/test/results/clientpositive/archive.q.out /hive/trunk/ql/src/test/results/clientpositive/archive_excludeHadoop20.q.out /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-3338 : Archives broken for hadoop 1.0 (Vikram Dixit via Ashutosh Chauhan) (Revision 1379908)

          Result = ABORTED
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379908
          Files :

          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
          • /hive/trunk/ql/src/test/queries/clientpositive/archive.q
          • /hive/trunk/ql/src/test/queries/clientpositive/archive_excludeHadoop20.q
          • /hive/trunk/ql/src/test/results/clientpositive/archive.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/archive_excludeHadoop20.q.out
          • /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
          • /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
          • /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3338 : Archives broken for hadoop 1.0 (Vikram Dixit via Ashutosh Chauhan) (Revision 1379908) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379908 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java /hive/trunk/ql/src/test/queries/clientpositive/archive.q /hive/trunk/ql/src/test/queries/clientpositive/archive_excludeHadoop20.q /hive/trunk/ql/src/test/results/clientpositive/archive.q.out /hive/trunk/ql/src/test/results/clientpositive/archive_excludeHadoop20.q.out /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
          Hide
          Ashutosh Chauhan added a comment -

          This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          Show
          Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

            People

            • Assignee:
              Vikram Dixit K
              Reporter:
              Vikram Dixit K
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development