Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.10.0
    • Component/s: Shims
    • Labels:
      None

      Description

      Hadoop archiving through hive is broken for 1.0. The hadoop archive command line and paths have changed for 1.0 and were not addressed in the shim layer.

      1. HIVE-3338.patch
        12 kB
        Vikram Dixit K
      2. HIVE-3338_2.patch
        15 kB
        Vikram Dixit K
      3. HIVE-3338_3.patch
        65 kB
        Vikram Dixit K
      4. HIVE-3338_4.patch
        64 kB
        Vikram Dixit K
      5. HIVE-3338_5.patch
        70 kB
        Vikram Dixit K

        Issue Links

          Activity

          Vikram Dixit K created issue -
          Show
          Vikram Dixit K added a comment - <a href= https://issues.apache.org/jira/browse/HIVE-3025?focusedCommentId=13422589&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13422589 >Link</a> to comment
          Vikram Dixit K made changes -
          Field Original Value New Value
          Attachment HIVE-3338.patch [ 12539338 ]
          Vikram Dixit K made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Vikram Dixit K added a comment -

          There is one issue that is not resolved. For the archive.q test, the golden results with 0.20 hadoop seem incorrect.

          The last queries are:
          SELECT SUM(hash(col)) FROM (SELECT transform using 'tr "\t" "_"' AS col
          FROM (SELECT * FROM old_name WHERE ds='1') subq1) subq2;
          ALTER TABLE old_name RENAME TO new_name;
          SELECT SUM(hash(col)) FROM (SELECT transform using 'tr "\t" "_"' AS col
          FROM (SELECT * FROM new_name WHERE ds='1') subq1) subq2;

          The select queries seem to produce different results in 20. The first one produces 48656137. But, after rename of the table, it produces NULL. This doesn't make sense to me. With my changes for 1.0, I see the proper results of 48656137 in both cases. It would be great if someone can explain to me why we expect NULL in the case of 20.

          Thanks,
          Vikram

          Show
          Vikram Dixit K added a comment - There is one issue that is not resolved. For the archive.q test, the golden results with 0.20 hadoop seem incorrect. The last queries are: SELECT SUM(hash(col)) FROM (SELECT transform using 'tr "\t" "_"' AS col FROM (SELECT * FROM old_name WHERE ds='1') subq1) subq2; ALTER TABLE old_name RENAME TO new_name; SELECT SUM(hash(col)) FROM (SELECT transform using 'tr "\t" "_"' AS col FROM (SELECT * FROM new_name WHERE ds='1') subq1) subq2; The select queries seem to produce different results in 20. The first one produces 48656137. But, after rename of the table, it produces NULL. This doesn't make sense to me. With my changes for 1.0, I see the proper results of 48656137 in both cases. It would be great if someone can explain to me why we expect NULL in the case of 20. Thanks, Vikram
          Hide
          Vikram Dixit K added a comment -

          This resolves the issues in 1.0. The renaming of an archived table does not work in case of 20. This patch does not address that issue.

          Show
          Vikram Dixit K added a comment - This resolves the issues in 1.0. The renaming of an archived table does not work in case of 20. This patch does not address that issue.
          Vikram Dixit K made changes -
          Attachment HIVE-3338_2.patch [ 12540122 ]
          Hide
          Vikram Dixit K added a comment -

          Missed the .out file for the new test.

          Show
          Vikram Dixit K added a comment - Missed the .out file for the new test.
          Vikram Dixit K made changes -
          Attachment HIVE-3338_3.patch [ 12540166 ]
          Hide
          Vikram Dixit K added a comment -

          build file change reverted.

          Show
          Vikram Dixit K added a comment - build file change reverted.
          Vikram Dixit K made changes -
          Attachment HIVE-3338_4.patch [ 12540171 ]
          Hide
          Ashutosh Chauhan added a comment -

          @Vikram,
          Can you create either a phabricator or review board entry of your patch ?

          Show
          Ashutosh Chauhan added a comment - @Vikram, Can you create either a phabricator or review board entry of your patch ?
          Ashutosh Chauhan made changes -
          Assignee Vikram Dixit K [ vikram.dixit ]
          Hide
          Ashutosh Chauhan added a comment -

          Vikram,
          I left some comments on https://reviews.apache.org/r/6533/
          Also patch needs to be rebased.

          Show
          Ashutosh Chauhan added a comment - Vikram, I left some comments on https://reviews.apache.org/r/6533/ Also patch needs to be rebased.
          Hide
          Vikram Dixit K added a comment -

          Addressed Ashutosh's comments.

          Show
          Vikram Dixit K added a comment - Addressed Ashutosh's comments.
          Vikram Dixit K made changes -
          Attachment HIVE-3338_5.patch [ 12543335 ]
          Hide
          Ashutosh Chauhan added a comment -

          +1 will commit if tests pass.

          Show
          Ashutosh Chauhan added a comment - +1 will commit if tests pass.
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Vikram!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Vikram!
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.10.0 [ 12320745 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1642 (See https://builds.apache.org/job/Hive-trunk-h0.21/1642/)
          HIVE-3338 : Archives broken for hadoop 1.0 (Vikram Dixit via Ashutosh Chauhan) (Revision 1379908)

          Result = FAILURE
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379908
          Files :

          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
          • /hive/trunk/ql/src/test/queries/clientpositive/archive.q
          • /hive/trunk/ql/src/test/queries/clientpositive/archive_excludeHadoop20.q
          • /hive/trunk/ql/src/test/results/clientpositive/archive.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/archive_excludeHadoop20.q.out
          • /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
          • /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
          • /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1642 (See https://builds.apache.org/job/Hive-trunk-h0.21/1642/ ) HIVE-3338 : Archives broken for hadoop 1.0 (Vikram Dixit via Ashutosh Chauhan) (Revision 1379908) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379908 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java /hive/trunk/ql/src/test/queries/clientpositive/archive.q /hive/trunk/ql/src/test/queries/clientpositive/archive_excludeHadoop20.q /hive/trunk/ql/src/test/results/clientpositive/archive.q.out /hive/trunk/ql/src/test/results/clientpositive/archive_excludeHadoop20.q.out /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-3338 : Archives broken for hadoop 1.0 (Vikram Dixit via Ashutosh Chauhan) (Revision 1379908)

          Result = ABORTED
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379908
          Files :

          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
          • /hive/trunk/ql/src/test/queries/clientpositive/archive.q
          • /hive/trunk/ql/src/test/queries/clientpositive/archive_excludeHadoop20.q
          • /hive/trunk/ql/src/test/results/clientpositive/archive.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/archive_excludeHadoop20.q.out
          • /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
          • /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
          • /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3338 : Archives broken for hadoop 1.0 (Vikram Dixit via Ashutosh Chauhan) (Revision 1379908) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379908 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java /hive/trunk/ql/src/test/queries/clientpositive/archive.q /hive/trunk/ql/src/test/queries/clientpositive/archive_excludeHadoop20.q /hive/trunk/ql/src/test/results/clientpositive/archive.q.out /hive/trunk/ql/src/test/results/clientpositive/archive_excludeHadoop20.q.out /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
          Hide
          Ashutosh Chauhan added a comment -

          This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          Show
          Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
          Ashutosh Chauhan made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Ashutosh Chauhan made changes -
          Link This issue relates to HIVE-3025 [ HIVE-3025 ]
          Vikram Dixit K made changes -
          Link This issue supercedes HIVE-3025 [ HIVE-3025 ]
          Hide
          Lefty Leverenz added a comment -

          This removed the configuration parameter hive.archive.har.parentdir.settable in Hive 0.10.0.

          The wiki has been updated (at last):

          Show
          Lefty Leverenz added a comment - This removed the configuration parameter hive.archive.har.parentdir.settable in Hive 0.10.0. The wiki has been updated (at last): hive.archive.har.parentdir.settable
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          2m 15s 1 Vikram Dixit K 06/Aug/12 19:54
          Patch Available Patch Available Resolved Resolved
          26d 5h 16m 1 Ashutosh Chauhan 02/Sep/12 01:10
          Resolved Resolved Closed Closed
          130d 19h 42m 1 Ashutosh Chauhan 10/Jan/13 19:53

            People

            • Assignee:
              Vikram Dixit K
              Reporter:
              Vikram Dixit K
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development