Hive
  1. Hive
  2. HIVE-2643

Recent patch prevents Hadoop confs from loading in 0.20.204

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: Configuration
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      After the application of HIVE-2362, we're unable to run queries against the Hive cluster (RC2 and 3):

      java.io.IOException: Failed to set permissions of path: /user/jhoman-667872320/.staging to 0700
              at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:680)
              at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:653)
              at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:483)
              at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318)
              at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183)
              at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)

      this is due to all the hadoop confs not being loaded and the LocalJobRunner (which builds staging directories in this manner) being used instead of the regular one. It's also not possible to access the hdfs since no default fs is specified.

      Reverting 2362 fixes this, but I've not yet looked at that patch to see the exact cause.

      1. HIVE-2643.D759.2.patch
        114 kB
        Carl Steinbach
      2. HIVE-2643.1.patch.txt
        6 kB
        Carl Steinbach
      3. ASF.LICENSE.NOT.GRANTED--HIVE-2643.D789.1.patch
        2 kB
        Phabricator
      4. ASF.LICENSE.NOT.GRANTED--HIVE-2643.D765.2.patch
        20 kB
        Phabricator
      5. ASF.LICENSE.NOT.GRANTED--HIVE-2643.D765.1.patch
        20 kB
        Phabricator
      6. ASF.LICENSE.NOT.GRANTED--HIVE-2643.D759.2.patch
        20 kB
        Phabricator
      7. ASF.LICENSE.NOT.GRANTED--HIVE-2643.D759.1.patch
        20 kB
        Phabricator
      8. ASF.LICENSE.NOT.GRANTED--HIVE-2643.D753.1.patch
        0.6 kB
        Phabricator
      9. ASF.LICENSE.NOT.GRANTED--HIVE-2643.D747.1.patch
        6 kB
        Phabricator

        Issue Links

          Activity

          Hide
          Phabricator added a comment -

          cwsteinbach requested code review of "HIVE-2643 [jira] HIVE-2362 preventing Hadoop confs from loading in 0.20.204".
          Reviewers: JIRA

          HIVE-2643

          After the application of <a href="https://issues.apache.org/jira/browse/HIVE-2362" title="HiveConf properties not appearing in the output of 'set' or 'set -v'"><del>HIVE-2362</del></a>, we're unable to run queries against the Hive cluster (RC2 and 3): <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>java.io.IOException: Failed to set permissions of path: /user/jhoman-667872320/.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:680) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:653) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:483) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)</pre> </div>
          </div>

          this is due to all the hadoop confs not being loaded and the LocalJobRunner (which builds staging directories in this manner) being used instead of the regular one. It's also not possible to access the hdfs since no default fs is specified.

          Reverting 2362 fixes this, but I've not yet looked at that patch to see the exact cause.

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D747

          AFFECTED FILES
          common/build.xml
          common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java
          common/src/test/resources/core-site.xml
          common/src/test/resources/hive-site.xml

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/1593/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - cwsteinbach requested code review of " HIVE-2643 [jira] HIVE-2362 preventing Hadoop confs from loading in 0.20.204". Reviewers: JIRA HIVE-2643 After the application of <a href="https://issues.apache.org/jira/browse/HIVE-2362" title="HiveConf properties not appearing in the output of 'set' or 'set -v'"><del> HIVE-2362 </del></a>, we're unable to run queries against the Hive cluster (RC2 and 3): <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>java.io.IOException: Failed to set permissions of path: /user/jhoman-667872320/.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:680) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:653) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:483) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)</pre> </div> </div> this is due to all the hadoop confs not being loaded and the LocalJobRunner (which builds staging directories in this manner) being used instead of the regular one. It's also not possible to access the hdfs since no default fs is specified. Reverting 2362 fixes this, but I've not yet looked at that patch to see the exact cause. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D747 AFFECTED FILES common/build.xml common/src/java/org/apache/hadoop/hive/conf/HiveConf.java common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java common/src/test/resources/core-site.xml common/src/test/resources/hive-site.xml MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/1593/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
          Hide
          Carl Steinbach added a comment -

          Running tests now.

          Show
          Carl Steinbach added a comment - Running tests now.
          Hide
          Phabricator added a comment -

          cwsteinbach requested code review of "HIVE-2643 [jira] HIVE-2362 preventing Hadoop confs from loading in 0.20.204".
          Reviewers: JIRA

          lint check

          After the application of <a href="https://issues.apache.org/jira/browse/HIVE-2362" title="HiveConf properties not appearing in the output of 'set' or 'set -v'"><del>HIVE-2362</del></a>, we're unable to run queries against the Hive cluster (RC2 and 3): <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>java.io.IOException: Failed to set permissions of path: /user/jhoman-667872320/.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:680) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:653) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:483) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)</pre> </div>
          </div>

          this is due to all the hadoop confs not being loaded and the LocalJobRunner (which builds staging directories in this manner) being used instead of the regular one. It's also not possible to access the hdfs since no default fs is specified.

          Reverting 2362 fixes this, but I've not yet looked at that patch to see the exact cause.

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D753

          AFFECTED FILES
          common/src/java/org/apache/hadoop/hive/conf/HiveConf.java

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/1599/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - cwsteinbach requested code review of " HIVE-2643 [jira] HIVE-2362 preventing Hadoop confs from loading in 0.20.204". Reviewers: JIRA lint check After the application of <a href="https://issues.apache.org/jira/browse/HIVE-2362" title="HiveConf properties not appearing in the output of 'set' or 'set -v'"><del> HIVE-2362 </del></a>, we're unable to run queries against the Hive cluster (RC2 and 3): <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>java.io.IOException: Failed to set permissions of path: /user/jhoman-667872320/.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:680) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:653) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:483) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)</pre> </div> </div> this is due to all the hadoop confs not being loaded and the LocalJobRunner (which builds staging directories in this manner) being used instead of the regular one. It's also not possible to access the hdfs since no default fs is specified. Reverting 2362 fixes this, but I've not yet looked at that patch to see the exact cause. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D753 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/1599/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
          Hide
          Phabricator added a comment -

          cwsteinbach requested code review of "HIVE-2643 [jira] HIVE-2362 preventing Hadoop confs from loading in 0.20.204".
          Reviewers: JIRA

          HIVE-2643

          After the application of <a href="https://issues.apache.org/jira/browse/HIVE-2362" title="HiveConf properties not appearing in the output of 'set' or 'set -v'"><del>HIVE-2362</del></a>, we're unable to run queries against the Hive cluster (RC2 and 3): <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>java.io.IOException: Failed to set permissions of path: /user/jhoman-667872320/.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:680) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:653) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:483) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)</pre> </div>
          </div>

          this is due to all the hadoop confs not being loaded and the LocalJobRunner (which builds staging directories in this manner) being used instead of the regular one. It's also not possible to access the hdfs since no default fs is specified.

          Reverting 2362 fixes this, but I've not yet looked at that patch to see the exact cause.

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D759

          AFFECTED FILES
          common/build.xml
          common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java
          common/src/test/resources/core-site.xml
          common/src/test/resources/hive-site.xml
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/1605/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - cwsteinbach requested code review of " HIVE-2643 [jira] HIVE-2362 preventing Hadoop confs from loading in 0.20.204". Reviewers: JIRA HIVE-2643 After the application of <a href="https://issues.apache.org/jira/browse/HIVE-2362" title="HiveConf properties not appearing in the output of 'set' or 'set -v'"><del> HIVE-2362 </del></a>, we're unable to run queries against the Hive cluster (RC2 and 3): <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>java.io.IOException: Failed to set permissions of path: /user/jhoman-667872320/.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:680) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:653) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:483) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)</pre> </div> </div> this is due to all the hadoop confs not being loaded and the LocalJobRunner (which builds staging directories in this manner) being used instead of the regular one. It's also not possible to access the hdfs since no default fs is specified. Reverting 2362 fixes this, but I've not yet looked at that patch to see the exact cause. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D759 AFFECTED FILES common/build.xml common/src/java/org/apache/hadoop/hive/conf/HiveConf.java common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java common/src/test/resources/core-site.xml common/src/test/resources/hive-site.xml ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/1605/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
          Hide
          Carl Steinbach added a comment -

          Changing the title of this ticket since referencing another JIRA seems to confuse arc.

          Show
          Carl Steinbach added a comment - Changing the title of this ticket since referencing another JIRA seems to confuse arc.
          Hide
          Phabricator added a comment -

          cwsteinbach updated the revision "HIVE-2643 [jira] HIVE-2362 preventing Hadoop confs from loading in 0.20.204".
          Reviewers: JIRA

          Fix for HIVE-2643

          REVISION DETAIL
          https://reviews.facebook.net/D759

          AFFECTED FILES
          common/build.xml
          common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java
          common/src/test/resources/core-site.xml
          common/src/test/resources/hive-site.xml
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java

          Show
          Phabricator added a comment - cwsteinbach updated the revision " HIVE-2643 [jira] HIVE-2362 preventing Hadoop confs from loading in 0.20.204". Reviewers: JIRA Fix for HIVE-2643 REVISION DETAIL https://reviews.facebook.net/D759 AFFECTED FILES common/build.xml common/src/java/org/apache/hadoop/hive/conf/HiveConf.java common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java common/src/test/resources/core-site.xml common/src/test/resources/hive-site.xml ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
          Hide
          Phabricator added a comment -

          cwsteinbach requested code review of "HIVE-2643 [jira] Recent patch prevents Hadoop confs from loading in 0.20.204".
          Reviewers: JIRA

          HIVE-2643

          After the application of <a href="https://issues.apache.org/jira/browse/HIVE-2362" title="HiveConf properties not appearing in the output of 'set' or 'set -v'"><del>HIVE-2362</del></a>, we're unable to run queries against the Hive cluster (RC2 and 3): <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>java.io.IOException: Failed to set permissions of path: /user/jhoman-667872320/.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:680) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:653) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:483) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)</pre> </div>
          </div>

          this is due to all the hadoop confs not being loaded and the LocalJobRunner (which builds staging directories in this manner) being used instead of the regular one. It's also not possible to access the hdfs since no default fs is specified.

          Reverting 2362 fixes this, but I've not yet looked at that patch to see the exact cause.

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D765

          AFFECTED FILES
          common/build.xml
          common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java
          common/src/test/resources/core-site.xml
          common/src/test/resources/hive-site.xml
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/1617/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - cwsteinbach requested code review of " HIVE-2643 [jira] Recent patch prevents Hadoop confs from loading in 0.20.204". Reviewers: JIRA HIVE-2643 After the application of <a href="https://issues.apache.org/jira/browse/HIVE-2362" title="HiveConf properties not appearing in the output of 'set' or 'set -v'"><del> HIVE-2362 </del></a>, we're unable to run queries against the Hive cluster (RC2 and 3): <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>java.io.IOException: Failed to set permissions of path: /user/jhoman-667872320/.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:680) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:653) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:483) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)</pre> </div> </div> this is due to all the hadoop confs not being loaded and the LocalJobRunner (which builds staging directories in this manner) being used instead of the regular one. It's also not possible to access the hdfs since no default fs is specified. Reverting 2362 fixes this, but I've not yet looked at that patch to see the exact cause. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D765 AFFECTED FILES common/build.xml common/src/java/org/apache/hadoop/hive/conf/HiveConf.java common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java common/src/test/resources/core-site.xml common/src/test/resources/hive-site.xml ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/1617/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
          Hide
          John Sichi added a comment -

          Marek and I will take a look into the arc confusion with the JIRA reference.

          Show
          John Sichi added a comment - Marek and I will take a look into the arc confusion with the JIRA reference.
          Hide
          Carl Steinbach added a comment -

          Investigating some test failures.

          Show
          Carl Steinbach added a comment - Investigating some test failures.
          Hide
          Jakob Homan added a comment -

          I've applied HIVE-2643.1.patch.txt (the only ASF-licensed patch; are the others artifacts of the issue name confusion?) to the RC3 tag in svn, built a new package and run against our 204 cluster and everything is working now. Didn't do a code review, but +1 on the result.

          Show
          Jakob Homan added a comment - I've applied HIVE-2643 .1.patch.txt (the only ASF-licensed patch; are the others artifacts of the issue name confusion?) to the RC3 tag in svn, built a new package and run against our 204 cluster and everything is working now. Didn't do a code review, but +1 on the result.
          Hide
          Phabricator added a comment -

          mareksapotafb has abandoned the revision "HIVE-2643 [jira] HIVE-2362 preventing Hadoop confs from loading in 0.20.204".

          REVISION DETAIL
          https://reviews.facebook.net/D753

          Show
          Phabricator added a comment - mareksapotafb has abandoned the revision " HIVE-2643 [jira] HIVE-2362 preventing Hadoop confs from loading in 0.20.204". REVISION DETAIL https://reviews.facebook.net/D753
          Hide
          Phabricator added a comment -

          mareksapotafb has abandoned the revision "HIVE-2643 [jira] HIVE-2362 preventing Hadoop confs from loading in 0.20.204".

          REVISION DETAIL
          https://reviews.facebook.net/D747

          Show
          Phabricator added a comment - mareksapotafb has abandoned the revision " HIVE-2643 [jira] HIVE-2362 preventing Hadoop confs from loading in 0.20.204". REVISION DETAIL https://reviews.facebook.net/D747
          Hide
          Phabricator added a comment -

          mareksapotafb has abandoned the revision "HIVE-2643 [jira] HIVE-2362 preventing Hadoop confs from loading in 0.20.204".

          REVISION DETAIL
          https://reviews.facebook.net/D759

          Show
          Phabricator added a comment - mareksapotafb has abandoned the revision " HIVE-2643 [jira] HIVE-2362 preventing Hadoop confs from loading in 0.20.204". REVISION DETAIL https://reviews.facebook.net/D759
          Hide
          Phabricator added a comment -

          cwsteinbach requested code review of "HIVE-2643 [jira] Recent patch prevents Hadoop confs from loading in 0.20.204".
          Reviewers: JIRA

          cp

          After the application of <a href="https://issues.apache.org/jira/browse/HIVE-2362" title="HiveConf properties not appearing in the output of 'set' or 'set -v'"><del>HIVE-2362</del></a>, we're unable to run queries against the Hive cluster (RC2 and 3): <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>java.io.IOException: Failed to set permissions of path: /user/jhoman-667872320/.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:680) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:653) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:483) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)</pre> </div>
          </div>

          this is due to all the hadoop confs not being loaded and the LocalJobRunner (which builds staging directories in this manner) being used instead of the regular one. It's also not possible to access the hdfs since no default fs is specified.

          Reverting 2362 fixes this, but I've not yet looked at that patch to see the exact cause.

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D789

          AFFECTED FILES
          build.xml
          common/src/java/org/apache/hadoop/hive/conf/HiveConf.java

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/1671/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - cwsteinbach requested code review of " HIVE-2643 [jira] Recent patch prevents Hadoop confs from loading in 0.20.204". Reviewers: JIRA cp After the application of <a href="https://issues.apache.org/jira/browse/HIVE-2362" title="HiveConf properties not appearing in the output of 'set' or 'set -v'"><del> HIVE-2362 </del></a>, we're unable to run queries against the Hive cluster (RC2 and 3): <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>java.io.IOException: Failed to set permissions of path: /user/jhoman-667872320/.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:680) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:653) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:483) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)</pre> </div> </div> this is due to all the hadoop confs not being loaded and the LocalJobRunner (which builds staging directories in this manner) being used instead of the regular one. It's also not possible to access the hdfs since no default fs is specified. Reverting 2362 fixes this, but I've not yet looked at that patch to see the exact cause. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D789 AFFECTED FILES build.xml common/src/java/org/apache/hadoop/hive/conf/HiveConf.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/1671/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
          Hide
          Phabricator added a comment -

          cwsteinbach updated the revision "HIVE-2643 [jira] Recent patch prevents Hadoop confs from loading in 0.20.204".
          Reviewers: JIRA

          Fixed some test diffs.

          REVISION DETAIL
          https://reviews.facebook.net/D765

          AFFECTED FILES
          build.xml
          common/build.xml
          common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java
          common/src/test/resources/core-site.xml
          common/src/test/resources/hive-site.xml
          conf/hive-default.xml.template
          conf/hive-default.xml
          conf/hive-site.xml

          Show
          Phabricator added a comment - cwsteinbach updated the revision " HIVE-2643 [jira] Recent patch prevents Hadoop confs from loading in 0.20.204". Reviewers: JIRA Fixed some test diffs. REVISION DETAIL https://reviews.facebook.net/D765 AFFECTED FILES build.xml common/build.xml common/src/java/org/apache/hadoop/hive/conf/HiveConf.java common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java common/src/test/resources/core-site.xml common/src/test/resources/hive-site.xml conf/hive-default.xml.template conf/hive-default.xml conf/hive-site.xml
          Hide
          Carl Steinbach added a comment -

          HIVE-2643.D759.2.patch is the latest version of the patch. It is ready for review here: https://reviews.facebook.net/D765

          I'm running full tests now.

          Show
          Carl Steinbach added a comment - HIVE-2643 .D759.2.patch is the latest version of the patch. It is ready for review here: https://reviews.facebook.net/D765 I'm running full tests now.
          Hide
          Carl Steinbach added a comment -

          @Jakob: Can you please try testing HIVE-2643.D759.2.patch? This version has some changes that were not included in HIVE-2643.1.patch.txt. Thanks!

          Show
          Carl Steinbach added a comment - @Jakob: Can you please try testing HIVE-2643 .D759.2.patch? This version has some changes that were not included in HIVE-2643 .1.patch.txt. Thanks!
          Hide
          Jakob Homan added a comment -

          Applied against RC3 and tested again. Everything works. Also reviewed the patch. +1, with the minor nit that having the test conf return "core-site.xml" for the default filesystem makes the test a bit wonky to read ("... wait, that's not right. oh, ok"), but that's not worth spinning another patch for if nothing else is blocking this version.

          Show
          Jakob Homan added a comment - Applied against RC3 and tested again. Everything works. Also reviewed the patch. +1, with the minor nit that having the test conf return "core-site.xml" for the default filesystem makes the test a bit wonky to read ("... wait, that's not right. oh, ok"), but that's not worth spinning another patch for if nothing else is blocking this version.
          Hide
          John Sichi added a comment -

          +1, passed tests, committing to trunk.

          Show
          John Sichi added a comment - +1, passed tests, committing to trunk.
          Hide
          Phabricator added a comment -

          jsichi has added reviewers to the revision "HIVE-2643 [jira] HIVE-2362 preventing Hadoop confs from loading in 0.20.204".
          Added Reviewers: jsichi

          REVISION DETAIL
          https://reviews.facebook.net/D759

          Show
          Phabricator added a comment - jsichi has added reviewers to the revision " HIVE-2643 [jira] HIVE-2362 preventing Hadoop confs from loading in 0.20.204". Added Reviewers: jsichi REVISION DETAIL https://reviews.facebook.net/D759
          Hide
          John Sichi added a comment -

          Committed to trunk and 0.8-r2. Thanks Carl!

          Show
          John Sichi added a comment - Committed to trunk and 0.8-r2. Thanks Carl!
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.23.0 #19 (See https://builds.apache.org/job/Hive-trunk-h0.23.0/19/)
          HIVE-2643. Recent patch prevents Hadoop confs from loading in 0.20.204
          (Carl Steinbach via jvs)

          jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213514
          Files :

          • /hive/trunk/build.xml
          • /hive/trunk/common/build.xml
          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java
          • /hive/trunk/common/src/test/resources/core-site.xml
          • /hive/trunk/common/src/test/resources/hive-site.xml
          • /hive/trunk/conf/hive-default.xml
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/conf/hive-site.xml
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.23.0 #19 (See https://builds.apache.org/job/Hive-trunk-h0.23.0/19/ ) HIVE-2643 . Recent patch prevents Hadoop confs from loading in 0.20.204 (Carl Steinbach via jvs) jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213514 Files : /hive/trunk/build.xml /hive/trunk/common/build.xml /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java /hive/trunk/common/src/test/resources/core-site.xml /hive/trunk/common/src/test/resources/hive-site.xml /hive/trunk/conf/hive-default.xml /hive/trunk/conf/hive-default.xml.template /hive/trunk/conf/hive-site.xml
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1142 (See https://builds.apache.org/job/Hive-trunk-h0.21/1142/)
          HIVE-2643. Recent patch prevents Hadoop confs from loading in 0.20.204
          (Carl Steinbach via jvs)

          jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213514
          Files :

          • /hive/trunk/build.xml
          • /hive/trunk/common/build.xml
          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java
          • /hive/trunk/common/src/test/resources/core-site.xml
          • /hive/trunk/common/src/test/resources/hive-site.xml
          • /hive/trunk/conf/hive-default.xml
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/conf/hive-site.xml
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1142 (See https://builds.apache.org/job/Hive-trunk-h0.21/1142/ ) HIVE-2643 . Recent patch prevents Hadoop confs from loading in 0.20.204 (Carl Steinbach via jvs) jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213514 Files : /hive/trunk/build.xml /hive/trunk/common/build.xml /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java /hive/trunk/common/src/test/resources/core-site.xml /hive/trunk/common/src/test/resources/hive-site.xml /hive/trunk/conf/hive-default.xml /hive/trunk/conf/hive-default.xml.template /hive/trunk/conf/hive-site.xml

            People

            • Assignee:
              Carl Steinbach
              Reporter:
              Jakob Homan
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development