Hive
  1. Hive
  2. HIVE-7558

HCatLoader reuses credentials across jobs

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.1
    • Fix Version/s: 0.14.0
    • Component/s: HCatalog
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HCatLoader reuses credentials of stage1 in stage2 for some of the pig queries. This causes stage-2 to fail, if stage-2 runs for more than 10 mins. Pig queries which loads data using HCatLoader, filters only by partition columns and does an order by will run into this problem. Exceptions will be very similar to the following:

      2014-07-22 17:28:49,337 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Unable to recreate exception from backed error: AttemptID:<attemptid> Info:RemoteTrace:
      org.apache.hadoop.security.token.SecretManager$InvalidToken: token (HDFS_DELEGATION_TOKEN token <tokenid> for <user>) can't be found in cache
      at org.apache.hadoop.ipc.Client.call(Client.java:1095)
      at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195)
      at $Proxy7.getFileInfo(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67)
      at $Proxy7.getFileInfo(Unknown Source)
      at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1305)
      at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:734)
      at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:176)
      at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:51)
      at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:284)
      at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:282)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1300)
      at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:281)
      at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:51)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:722)
      at LocalTrace:
      org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl: token (HDFS_DELEGATION_TOKEN token <tokenid> for <user>) can't be found in cache
      at org.apache.hadoop.yarn.server.nodemanager.api.protocolrecords.impl.pb.LocalResourceStatusPBImpl.convertFromProtoFormat(LocalResourceStatusPBImpl.java:217)
      at org.apache.hadoop.yarn.server.nodemanager.api.protocolrecords.impl.pb.LocalResourceStatusPBImpl.getException(LocalResourceStatusPBImpl.java:147)
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.update(ResourceLocalizationService.java:823)
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.processHeartbeat(ResourceLocalizationService.java:497)
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.heartbeat(ResourceLocalizationService.java:224)
      at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.service.LocalizationProtocolPBServiceImpl.heartbeat(LocalizationProtocolPBServiceImpl.java:46)
      at org.apache.hadoop.yarn.proto.LocalizationProtocol$LocalizationProtocolService$2.callBlockingMethod(LocalizationProtocol.java:57)
      at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:353)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1476)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1472)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1300)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1470)

      1. HIVE-7558.patch
        0.8 kB
        Thiruvel Thirumoolan

        Activity

        Hide
        Thejas M Nair added a comment -

        This has been fixed in 0.14 release. Please open new jira if you see any issues.

        Show
        Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
        Hide
        Daniel Dai added a comment -

        Patch committed to trunk. Thanks Thiruvel!

        Show
        Daniel Dai added a comment - Patch committed to trunk. Thanks Thiruvel!
        Hide
        Thiruvel Thirumoolan added a comment -
        Show
        Thiruvel Thirumoolan added a comment - Thanks Daniel Dai . Review link @ https://reviews.apache.org/r/24163/
        Hide
        Daniel Dai added a comment -

        +1. This only occurs on secure cluster, so it is hard to write a unit test. I will commit the patch shortly if no objection.

        Show
        Daniel Dai added a comment - +1. This only occurs on secure cluster, so it is hard to write a unit test. I will commit the patch shortly if no objection.
        Hide
        Thiruvel Thirumoolan added a comment -

        Attaching patch. Do not copy job's credentials in HCatLoader's objects.

        Show
        Thiruvel Thirumoolan added a comment - Attaching patch. Do not copy job's credentials in HCatLoader's objects.

          People

          • Assignee:
            Thiruvel Thirumoolan
            Reporter:
            Thiruvel Thirumoolan
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development