Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13696

Distcp between 2 secure clusters and non encrypted zones fails with connection timeout

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • None
    • None
    • distcp
    • None

    Description

      We are trying to do a distcp between 2 secure clusters. But distcp is between 2 unencrypted zones.
      eg : hadoop distcp sourceHdfs://tmp/text1.txt destHdfs://tmp
      throws

      org.apache.oozie.action.ActionExecutorException: JA009: connect timed out
      	at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:463)
      	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:437)
      	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1247)
      	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1425)
      	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
      	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
      	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.net.SocketTimeoutException: connect timed out
      	at java.net.PlainSocketImpl.socketConnect(Native Method)
      	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
      	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
      	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
      	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
      	at java.net.Socket.connect(Socket.java:589)
      	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
      	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
      	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
      	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
      	at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
      	at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
      	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
      	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
      	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
      	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
      	at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
      	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:184)
      	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:139)
      	at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
      	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:308)
      	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:180)
      	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:382)
      	at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1014)
      	at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1008)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
      	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1008)
      	at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2333)
      	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140)
      	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
      	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
      	at org.apache.hadoop.mapreduce.JobSubmitter.populateTokenCache(JobSubmitter.java:433)
      	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:173)
      	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
      	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
      	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
      	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:578)
      	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:573)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
      	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:573)
      	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:564)
      	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1232)
      	... 9 more
      

      We need to open dest cluster kms port 16000 for communication from source, then this works.

      This was working in cdh 5.5.1 using hadoop 2.6.0 without us opening up kms port on the destination cluster when we upgraded to 5.13.2 cdh we started facing this issue

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            rohit.peg Rohit Pegallapati

            Dates

              Created:
              Updated:

              Slack

                Issue deployment