Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13696

Distcp between 2 secure clusters and non encrypted zones fails with connection timeout

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: distcp
    • Labels:
      None

      Description

      We are trying to do a distcp between 2 secure clusters. But distcp is between 2 unencrypted zones.
      eg : hadoop distcp sourceHdfs://tmp/text1.txt destHdfs://tmp
      throws

      org.apache.oozie.action.ActionExecutorException: JA009: connect timed out
      	at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:463)
      	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:437)
      	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1247)
      	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1425)
      	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
      	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
      	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.net.SocketTimeoutException: connect timed out
      	at java.net.PlainSocketImpl.socketConnect(Native Method)
      	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
      	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
      	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
      	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
      	at java.net.Socket.connect(Socket.java:589)
      	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
      	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
      	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
      	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
      	at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
      	at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
      	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
      	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
      	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
      	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
      	at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
      	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:184)
      	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:139)
      	at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
      	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:308)
      	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:180)
      	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:382)
      	at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1014)
      	at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1008)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
      	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1008)
      	at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2333)
      	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140)
      	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
      	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
      	at org.apache.hadoop.mapreduce.JobSubmitter.populateTokenCache(JobSubmitter.java:433)
      	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:173)
      	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
      	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
      	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
      	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:578)
      	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:573)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
      	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:573)
      	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:564)
      	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1232)
      	... 9 more
      

      We need to open dest cluster kms port 16000 for communication from source, then this works.

      This was working in cdh 5.5.1 using hadoop 2.6.0 without us opening up kms port on the destination cluster when we upgraded to 5.13.2 cdh we started facing this issue

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                rohit.peg Rohit Pegallapati
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated: