Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6173

Document the configuration of deploying MR over distributed cache with enabling wired encryption at the same time

    XMLWordPrintableJSON

    Details

    • Target Version/s:

      Description

      Use the current documented configuration (specified in http://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html) to work with the cluster enabling shuffle encryption (http://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html) will cause the job failed with exception below:

      2014-10-10 02:17:16,600 WARN [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to tassapol-centos5nano1-3.cs1cloud.internal:13562 with 1 map outputs
      javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      	at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
      	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1731)
      	at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:241)
      	at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:235)
      	at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1206)
      	at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:136)
      	at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
      	at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
      	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:925)
      	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170)
      	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1197)
      	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1181)
      	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:434)
      	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.setNewClient(AbstractDelegateHttpsURLConnection.java:81)
      	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.setNewClient(AbstractDelegateHttpsURLConnection.java:61)
      	at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:584)
      	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1193)
      	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
      	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:318)
      	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:427)
      

      This is due to ssl-client.xml is not included in MR tar ball when we deploy it over distributed cache. Putting the ssl-client.xml on CLASSPATH of MR job can resolve the problem and we should document it.

        Attachments

        1. MAPREDUCE-6173-v2.patch
          4 kB
          Junping Du
        2. MAPREDUCE-6173.patch
          4 kB
          Junping Du
        3. Screen Shot for MAPREDUCE-6173.png
          1.20 MB
          Junping Du

          Issue Links

            Activity

              People

              • Assignee:
                djp Junping Du
                Reporter:
                djp Junping Du
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: