Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-3157

Setup truststore so that it also works in HTTP only mode

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: trunk, 5.0.0b1
    • Fix Version/s: 5.0.0
    • Component/s: None
    • Labels:
      None

      Description

      oozie.https.truststore.file is not read and used when oozie.https.enabled is false in oozie-site xml. As a result, the Oozie server will be unable to communicate with servers with unsigned certificate. It is a critical problem as authentication may involve external servers (for example KMS with self-signed certificate). Submitting a workflow in such an environment can result in an exception like:

      2018-01-08 10:13:51,471 WARN org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider: SERVER[myserver] KMS provider at [https://myserver:16000/kms/v1/] threw an IOException: 
      javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to
       find valid certification path to requested target
              at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
              at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959)
              at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
              at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
              at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)
              at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
              at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
              at sun.security.ssl.Handshaker.process_record(Handshaker.java:961)
              at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
              at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
              at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
              at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
              at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
              at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
              at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
              at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:186)
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:144)
              at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
              at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333)
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:477)
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:472)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:471)
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776)
              at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:287)
              at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:283)
              at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:123)
              at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:283)
              at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
              at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:926)
              at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:945)
              at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:315)
              at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:310)
              at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
              at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:322)
              at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:949)
              at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:392)
              at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:263)
              at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:99)
      

      Background

      • When EmbeddedOozieServer is created it checks if it is configured with HTTPS. If so it creates an sslConnector that sets up keystore and truststore via SSLServerConnectorFactory using Jetty's SslContextFactory.
      • If we are not using HTTPS, truststore specified in Oozie configuration is ignored. As a workaround we can pass javax.net.ssl.trustStore and javax.net.ssl.trustStorePassword via system properties upon JVM startup via the JETTY_OPTS environment variable.

      A possible solution

      • Set system property javax.net.ssl.trustStore and javax.net.ssl.trustStorePassword with System.setProperty() using Oozie configuration properties if they are not set by a user.

        Attachments

        1. OOZIE-3157-007.patch
          13 kB
          Julia Kinga Marton
        2. OOZIE-3157-006.patch
          13 kB
          Julia Kinga Marton
        3. OOZIE-3157-005.patch
          13 kB
          Julia Kinga Marton
        4. OOZIE-3157-004.patch
          16 kB
          Julia Kinga Marton
        5. OOZIE-3157-003.patch
          16 kB
          Julia Kinga Marton
        6. OOZIE-3157-002.patch
          16 kB
          Julia Kinga Marton
        7. OOZIE-3157-001.patch
          9 kB
          Julia Kinga Marton

          Issue Links

            Activity

              People

              • Assignee:
                kmarton Julia Kinga Marton
                Reporter:
                asasvari Attila Sasvari
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: