Uploaded image for project: 'Apache Knox'
  1. Apache Knox
  2. KNOX-762

Remove dependency on httpcomponents httpclient 4.5.2

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Server
    • Labels:
      None

      Description

      Reported by Benjamin Ruland:

      I am experiencing problems with Knox while using WebHDFS in a cluster with Kerberos and SSL.
      The KDC is an Microsoft AD 2012. Kerberos-Encryption is set to AES256. Knox is connected to AD via LDAP sync (this is working fine for other Knox services).
      I am running HDP 2.5 with Knox 0.9.0

      In general, the cluster runs fine. WebHDFS using SPNEGO is working.

      But when accessing WebHDFS over Knox, I get an 401 error and some strange logs.
      I suspect that Knox is trying to get a ticket for a HTTPS/namenode@REALM principal, which does not exist. Although running SSL, all principals for SPNEGO are HTTP/...

      I this a Knox Bug or is this a misconfiguration at some point?

      It would be great, if someone has advice.

      Best regards,
      Benjamin

      The used command is:

      [root@utilitynode ~]# curl -ik -u validuser "https://utilitynode:8443/gateway/default/webhdfs/v1/?OP=LISTSTATUS"
      Enter host password for user 'validuser':
      HTTP/1.1 401 Unauthorized
      Date: Wed, 12 Oct 2016 07:47:41 GMT
      Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Tue,11-Oct-2016 07:47:41 GMT
      WWW-Authenticate: BASIC realm="application"
      Content-Length: 0
      Server: Jetty(9.2.15.v20160210)

      Debug Log in knox gateway.log

      2016-10-12 09:51:49,735 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /webhdfs/v1/
      2016-10-12 09:51:49,740 DEBUG hadoop.gateway (KnoxLdapRealm.java:getUserDn(673)) - Searching from OU=someOU,DC=somedomain,DC=de where (&(objectclass=person)(sAMAccountName=validuser)) scope subtree
      2016-10-12 09:51:49,745 INFO hadoop.gateway (KnoxLdapRealm.java:getUserDn(679)) - Computed userDn: CN=validuser,OU=Users,OU=someOU,DC=somedomain,DC=de using ldapSearch for principal: validuser
      2016-10-12 09:51:49,749 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://utilitynode:8443/gateway/default/webhdfs/v1/?OP=LISTSTATUS, direction: IN via explicit rule: WEBHDFS/webhdfs/inbound/namenode/root to URL: https://utilitynode.somedomain.de:50470/webhdfs/v1/?OP=LISTSTATUS
      2016-10-12 09:51:49,749 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(120)) - Dispatch request: GET https://utilitynode.somedomain.de:50470/webhdfs/v1/?OP=LISTSTATUS&doAs=validuser
      2016-10-12 09:51:49,781 WARN auth.HttpAuthenticator (HttpAuthenticator.java:generateAuthResponse(207)) - NEGOTIATE authentication error: No valid credentials provided (Mechanism level: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7)))
      2016-10-12 09:51:49,782 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(133)) - Dispatch response status: 401
      2016-10-12 09:51:49,783 DEBUG hadoop.gateway (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit character set ISO-8859-1 for entity of type text/html
      2016-10-12 09:51:49,783 DEBUG hadoop.gateway (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound response entity content type: text/html; charset=iso-8859-1

      Log in knox gateway.out

      Found ticket for knox/utilitynode.somedomain.de@SOMEDOMAIN.DE to go to krbtgt/somedomain.de@SOMEDOMAIN.DE expiring on Wed Oct 12 19:53:51 CEST 2016
      Entered Krb5Context.initSecContext with state=STATE_NEW
      Service ticket not found in the subject
      >>> Credentials acquireServiceCreds: same realm
      default etypes for default_tgs_enctypes: 18.
      >>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
      >>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
      getKDCFromDNS using UDP
      >>> KrbKdcReq send: kdc=domaincontroller.somedomain.de. TCP:88, timeout=30000, number of retries =3, #bytes=1661
      >>> KDCCommunication: kdc=domaincontroller.somedomain.de. TCP:88, timeout=30000,Attempt =1, #bytes=1661
      >>>DEBUG: TCPClient reading 127 bytes
      >>> KrbKdcReq send: #bytes read=127
      >>> KdcAccessibility: remove domaincontroller.somedomain.de.:88
      >>> KDCRep: init() encoding tag is 126 req type is 13
      >>>KRBError:
      sTime is Wed Oct 12 09:53:51 CEST 2016 1476258831000
      suSec is 8354 suSec is 8354
      error code is 7
      error Message is Server not found in Kerberos database
      sname is HTTPS/namenode.somedomain.de@SOMEDOMAIN.DE
      msgType is 30

      Extracts from topology config:

      <topology>

      <gateway>

      <provider>
      <role>authentication</role>
      <name>ShiroProvider</name>
      <enabled>true</enabled>

      <!-- LDAP Sync properties sit here -->

      <provider>
      <role>identity-assertion</role>
      <name>Default</name>
      <enabled>true</enabled>
      </provider>

      <provider>
      <role>authorization</role>
      <name>XASecurePDPKnox</name>
      <enabled>true</enabled>
      </provider>

      <provider>
      <role>ha</role>
      <name>HaProvider</name>
      <enabled>true</enabled>
      <param>
      <name>WEBHDFS</name>
      <value>maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;enabled=true</value>
      </param>
      </provider>

      </gateway>

      <service>
      <role>NAMENODE</role>
      <url>hdfs://namenode.somedomain.de:8020</url>
      <url>hdfs://namenode2.somedomain.de:8020</url>
      </service>

      <service>
      <role>WEBHDFS</role>
      <url>https://namenode.somedomain.de:50470/webhdfs</url>
      <url>https://namenode2.somedomain.de:50470/webhdfs</url>
      </service>

      </topology>

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                lmccay Larry McCay
                Reporter:
                lmccay Larry McCay
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: