[HADOOP-17140] KMSClientProvider Sends HTTP GET with null "Content-Type" Header

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.3
• Component/s: kms
• Assignee: Unassigned
• Reporter: Axton Grams
• Created: 20/Jul/20 15:12
• Updated: 20/Jul/20 20:15

Description

Hive Server uses 'org.apache.hadoop.crypto.key.kms.KMSClientProvider' when interacting with HDFS TDE zones. This triggers a call to the KMS server. If the request method is a GET, the HTTP Header Content-Type is sent with a null value.

When using Ranger KMS, the embedded Tomcat server returns a HTTP 400 error with the following error message:

HTTP Status 400 - Bad Content-Type header value: ''
The request sent by the client was syntactically incorrect.

This only occurs with HTTP GET method calls. 

This is a captured HTTP request:

 

GET /kms/v1/key/xxx/_metadata?doAs=yyy&doAs=yyy HTTP/1.1
Cookie: hadoop.auth="u=hive&p=hive/[e-mail]&t=kerberos-dt&e=123789456&s=xxx="
Content-Type:
Cache-Control: no-cache
Pragma: no-cache
User-Agent: Java/1.8.0_241
Host: kms.domain.com:9292
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive

 

Note the empty 'Content-Type' header.

And the corresponding response:

 

HTTP/1.1 400 Bad Request
Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 1034
Date: Thu, 16 Jul 2020 04:23:18 GMT
Connection: close

 

This is the stack trace from the Hive server:

 

Caused by: java.io.IOException: HTTP status [400], message [Bad Request]
at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:169)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:608)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:597)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:566)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.getMetadata(KMSClientProvider.java:861)
at org.apache.hadoop.hive.shims.Hadoop23Shims$HdfsEncryptionShim.compareKeyStrength(Hadoop23Shims.java:1506)
at org.apache.hadoop.hive.shims.Hadoop23Shims$HdfsEncryptionShim.comparePathKeyStrength(Hadoop23Shims.java:1442)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.comparePathKeyStrength(SemanticAnalyzer.java:1990)
... 38 more

 

This looks to occur in https://github.com/hortonworks/hadoop-release/blob/HDP-2.6.5.165-3-tag/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java#L591-L599

      if (authRetryCount > 0) {
        String contentType = conn.getRequestProperty(CONTENT_TYPE);
        String requestMethod = conn.getRequestMethod();
        URL url = conn.getURL();
        conn = createConnection(url, requestMethod);
        conn.setRequestProperty(CONTENT_TYPE, contentType);
        return call(conn, jsonOutput, expectedResponse, klass,
            authRetryCount - 1);
      }

 I think when a GET method is received, the Content-Type header is not defined, then in line 592:

 String contentType = conn.getRequestProperty(CONTENT_TYPE);

The code attempts to retrieve the CONTENT_TYPE Request Property, which returns null.

Then in line 596:

conn.setRequestProperty(CONTENT_TYPE, contentType);

The null content type is used to construct the HTTP call to the KMS server.

A null Content-Type header is not allowed/considered malformed by the receiving KMS server.

I propose this code be updated to inspect the value returned by conn.getRequestProperty(CONTENT_TYPE), and not use a null value to construct the new KMS connection.

Proposed pseudo-patch:

--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
@@ -593,7 +593,9 @@ public HttpURLConnection run() throws Exception {
         String requestMethod = conn.getRequestMethod();
         URL url = conn.getURL();
         conn = createConnection(url, requestMethod);
-        conn.setRequestProperty(CONTENT_TYPE, contentType);
+        if (conn.getRequestProperty(CONTENT_TYPE) != null) {
+          conn.setRequestProperty(CONTENT_TYPE, contentType);
+        }
         return call(conn, jsonOutput, expectedResponse, klass,
             authRetryCount - 1);
       }

This should not impact any other use of this class and should only address cases where a null is returned for Content-Type.