[NIFI-6920] InvokeHttp convertAttributesFromHeaders() method creates attributes with empty string keys, leading to failure on validation

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.9.2, 1.12.0
• Fix Version/s: 1.14.0
• Component/s: Core Framework
• Assignee: David Handermann
• Reporter: Andrew Greenburg
• Created: 02/Dec/19 19:44
• Updated: 17/Mar/22 01:34
• Resolved: 17/Mar/22 01:34

Description

When the InvokeHttp processor receives a response, it automatically adds all of the HTTP response headers to the response flowfile as attributes.

See line 854 of https://github.com/apache/nifi/blob/rel/nifi-1.9.2/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java :

// write the response headers as attributes
// this will overwrite any existing flowfile attributes
responseFlowFile = session.putAllAttributes(responseFlowFile, convertAttributesFromHeaders(url, responseHttp));

 
In some cases, the Response contains headers where the key is an empty string, but the convertAttributesFromHeaders() method only has a safety check for keys that are null:

    private Map<String, String> convertAttributesFromHeaders(URL url, Response responseHttp){
        // create a new hashmap to store the values from the connection
        Map<String, String> map = new HashMap<>();
        responseHttp.headers().names().forEach( (key) -> {
                if (key == null) {
                    return;
                }

                List<String> values = responseHttp.headers().values(key);

                // we ignore any headers with no actual values (rare)
                if (values == null || values.isEmpty()) {
                    return;
                }

                // create a comma separated string from the values, this is stored in the map
                String value = csv(values);

                // put the csv into the map
                map.put(key, value);
        });

        if (responseHttp.request().isHttps()) {
            Principal principal = responseHttp.handshake().peerPrincipal();

            if (principal != null) {
                map.put(REMOTE_DN, principal.getName());
            }
        }

        return map;
    }

This causes the following error in Nifi, which routes the flowfile to failure:

2019-12-02 19:18:36,416 ERROR [Timer-Driven Process Thread-2] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=e127afdd-12b8-38cc-a2d3-af051d3965cb] Routing to Failure due to exception: java.lang.IllegalArgumentException: Invalid attribute key: <Empty String>: java.lang.IllegalArgumentException: Invalid attribute key: <Empty String>
java.lang.IllegalArgumentException: Invalid attribute key: <Empty String>
	at org.apache.nifi.flowfile.FlowFile$KeyValidator.validateKey(FlowFile.java:120)
	at org.apache.nifi.controller.repository.StandardFlowFileRecord$Builder.addAttributes(StandardFlowFileRecord.java:238)
	at org.apache.nifi.controller.repository.StandardProcessSession.putAllAttributes(StandardProcessSession.java:1796)
	at org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:854)
	at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
	at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
	at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
	at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
	at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Here is an HTTP response that I can consistently reproduce this issue with:

HTTP/1.1 200 OK
Server: openresty/1.15.8.1
Date: Wed, 27 Nov 2019 15:07:20 GMT
Content-Type: application/json;charset=UTF-8
Content-Length: 190
Connection: keep-alive
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1 ; mode=block
Referrer-Policy: no-referrer
set-cookie: access_token=vmpxF4JrHGOeRvoSB; Max-Age=1209600; Expires=Wed, 11 Dec 2019 15:07:20 GMT; Path=/
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Security-Policy: script-src https://10.0.0.196:* 'unsafe-inline' 'unsafe-eval' blob: ;
                                         img-src https://10.0.0.196:* data: ;
                                       frame-src https://10.0.0.196:* 'unsafe-inline' 'unsafe-eval' blob: ;
                                       style-src https://10.0.0.196:* 'unsafe-inline' ;
                                        font-src https://10.0.0.196:* data: ;
                                     default-src https://10.0.0.196:* ;
                                      object-src 'none' 

Issue Links

• is fixed by: NIFI-8304 Improve reliability of TLS unit tests
• links to: GitHub Pull Request #4517