[NIFI-6999] Encrypt Config Toolkit fails on very large flow.xml.gz files

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.2.0, 1.10.0
• Fix Version/s: 1.13.0
• Component/s: Tools and Build
• Labels: documentation encryption heap security streaming toolkit
• Assignee: Nathan Gough
• Reporter: Andy LoPresto
• Created: 09/Jan/20 17:50
• Updated: 28/Sep/21 16:20
• Resolved: 24/Mar/21 14:40

Description

A user reported failure when using the encrypt config toolkit to process (encrypt) a large flow.xml.gz. The compressed file was 49 MB, but was 687 MB uncompressed. It contained 545 encrypted values, and approximately 90 templates. This caused the toolkit to fail during loadFlowXml() unless the toolkit invocation set the heap to 8 GB via -Xms2g -Xmx8g. Even with the expanded heap, the serialization of the newly-encrypted flow XML to the file system fails with the following exception:

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.lang.StringCoding.encode(StringCoding.java:350)
at java.lang.String.getBytes(String.java:941)
at org.apache.commons.io.IOUtils.write(IOUtils.java:1857)
at org.apache.commons.io.IOUtils$write$0.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)
at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure5$_closure20.doCall(ConfigEncryptionTool.groovy:692)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
at groovy.lang.Closure.call(Closure.java:426)
at groovy.lang.Closure.call(Closure.java:442)
at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1754)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure5.doCall(ConfigEncryptionTool.groovy:691)

The immediate fix was to remove the duplicated template definitions in the flow definition, returning the file to a reasonable size. However, if run as an inline replacement, this can cause the flow.xml.gz to be overwritten with an empty file, potentially leading to data loss. The following steps should be taken:

1. Guard against loading/operating on/serializing large files (log statements, simple conditional checks)
2. Handle large files internally (change from direct String access to BufferedInputStream, etc.)
3. Document the internal memory usage of the toolkit in the toolkit guide
4. Document best practices and steps to resolve issue in the toolkit guide

Issue Links

• fixes: NIFI-7000 Document EC toolkit memory usage and best practices
• fixes: NIFI-7001 Guard against loading/operating on/serializing large files in EC toolkit
• fixes: NIFI-7002 Handle large files internally in EC toolkit
• fixes: NIFI-4369 ConfigEncryptionTool should stream flow contents
• links to: GitHub Pull Request #4715

Sub-Tasks

• Document EC toolkit memory usage and best practices: Resolved
• Guard against loading/operating on/serializing large files in EC toolkit: Resolved
• Handle large files internally in EC toolkit: Resolved