[NIFI-10235] Provenance replay fails when repository encryption is enabled

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16.3
• Fix Version/s: 2.0.0-M1, 1.23.0
• Component/s: Core Framework, Security
• Labels: encryption provenance replay
• Environment: RHEL 8.5
• Assignee: David Handermann
• Reporter: Peter Kimberley
• Created: 15/Jul/22 06:59
• Updated: 28/Jul/23 18:29
• Resolved: 28/Jul/23 18:29

Description

Problem summary

When repository encryption is enabled, replaying a DROP provenance record fails, with the following error appearing in the logs:

org.apache.nifi.processor.exception.FlowFileAccessException: Failed to export StandardFlowFileRecord[uuid=df985fc5-23da-4094-8783-2e0186bcb92d,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1657864218374-23, container=default, section=23], offset=379, length=1048576],offset=0,name=b29633c4-324e-42fe-b3e8-1ea455fc3650,size=1048576] to /opt/nifi/nifi-current/data/store/.b29633c4-324e-42fe-b3e8-1ea455fc3650 due to java.io.EOFException: Attempted to copy 1048576 bytes but only 1048197 bytes were available

 
I've observed that the difference between the sizes mentioned in the log is always 379 bytes, regardless of the length of the input file.
 
With repository encryption disabled, provenance replay works as expected.

Configuration

1. NiFi v1.16.3 running as a three-node cluster in Kubernetes.
2. Each node has up to 8GB memory and 4 CPUs available to it.
3. Testing has included both NFS and ephemeral (emptyDir) storage.
4. The encryption key was generated by the following command, using the same JDK version:
   1. keytool -genseckey -alias key-1 -keyalg AES -keysize 256 -keystore repository.p12 -storetype PKCS12

nifi.properties

nifi.repository.encryption.protocol.version=1
nifi.repository.encryption.key.id=key-1
nifi.repository.encryption.key.provider=KEYSTORE
nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12
nifi.repository.encryption.key.provider.keystore.password=<password>

Processor group

GenerateFlowFile processor generating 1MB random files every second to a PutFile processor. Have also tested with InvokeHTTP.

Other comments

With repository encryption enabled, I am able to download files via the provenance UI (suggesting that encryption/decryption works). The processor group also performs all other actions as expected.

Not having the ability to replay provenance records is a blocker for our deployment, which requires data to be encrypted at rest and in transit.

Issue Links

• is fixed by: NIFI-11670 Encrypted Content Repository is very slow when FlowFiles have a non-zero Content Claim Offset
• links to: GitHub Pull Request #6228