[NIFI-7388] Bug reports about VolatileContentRepository and VolatileFlowFileRepository

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.11.4
• Component/s: Core Framework
• Environment: jdk8_141 ; docker: 18.04.0-ce ; Kubernetes: v1.13.3
• Assignee: Unassigned
• Reporter: zhangxinchen
• Created: 23/Apr/20 02:25
• Updated: 29/Mar/22 11:18
• Resolved: 29/Mar/22 11:18

Description

1. VolatileContentRepository
when maxSize = 100MB and blockSize = 2KB, there should be 51200 "slots". If
we write one kb by one kb, 102400 one kb should be written in, but when
writing 51201th one kb, "java.io.IOException: Content Repository is out of
space" occurs. Here's the Junit Test I write.

@Test
public void test() throws IOException {
System.setProperty(NiFiProperties.PROPERTIES_FILE_PATH,
TestVolatileContentRepository.class.getResource("/conf/nifi.properties").getFile());
final Map<String, String> addProps = new HashMap<>();
addProps.put(VolatileContentRepository.BLOCK_SIZE_PROPERTY, "2 KB");
final NiFiProperties nifiProps =
NiFiProperties.createBasicNiFiProperties(null, addProps);
final VolatileContentRepository contentRepo = new
VolatileContentRepository(nifiProps);
contentRepo.initialize(claimManager);
// can write 100 * 1024 /1 = 102400, but after 51201, blocks exhausted
for (int idx =0; idx < 51201; ++idx) {
final ContentClaim claim = contentRepo.create(true);
try (final OutputStream out = contentRepo.write(claim))

{ final byte[] oneK = new byte[1024]; Arrays.fill(oneK, (byte) 55); out.write(oneK); }

}
}

2. VolatileFlowFileRepository
When the backpressure occurs, FileSystemSwapManager will swap out FlowFiles
to disk whenever swapQueue size exceeds 10000, there's no problem in
swap-out process BUT in swap-in process, VolatileFlowFileRepository does
not "acknowledge" the FlowFiles which has been swap out when
FileSystemSwapManager swaps in FlowFiles from the disk and logs the warning
information "Cannot swap in FlowFiles from location..." because the
implementation of "isValidSwapLocationSuffix" in VolatileFlowFileRepository
is always FALSE.
And the queue is still like FULL when checking the NiFi frontend, the
upstream processor is STUCKED, maybe FileSystemSwapManager "thinks" these
FlowFiles are still not consumed.

Issue Links

• is superceded by: NIFI-9823 Remove VolatileContentRepository