When we flush a memory component of an index, we would set an LSN to the result disk component. The LSN is set as the last operation which modifies that memory component. Thus, given an index, the FLUSH_LSNs of its flushed disk components should be increasing, i.e., later flushed components get larger LSNs.
However, currently I observed a bug that later flushed disk components get a smaller LSN, which breaks this properly.
A brief explanation of this bug is as follows. Suppose we have one dataset D, and two partitions on one NC. Suppose D only has a primary index P. Further suppose P1 and P2 are two partitioned indexes for the two partitions on this NC. This implies P1 and P2 share the same PrimaryIndexOperationTracker, and they would always be flushed together.
The LSN for flushed disk components is ILSMIOOperationCallback. Now suppose an index has two memory components. ILSMIOOperationCallback maintains an array mutableLastLSNs of length 2 to track the FLUSH_LSN for two memory components. Before scheduling each flush operation, PrimaryIndexOperationTracker needs to call ILSMIOOperationCallback to set the FLUSH_LSN.
Now consider the following scenario (which happens but very rarely). Initially,
Suppose dataset D needs to be flushed, and mutableLastLSNs is set as follows:
Then, suppose the flush operation of P2 is fast, and produces a disk component P2.d1 (P2.d1.LSN = 1). Data continues to come into P2, and it needs to be flushed again. However, P1 is still flushing the first memory component. Then mutableLastLSNs become:
Surprisingly, the flush operation of P2 is again fast, and produces a disk component P2.d2 (P2.d2.LSN = 2). Still, data continues to come into P2, and it needs to be flushed again. But P1 is still flushing the first memory component. Then mutableLastLSNs become:
At this time, P1 finishes its first flush operation, and produced the disk component P1.d1 (P1.d1.LSN = 3). This in incorrect, since P1.d1.LSN should be 1, not 3. Its original value is overwritten by the flush request of P2!
To reproduce this bug, one needs to change the codebase slightly (i.e., LSMBTreeIOOperationCallback). I added a member variable
and added one check in the getComponentLSN method:
Then, after starting AsterixDB using AsterixHyracksIntegrationUtil, you can ingest the data and reproduce the bug using the following queries (you need to replace the path_to_sample_data with the attached file):
I attached the asterix-build-configuration-lsm.xml file and the sample data file as below.