Once validation compaction has been finished, all mismatching sstable sections for a token range will be used for streaming as return by StreamSession.getSSTableSectionsForRanges. Currently 2.1 will try to restrict the sstable candidates by checking if they can be found in CANONICAL_SSTABLES and will ignore them otherwise. At the same time IntervalTree in the DataTracker will be build based on replaced non-canonical sstables as well. In case of early opened sstables this becomes a problem, as the tree will be update with OpenReason.EARLY replacements that cannot be found in canonical. But whenever getSSTableSectionsForRanges will get a early instance from the view, it will fail to retrieve the corresponding canonical version from the map, as the different generation will cause a hashcode mismatch. Please find a test attached.
As a consequence not all sections for a range are streamed. In our case this has caused deleted data to reappear, as sections holding tombstones were left out due to this behavior.