This issue happens when we try to do streaming ingest with dynamic partition on already existing partitions. I checked in the code, we have following check in the AbstractRecordWriter.
Above addedPartitions is passed to addDynamicPartitions during TransactionBatch commit. So in case of already existing partitions, addedPartitions will be empty and addDynamicPartitions **will not move entries from TXN_COMPONENTS to COMPLETED_TXN_COMPONENTS. This results in Initiator not able to trigger auto compaction.
Another issue which has been observed is, we are not clearing addedPartitions on writer close, which results in information flowing across transactions.