[OPENNLP-1556] Improve speed of checksum computation in TwoPassDataIndexer

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.9.0, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.3.1, 2.3.2, 2.3.3
• Fix Version/s: 2.4.0
• Component/s: Machine Learning
• Assignee: Martin Wiesner
• Reporter: Martin Wiesner
• Created: 05/May/24 07:35
• Updated: 07/May/24 06:51
• Resolved: 07/May/24 06:51

Description

For training ML models, all observations (Events) are indexed via 
TwoPassDataIndexer#index(ObjectStream<Event> eventStream).

When #index(..) is run, a tmp file is written and read in again. For the purpose of checksum validation, instances of HashSumEventStream are used to validate the content processed.

Based on a rather slow toString() implementation in Event, a cryptographic (MD5) message digest is computed. This, however, is much slower than simply computing a checksum (such as a CRC32c value) for both directions (write/read). The (slowing) effect is more problematic when larger training corpora are (pre-)processed, that is, indexed in advance.

Aims:

• Speedup the (IO-bound) indexing part prior to the actual CPU-bound training phase.
• Switch from MD5 to CRC32c, as there is no need for a cryptographic hash function here; it's simply a checksum that is required to decide whether all bytes written are the same bytes that are read.
• Remove the untested class HashSumEventStream which is just a wrapper for calling a slow toString() in Event to get some bytes to use for the computation of a checksum / md.
• Provide a replacement for HashSumEventStream, e.g. ChecksumEventStream that makes use of the faster CRC32c checksum computation, avoiding cryptographic hash functions such as MD5.
• Make sure all existing tests hold.