Details
-
New Feature
-
Status: To Do
-
Minor
-
Resolution: Unresolved
-
None
-
None
-
None
Description
We are facing certain use cases in Metron production that happen to be related to noisy stream. For example, a wrong timestamp, duplicate hostname/IP address, etc. To deal with the normalization, we have added an additional step for the corresponding parsers to do the data cleaning. Clearly, parsing is a standard factor which is mostly related to the device that is generating the data and can be used for the same type of device everywhere, but normalization is very production dependent and there is no point of mixing normalization with parsing. It would be nice to have a sperate bolt in a parsing topologies to dedicate to production related cleaning process. In that case, eveybody can easily contribute to Metron community with additional parsers without being worried about mixing parsers and data cleaning process.