Details
-
Improvement
-
Status: To Do
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
Hi,
Currently, all the messages stored in ES have the original_string. Keeping the original string and all the parse data might conduct to store up to 2 times more data. This implied that the needed infrastructure to handle this will have to be much more important, so a bigger cost.
The user might be interested to keep it for legal reason or to be able to do text search on it but he might also decide to remove it because it has no added value and cost space.
Moreover the original_string will remain in HDFS in case they need it, so no lost of data.
The goal of this request is:
- to have a check box (or another UI interface) in the configuration of the sensor specifying if the original string has to be ingested in ES.
- Modifying the indexing storm topology to remove that field before saving the data to ES