[AMBARI-25880] For hadoop_logs collection of Infra-solr, indexed log_message field is too large to copy to key_log_message field

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 2.7.3
• Component/s: ambari-infra, ambari-logsearch
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: guluo
• Created: 01/Mar/23 07:49
• Updated: 01/Mar/23 08:04

Description

The hadoop_logs collection of Infra-solr is used to store logs generated by services, such as hadoop、spark， etc. 
However, Infra-solr will report an error when the length of the single-line log generated by services exceeds 2^15-2, Because Infra-solr cannot copy the data of field log_message to field key_log_messag

//代码占位符
org.apache.solr.common.SolrException: Exception writing document id 54d786cd-f6a7-4746-9b17-e9b2b0fe389a to the index; possible analysis error: Document contains at least one immense term in field="action" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98]...', original message: bytes can be at most 32766 in length; got 32767. Perhaps the document has an indexed string field (solr.StrField) which is too large 

Reproduce Step:

1 Create a json File named data.json
[
   

{         "log_message": "********",         "id": "54d786cd-f6a7-4746-9b17-e9b2b0fe3efa"     }

]
Note: the content of field log_message is a randomly generated string with a length exceeding 2^15-2

2 Update the hadoop_logs
curl http://ip:port/solr/hadoop_logs/update?commit=true -H 'Content-type:application/json' --data-binary @data.json

3 Get the error message mentioned above

 

similar issue：

AMBARI-24067: Log Search: indexed string field could be too large for audit_logs

 

Issue Links

• links to: GitHub Pull Request #93