[IGNITE-5473] Create ignite troubleshooting logger

• Type: Improvement
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 2.0
• Component/s: general
• Labels: important observability
• Assignee: Unassigned
• Reporter: Alexey Goncharuk
• Created: 13/Jun/17 07:39
• Updated: 29/Dec/20 08:57

Description

Currently, we have two extremes of logging - either INFO wich logs almost nothing, or DEBUG, which will pollute logs with too verbose messages.

We should create a 'troubleshooting' logger, which should be easily enabled (via a system property, for example) and log all stability-critical node and cluster events:

• Connection events (both communication and discovery), handshake status
• ALL ignored messages and skipped actions (even those we assume are safe to ignore)
• Partition exchange stages and timings
• Verbose discovery state changes (this should make it easy to understand the reason for 'Node has not been connected to the topology')
• Transaction failover stages and actions
• All unlogged exceptions
• Responses that took more than N milliseconds when in normal they should return right away
• Long discovery SPI messages processing times
• Managed service deployment stages
• Marshaller mappings registration and notification
• Binary metadata registration and notification
• Continuous query registration / notification
  (add more)

The amount of logging should be chosen accurately so that it would be safe to enable this logger in production clusters.

Issue Links

• contains: IGNITE-7195 GridToStringBuilder should limit large collections output to first N elements
• is related to: IGNITE-5630 Exception for node disconnect
• is related to: IGNITE-5664 Implement self-check facilities for Ignite
• supercedes: IGNITE-5332 Add toString() to GridNearAtomicAbstractSingleUpdateRequest and it's inheritors
• links to: GitHub Pull Request #2262