Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2862

More flexible logging in multilang (Python, Ruby, JS)

    Details

      Description

      We're running a Storm topology written in Python, using storm.py from storm-multilang. As well as human-readable logs, the topology is also configured to write JSON logs which are sent to ELK.

      At the moment, when storm-core receives a "log" command, it outputs the pid, component name, and the message it received, like so:

      ShellLog pid:<pid>, name:<component name> <message>

      The code that does this is (currently) in ShellBolt line 254 and ShellSpout line 227.

      As well as the pid and component name, it would be great to have the task ID, tuple ID, and the ID of the originating tuple - but this would make parsing the string even more laborious than it is now, and would make the default log message too long.

      Would it be possible to put contextual information like this in the ThreadContext instead? Then our JSON layout could read from the context instead of parsing the string, and human-readable logs could use "%mdc" in the PatternLayout format string.

        Attachments

          Activity

            People

            • Assignee:
              hmcc Heather McCartney
              Reporter:
              hmcc Heather McCartney
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 5h 10m
                5h 10m