[STORM-1342] support multiple logviewers per host for container-isolated worker logs

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: storm-core
• Assignee: Unassigned
• Reporter: Erik Weathers
• Created: 24/Nov/15 03:59
• Updated: 01/Jul/17 21:37

Description

Storm-on-Mesos Worker Logs are in varying directories

When using storm-on-mesos with cgroups, each topology's workers are isolated into separate containers. By default the worker logs will be saved into container-specific sandbox directories. These directories are also topology-specific by definition, because, as just stated, the containers are specific to each topology.

Problem: Storm supports 1-and-only-1 Logviewer per Worker Host

A challenge with this different way of running Storm is that the Storm logviewer runs as a single instance on each worker host. This doesn't play well with having the topology worker logs in separate per-topology containers. The one logviewer doesn't know about the various sandbox directories that the Storm Workers are writing to. And if we just spawned new logviewers for each container, the problem is that the Storm UI only knows about 1 global port the logviewer, so you cannot just direct.

These problems are documented (or linked to) from Issue #6 in the storm-on-mesos project

Possible Solutions I can envision

1. configure the Storm workers to write to log directories that exist on the raw host outside of the container sandbox, and run a single logviewer on a host, which serves up the contents of that directory.
   • violates one of the basic reasons for using containers: isolation.
   • also prevents a standard use case for Mesos: running more than 1 instance of a Mesos Framework (e.g., "Storm Cluster") at once on same Mesos Cluster. e.g., for Blue-Green deployments.
   • a variation on this proposal is to somehow expose the sandbox dirs of all storm containers to this singleton logviewer process (still has above problems)
2. launch a separate logviewer in each container, and somehow register those logviewers with Storm such that Storm knows for a given host which logviewer port is assigned to a given topology.
   • this is the proposed solution
3. launch a separate logviewer in a separate container, somehow exposing the logs from these other containers to the host-global logviewer.

Storm Changes for the Proposed Solution

Nimbus or ZooKeeper could serve as a registrar, recording the association between a slot (host + worker port) and the logviewer port that is serving the workers logs. And the Storm-on-Mesos framework could update this registry when launching a new worker. (This proposal definitely calls for thorough vetting and thinking.)

Storm-on-Mesos Framework Changes for the Proposed Solution

Along with the interaction with the "registrar" proposed above, the storm-on-mesos framework can be enhanced to launch multiple logviewers on a given worker host, where each logviewer is dedicated to serving the worker logs from a specific topology's container/sandbox directory. This would be done by launching a logviewer process within the topology's container, and assigning it an arbitrary listening port that has been determined dynamically through mesos (which treats ports as one of the schedulable resource primitives of a worker host). Code implementing this logviewer-port-allocation logic already exists, but that specific portion of the code was reverted because of the issues that inspired this ticket.

Issue Links

• is related to: STORM-1758 Distributed log search doesn't work in dockerized environment
• is related to: STORM-1759 Viewing logs from the Storm UI doesn't work in dockerized environment
• is related to: STORM-1494 Storm UI - Add link to supervisor log