Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-6003

Add logging module for logging to an external program

    Details

    • Type: Improvement
    • Status: Reviewable
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: modules
    • Labels:
      None

      Description

      In the vein of the logrotate module for logging, there should be a similar module which provides support for logging to an arbitrary log handling program, with suitable task metadata provided by environment variables or command line arguments.

        Activity

        Hide
        wrouesnel Will Rouesnel added a comment -

        Joel Wilsson From a prettiness perspective I agree, but I wanted to preserve the ability to find MESOS_LOG injected env vars, and just specifying "executorinfo" didn't seem descriptive enough. Since you almost always need to post-process that field in some way, the user script/program is pretty likely to want to do something with it.

        Show
        wrouesnel Will Rouesnel added a comment - Joel Wilsson From a prettiness perspective I agree, but I wanted to preserve the ability to find MESOS_LOG injected env vars, and just specifying "executorinfo" didn't seem descriptive enough. Since you almost always need to post-process that field in some way, the user script/program is pretty likely to want to do something with it.
        Hide
        r4um Pranay Kanwar added a comment -

        Might run into MESOS-7386 if using docker and logger process dies

        Show
        r4um Pranay Kanwar added a comment - Might run into MESOS-7386 if using docker and logger process dies
        Hide
        wjoel Joel Wilsson added a comment -

        I guess this needs another look from Joseph Wu? It would be nice to have this feature merged.

        Show
        wjoel Joel Wilsson added a comment - I guess this needs another look from Joseph Wu ? It would be nice to have this feature merged.
        Hide
        r4um Pranay Kanwar added a comment -
        Show
        r4um Pranay Kanwar added a comment - 0.28.2 port in case anyone else needs it https://github.com/r4um/mesos-external-container-logger/tree/mesos-0.28.2-ubuntu-14.04.5
        Hide
        wjoel Joel Wilsson added a comment -

        Thanks, Will Rouesnel.

        I've updated my backport. I think I found one issue: executor_info_json_field should have a default value of "EXECUTORINFO_JSON" instead of "MESOS_EXECUTORINFO_JSON", since (as the help message explains) it's concatenated with the value of --mesos_field_prefix. Otherwise it gets turned into MESOS_LOG_MESOS_EXECUTORINFO_JSON.

        Show
        wjoel Joel Wilsson added a comment - Thanks, Will Rouesnel . I've updated my backport. I think I found one issue: executor_info_json_field should have a default value of "EXECUTORINFO_JSON" instead of "MESOS_EXECUTORINFO_JSON", since (as the help message explains) it's concatenated with the value of --mesos_field_prefix. Otherwise it gets turned into MESOS_LOG_MESOS_EXECUTORINFO_JSON.
        Hide
        wrouesnel Will Rouesnel added a comment -

        Huh I thought I had notified this thread - anyway - I've rebased the patchset for the changes that have been made to ContainerLogger interface, so this should be good to go again.

        Show
        wrouesnel Will Rouesnel added a comment - Huh I thought I had notified this thread - anyway - I've rebased the patchset for the changes that have been made to ContainerLogger interface, so this should be good to go again.
        Hide
        r4um Pranay Kanwar added a comment -

        Yes that would do, thanks a lot Joel Wilsson.

        Show
        r4um Pranay Kanwar added a comment - Yes that would do, thanks a lot Joel Wilsson .
        Hide
        wjoel Joel Wilsson added a comment -

        Pranay Kanwar I have back port of this as a standalone module here, if that's what you mean: https://github.com/wjoel/mesos-external-container-logger
        I'll update it with the latest changes from Will later this week.

        Show
        wjoel Joel Wilsson added a comment - Pranay Kanwar I have back port of this as a standalone module here, if that's what you mean: https://github.com/wjoel/mesos-external-container-logger I'll update it with the latest changes from Will later this week.
        Hide
        r4um Pranay Kanwar added a comment -

        Can this also be available for other/older versions in a separate repo like https://github.com/dcos/dcos-mesos-modules/tree/master/journald ?

        Show
        r4um Pranay Kanwar added a comment - Can this also be available for other/older versions in a separate repo like https://github.com/dcos/dcos-mesos-modules/tree/master/journald ?
        Hide
        wrouesnel Will Rouesnel added a comment -

        Rebased and building on master now, with added support for the new USER variable if someone could review.

        Show
        wrouesnel Will Rouesnel added a comment - Rebased and building on master now, with added support for the new USER variable if someone could review.
        Hide
        wjoel Joel Wilsson added a comment -

        Joseph Wu, I use it to do log forwarding with Filebeat and we're about to implement the same setup at work. It would be great if I didn't need to maintain and update a custom compiled module with this code for each new release.

        Will Rouesnel, that would be fantastic, thanks! This is a very useful feature, especially because it enables the combination of log forwarding and log rotation.

        Show
        wjoel Joel Wilsson added a comment - Joseph Wu , I use it to do log forwarding with Filebeat and we're about to implement the same setup at work. It would be great if I didn't need to maintain and update a custom compiled module with this code for each new release. Will Rouesnel , that would be fantastic, thanks! This is a very useful feature, especially because it enables the combination of log forwarding and log rotation.
        Hide
        wrouesnel Will Rouesnel added a comment -

        I can rebase to whatever we're up to now.

        Show
        wrouesnel Will Rouesnel added a comment - I can rebase to whatever we're up to now.
        Hide
        kaysoky Joseph Wu added a comment -

        Sorry, this dropped off my radar a while ago.

        The ContainerLogger code is subtly different now. If you want to revive the reviews, you can post new versions of them, rebased to master.

        (Just curious: what do you intend to use a module like this for?)

        Show
        kaysoky Joseph Wu added a comment - Sorry, this dropped off my radar a while ago. The ContainerLogger code is subtly different now. If you want to revive the reviews, you can post new versions of them, rebased to master. (Just curious: what do you intend to use a module like this for?)
        Hide
        wjoel Joel Wilsson added a comment -

        Will Rouesnel, Joseph Wu, is there anything I can do to help this move along?

        Show
        wjoel Joel Wilsson added a comment - Will Rouesnel , Joseph Wu , is there anything I can do to help this move along?
        Hide
        wjoel Joel Wilsson added a comment -

        Looks good to me as an outsider, but most of the code changes ended up in the documentation review request.

        Show
        wjoel Joel Wilsson added a comment - Looks good to me as an outsider, but most of the code changes ended up in the documentation review request.
        Hide
        wrouesnel Will Rouesnel added a comment -

        Latest round of fixups is waiting for review again.

        Show
        wrouesnel Will Rouesnel added a comment - Latest round of fixups is waiting for review again.
        Hide
        wrouesnel Will Rouesnel added a comment -

        Rebased and awaiting code review again against the latest master.

        Show
        wrouesnel Will Rouesnel added a comment - Rebased and awaiting code review again against the latest master.
        Show
        kaysoky Joseph Wu added a comment - Module: https://reviews.apache.org/r/51257/ Docs: https://reviews.apache.org/r/51258/

          People

          • Assignee:
            wrouesnel Will Rouesnel
            Reporter:
            wrouesnel Will Rouesnel
            Shepherd:
            Joseph Wu
          • Votes:
            3 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:

              Development