Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-1737

Oozie log streaming is slow

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.1.0
    • Component/s: None
    • Labels:
      None

      Description

      Most of cases, Oozie search huge set of log files and return lot of logs.
      Most of our users has complained that log streaming is slow and they can't easily find relevant information.

      To make log streaming fast and precise we should provide few options (which user can specify for log streaming) like
      1. Start time : Start date (if number, then offset(in hour) from coord/wf/bundle start)
      2. End time : End date (if number, then offset from coord/wf/bundle start)
      3. Log level :
      Setting log level to ERROR, will only give erred log and exception.
      In most of cases user want to find out error message and exception.
      4. Number of lines : limit number of lines of logs from start date.
      5. Recent logs: In most of the case user is only interested in recent activity.
      Similar to https://issues.apache.org/jira/browse/OOZIE-1716.
      User should be able to specify recent logs in hours.
      6. Grep for pattern.

      + in addition to this, we also need to clean up log statements( which may also include consolidating log statements).

        Attachments

        1. OOZIE-1737-V5.patch
          146 kB
          Purshotam Shah
        2. OOZIE-1737-V6.patch
          146 kB
          Purshotam Shah
        3. OOZIE-1737-V7.patch
          136 kB
          Purshotam Shah
        4. OOZIE-1737-V8.patch
          6 kB
          Purshotam Shah

          Issue Links

            Activity

              People

              • Assignee:
                puru Purshotam Shah
                Reporter:
                puru Purshotam Shah
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: