Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-1561

When using Oozie HA, the logs should also be HA

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: trunk
    • Fix Version/s: None
    • Component/s: HA
    • Labels:
      None

      Description

      Currently, if an Oozie server goes down, the logs from that server become unavailable until the server comes back up. In the meantime, the user may or may not be aware that log messages could be missing when Oozie streams logs to the user.

      We should come up with a way to make the logs HA.
      Some ideas:

      1. When rolling the logs, copy them into HDFS; Oozie servers can then read the log files directly from HDFS instead of each other
        • The downside to this is that there will be a window where logs could still be missing as they only show up in HDFS after rolling over (default = 1hr) and Oozie servers would still have to contact each other for the last hour of logs
        • The upside is that it minimizes the amount of logs that could be missing and would be fairly straightforward to implement
      2. Log directly to HDFS
        • The downside is that this may be complicated or tricky to get working properly
          • This also introduces a strict dependency on HDFS
        • The upside is that this would completely solve the issue and Oozie servers would simply get all logs directly from HDFS
      3. Log to ZooKeeper or a database
        • I think the log files will be too big to do this

      I've assigned this to myself, but if someone wants to tackle this, feel free to reassign it. I think idea 2 is the most practical, but I'm also open to other ideas on how to do this.

        Attachments

        1. OozielogHAtechnicaldesigndoc.pdf
          72 kB
          Bowen Zhang
        2. OozielogHAtechnicaldesigndoc.pdf
          51 kB
          Bowen Zhang

          Issue Links

            Activity

              People

              • Assignee:
                bowenzhangusa Bowen Zhang
                Reporter:
                rkanter Robert Kanter
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated: