Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-3382

[SSH action] Optimize process streams draining

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 5.1.0
    • Fix Version/s: 5.2.0
    • Component/s: None
    • Labels:
      None

      Description

      OOZIE-3354 improved SshActionExecutor to avoid Process#waitFor() blocks and modified the drainBuffers method to keep draining the standard output (and standard error) continuously.

      Right now the speed of the drain is hardwired. As long as the process is running the method only reads 1024 bytes in each cycle (half a second) which can take very long time if we want to drain several megabytes (for instance oozie.servlet.CallbackServlet.max.data.len is increased).

      Let's optimize the draining.

      We can either read 1024 bytes multiple times in each cycle (as long as there are data in the buffer), or we can increase the value of the buffer size (1024).

      In the latter case the default of the buffer size could be half of the oozie.servlet.CallbackServlet.max.data.len value, but we also need an additional property to specify the buffer size (to avoid memory problems because of using a very big buffer). We can keep 1024 as a minimum buffer size.

      It would be also useful to refactor the code and put the buffer draining into a separate class and create unit tests for the class. Using this class in ShellMain to avoid code duplication would also be very useful, but we have to fix OOZIE-3359 first.

        Attachments

        1. OOZIE-3382-04.patch
          58 kB
          Andras Salamon
        2. OOZIE-3382-03.patch
          29 kB
          Andras Salamon
        3. OOZIE-3382-02.patch
          29 kB
          Andras Salamon
        4. OOZIE-3382-01.patch
          28 kB
          Andras Salamon

          Issue Links

            Activity

              People

              • Assignee:
                asalamon74 Andras Salamon
                Reporter:
                asalamon74 Andras Salamon
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: