Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-4910

Improvements to spooled MappedByteBufferQueue files

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 4.15.1, 5.1.1, 4.16.1
    • Component/s: None
    • Labels:
      None

      Description

      A user ran into a JVM bug which appears to have caused a RegionServer to crash while running a topN aggregate query. This left a large number of files in /tmp after the RS had gone away (due to a JVM SIGBUS crash). MappedByteBufferQueue will buffer results in memory up to 20MB by default (controlled by phoenix.query.spoolThresholdBytes) and then start appending them to a file. I'm seeing two things which could be improved:

      • If the RS exits abnormally, there is no process to clean up files - would be nice to register the deleteOnExit() hook to try to clean these up.
      • There is no ability to control where MappedByteBufferQueue writes its spool file - would be nice to use something other than /tmp (I think we have a property to control this already in our config..)

      FYI Ankit Singhal

        Attachments

        1. PHOENIX-4910.002.patch
          29 kB
          Josh Elser
        2. PHOENIX-4910.001.patch
          27 kB
          Josh Elser

          Activity

            People

            • Assignee:
              elserj Josh Elser
              Reporter:
              elserj Josh Elser
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: