Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-2405

Improve performance and stability of server side sort for ORDER BY

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None

    Description

      We currently use memory mapped files to buffer data as it's being sorted in an ORDER BY (see MappedByteBufferQueue). The following types of exceptions have been seen to occur:

      Caused by: java.lang.OutOfMemoryError: Map failed
              at sun.nio.ch.FileChannelImpl.map0(Native Method)
              at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:904)
      

      Andrew Kyle Purtell has read that memory mapped files are not cleaned up after very well in Java:

      "Map failed" means the JVM ran out of virtual address space. If you search around stack overflow for suggestions on what to do when your app (in this case Phoenix) encounters this issue when using mapped buffers, the answers tend toward manually cleaning up the mapped buffers or explicitly triggering a full GC. See http://stackoverflow.com/questions/8553158/prevent-outofmemory-when-using-java-nio-mappedbytebuffer for example. There are apparently long standing JVM/JRE problems with reclamation of mapped buffers. I think we may want to explore in Phoenix a different way to achieve what the current code is doing.

      Instead of using memory mapped files, we could use heap memory, or perhaps there are other mechanisms too.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            RCheungIT Haoran Zhang
            jamestaylor James R. Taylor

            Dates

              Created:
              Updated:

              Slack

                Issue deployment