Apache AWF
  1. Apache AWF
  2. AWF-36

Multithreaded Deft to have multiple load-balanced IOLoop instance

    Details

    • Type: Bug Bug
    • Status: Closed
    • Resolution: Unresolved
    • Labels:
      None

      Description

      The main idea behind is to be able to have more than one IOLoop thread running in the JVM, so Defft can use every single core or CPU of the hosting system. I did something like that in my forked project.
      I see two orientation for this:

      • One infinite loop using a selector to listen for incoming request (accept) and pushing accepted sockets to one or more infinite thread loop. This is what I did in my fork . The infinite thread loop use another selector to handle read, write and all other operations, so channel events are still executed in the same single thread.
      • One selector holding listening sockets is shared between multiple infinite IOLoop threads. Each IOLoop hold its own selector for read and write operations. There should be a lock on the shared selector but this may be easier to develop.

      My purpose is to allow Deft to execute on all available CPU cores without breaking the infinite loop philosophy.

        Activity

        Tony Stevenson made changes -
        Field Original Value New Value
        Project Deft [ 12311521 ] Apache AWF [ 12313220 ]
        Key DEFT-139 AWF-36
        Reporter Niklas Gustavsson [ niklas ]
        Show
        Ulrich Stärk added a comment - closed by https://github.com/rschildmeijer/deft/commit/2a10a1bba2d4a0524be9c477489b7b1de9063c54
        Hide
        Ulrich Stärk added a comment -

        Should be ok to register the same SelectableChannel against multiple selectors. (as long as we have the modification (the null check) in handleAccept(..), multiple selectors might be "notified" about the io and call the handleAccept callback, inside each handleAccept and each IOHandler will try to do something like socket.accept(), but only one (first one?) will get a valid return value)

        Making the HttpProtocol immutable could possible be done. I dont know how to solve the field partials (a Map<SelectableChannel, PartialHttpRequest>) on top of my head. (Also I think its ok to create n HttpProtocol objets for n IOLoops).

        Your last point, about adding a new SelectableChannel: Outside of the IOLoop itself, do you mean do that from another thread? Simply doing ioLoop.addHandler(channel, ioHandler, SelectionKey.ACCEPT, null) is not threadsafe to do, instead you should do:

        ioLoop.addCallback(new AsyncCallback() {
        @Override public void onCallback()

        { ioLoop.addHandler(channel, handler, interestOps, attachment) }

        });

        (So you are sure that adding of the handler will be done from the correct ioloop thread)

        Show
        Ulrich Stärk added a comment - Should be ok to register the same SelectableChannel against multiple selectors. (as long as we have the modification (the null check) in handleAccept(..), multiple selectors might be "notified" about the io and call the handleAccept callback, inside each handleAccept and each IOHandler will try to do something like socket.accept(), but only one (first one?) will get a valid return value) Making the HttpProtocol immutable could possible be done. I dont know how to solve the field partials (a Map<SelectableChannel, PartialHttpRequest>) on top of my head. (Also I think its ok to create n HttpProtocol objets for n IOLoops). Your last point, about adding a new SelectableChannel: Outside of the IOLoop itself, do you mean do that from another thread? Simply doing ioLoop.addHandler(channel, ioHandler, SelectionKey.ACCEPT, null) is not threadsafe to do, instead you should do: ioLoop.addCallback(new AsyncCallback() { @Override public void onCallback() { ioLoop.addHandler(channel, handler, interestOps, attachment) } }); (So you are sure that adding of the handler will be done from the correct ioloop thread)
        Hide
        Ulrich Stärk added a comment -

        Hi, I am realy happy to see that this idea does interrest some people .
        As you said your first test looks pretty good.
        I have not applied the patch for now but I will for sure. From what I see, you register the same serverChannel against multiple selectors in different IOLoop instances. Are you sure we can do it without any risk ?
        For me HttpProtocol could be easily thread safe, if it is totally stateless so a single instance is needed.

        The last thing I see is that we should take care of how we add a new SocketChannels to an IOLoop instance from outside of the loop itself.

        Show
        Ulrich Stärk added a comment - Hi, I am realy happy to see that this idea does interrest some people . As you said your first test looks pretty good. I have not applied the patch for now but I will for sure. From what I see, you register the same serverChannel against multiple selectors in different IOLoop instances. Are you sure we can do it without any risk ? For me HttpProtocol could be easily thread safe, if it is totally stateless so a single instance is needed. The last thing I see is that we should take care of how we add a new SocketChannels to an IOLoop instance from outside of the loop itself.
        Hide
        Ulrich Stärk added a comment -

        Patch: https://gist.github.com/989674 (interesting parts, row 327-352, the rest is basically adding an IOLoop instance to some constructors and some methods)

        hello world benchmark: (2.66ghz quad core)
        single deft instance: 29k requests/s
        eight threads/ioloops : 82k requests/s

        pretty good imho.

        slemesle, you have spent some brain cycles on this problem, what are your comments about this? (probably easiest to apply the patch and look at the diff instead of the gist)

        Show
        Ulrich Stärk added a comment - Patch: https://gist.github.com/989674 (interesting parts, row 327-352, the rest is basically adding an IOLoop instance to some constructors and some methods) hello world benchmark: (2.66ghz quad core) single deft instance: 29k requests/s eight threads/ioloops : 82k requests/s pretty good imho. slemesle, you have spent some brain cycles on this problem, what are your comments about this? (probably easiest to apply the patch and look at the diff instead of the gist)
        Hide
        Ulrich Stärk added a comment -

        ...after some further thinking:
        Could be worth some experiments where we "clone" (deep copy) Application and HttpProtocol and tries to mimic the way multiprocessing work (no shared memory between any server code.). We also need to make the IOLoop ctor public (private at the moment)

        Show
        Ulrich Stärk added a comment - ...after some further thinking: Could be worth some experiments where we "clone" (deep copy) Application and HttpProtocol and tries to mimic the way multiprocessing work (no shared memory between any server code.). We also need to make the IOLoop ctor public (private at the moment)
        Hide
        Ulrich Stärk added a comment -

        The outcome of a discussion with inferno-:
        At first I thought that the simple "multicore" solution used by tornado would be easily translated to deft. After a deeper digging we realized thats probably not the case. The reason for this is that the multicore support would be done by using threads (with a single ioloop "attached" to it) with shared user space memory....and the threads would share some instances (e.g Application and HttpProtocol) that are not thread safe.

        Show
        Ulrich Stärk added a comment - The outcome of a discussion with inferno-: At first I thought that the simple "multicore" solution used by tornado would be easily translated to deft. After a deeper digging we realized thats probably not the case. The reason for this is that the multicore support would be done by using threads (with a single ioloop "attached" to it) with shared user space memory....and the threads would share some instances (e.g Application and HttpProtocol) that are not thread safe.
        Hide
        Ulrich Stärk added a comment -

        In tornado you can pre-fork a given number of threads, each thread will have it's own ioloop and registers for read event on the listening socket and whenever it gets a read event it performs accept() on the socket. One of the threads will get the connection and can start performing it's job. (accept() is thread safe, see manpage for more info).

        The code is available here for more analysis: https://github.com/facebook/tornado/blob/master/tornado/httpserver.py#L206

        I have no idea how this would translate to the java world.

        Show
        Ulrich Stärk added a comment - In tornado you can pre-fork a given number of threads, each thread will have it's own ioloop and registers for read event on the listening socket and whenever it gets a read event it performs accept() on the socket. One of the threads will get the connection and can start performing it's job. (accept() is thread safe, see manpage for more info). The code is available here for more analysis: https://github.com/facebook/tornado/blob/master/tornado/httpserver.py#L206 I have no idea how this would translate to the java world.
        Hide
        Ulrich Stärk added a comment -

        From this [link](http://www.tornadoweb.org/documentation#running-tornado-in-production)

        >We typically run one Tornado frontend per core on the machine (sometimes more depending on utilization).

        Show
        Ulrich Stärk added a comment - From this [link] ( http://www.tornadoweb.org/documentation#running-tornado-in-production ) >We typically run one Tornado frontend per core on the machine (sometimes more depending on utilization).
        Hide
        Ulrich Stärk added a comment -

        Do you know how facebook/tornado handles their multi processing support? (I'm just curios)

        Show
        Ulrich Stärk added a comment - Do you know how facebook/tornado handles their multi processing support? (I'm just curios)
        Anonymous created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Niklas Gustavsson
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development