Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.5.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The Multiplexer helps the FairCallQueue decide which of its internal sub-queues to read from during a poll() or take(). It controls the penalty of being in a lower queue. Without the mux, the FairCallQueue would have issues with starvation of low-priority requests.

      The WeightedRoundRobinMultiplexer is an implementation which uses a weighted round robin approach to muxing the sub-queues. It is configured with an integer list pattern.

      For example: 10, 5, 5, 2 means:

      • Read queue 0 10 times
      • Read queue 1 5 times
      • Read queue 2 5 times
      • Read queue 3 2 times
      • Repeat
      1. HADOOP-10279.patch
        11 kB
        Chris Li
      2. HADOOP-10279.patch
        11 kB
        Chris Li
      3. subtask2_add_mux.patch
        11 kB
        Chris Li
      4. WeightedRoundRobinMultiplexer.java
        5 kB
        Chris Li

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1807 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1807/)
          HADOOP-10279. Create multiplexer, a requirement for the fair queue. (Contributed by Chris Li) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604090)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WeightedRoundRobinMultiplexer.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestWeightedRoundRobinMultiplexer.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1807 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1807/ ) HADOOP-10279 . Create multiplexer, a requirement for the fair queue. (Contributed by Chris Li) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604090 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WeightedRoundRobinMultiplexer.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestWeightedRoundRobinMultiplexer.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1780 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1780/)
          HADOOP-10279. Create multiplexer, a requirement for the fair queue. (Contributed by Chris Li) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604090)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WeightedRoundRobinMultiplexer.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestWeightedRoundRobinMultiplexer.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1780 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1780/ ) HADOOP-10279 . Create multiplexer, a requirement for the fair queue. (Contributed by Chris Li) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604090 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WeightedRoundRobinMultiplexer.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestWeightedRoundRobinMultiplexer.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #589 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/589/)
          HADOOP-10279. Create multiplexer, a requirement for the fair queue. (Contributed by Chris Li) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604090)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WeightedRoundRobinMultiplexer.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestWeightedRoundRobinMultiplexer.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #589 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/589/ ) HADOOP-10279 . Create multiplexer, a requirement for the fair queue. (Contributed by Chris Li) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604090 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WeightedRoundRobinMultiplexer.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestWeightedRoundRobinMultiplexer.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5742 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5742/)
          HADOOP-10279. Create multiplexer, a requirement for the fair queue. (Contributed by Chris Li) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604090)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WeightedRoundRobinMultiplexer.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestWeightedRoundRobinMultiplexer.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5742 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5742/ ) HADOOP-10279 . Create multiplexer, a requirement for the fair queue. (Contributed by Chris Li) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604090 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WeightedRoundRobinMultiplexer.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestWeightedRoundRobinMultiplexer.java
          Hide
          Arpit Agarwal added a comment -

          I committed this to trunk and branch-2. Thanks for the contribution Chris Li!

          Show
          Arpit Agarwal added a comment - I committed this to trunk and branch-2. Thanks for the contribution Chris Li !
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12651270/HADOOP-10279.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4122//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4122//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12651270/HADOOP-10279.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4122//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4122//console This message is automatically generated.
          Hide
          Arpit Agarwal added a comment -

          +1 pending Jenkins.

          Thanks for the updated patch Chris.

          Show
          Arpit Agarwal added a comment - +1 pending Jenkins. Thanks for the updated patch Chris.
          Hide
          Chris Li added a comment -

          Thanks for the feedback Arpit. Uploaded new patch with the changes

          Show
          Chris Li added a comment - Thanks for the feedback Arpit. Uploaded new patch with the changes
          Hide
          Arpit Agarwal added a comment -

          Hi Chris, thanks for the additional clarification.

          This change looks fine to me. Just a couple of nitpicks:

          1. Coding style, spaces around = and operators.
          2. advanceIndex and getCurrentIndex look like they can be private.
          Show
          Arpit Agarwal added a comment - Hi Chris, thanks for the additional clarification. This change looks fine to me. Just a couple of nitpicks: Coding style, spaces around = and operators. advanceIndex and getCurrentIndex look like they can be private.
          Hide
          Chris Li added a comment -

          Added comments to and linked all 3 parts of the FairCallQueue. They are intended to be pluggable but I haven't gotten around to that yet since we're still experimenting.

          Show
          Chris Li added a comment - Added comments to and linked all 3 parts of the FairCallQueue. They are intended to be pluggable but I haven't gotten around to that yet since we're still experimenting.
          Hide
          Chris Li added a comment -

          I'll put a writeup in HADOOP-10282 and let you know when it's done

          Show
          Chris Li added a comment - I'll put a writeup in HADOOP-10282 and let you know when it's done
          Hide
          Arpit Agarwal added a comment -

          Thanks, I got that part.

          However I am not sure if this class is intended to be replaceable with alternative mux implementations, or is WeightedRoundRobinMultiplexer the only option. I didn't look through HADOOP-10282 patch yet, but perhaps if there was a short description of how the classes fit together it would be easier to review.

          Show
          Arpit Agarwal added a comment - Thanks, I got that part. However I am not sure if this class is intended to be replaceable with alternative mux implementations, or is WeightedRoundRobinMultiplexer the only option. I didn't look through HADOOP-10282 patch yet, but perhaps if there was a short description of how the classes fit together it would be easier to review.
          Hide
          Chris Li added a comment -

          Hi Arapit, sorry for the lack of context; the mux is a small component of the FairCallQueue here: https://issues.apache.org/jira/browse/HADOOP-10282

          It could even exist as a nested class, but it might be useful to make swappable to enable different behaviors

          Show
          Chris Li added a comment - Hi Arapit, sorry for the lack of context; the mux is a small component of the FairCallQueue here: https://issues.apache.org/jira/browse/HADOOP-10282 It could even exist as a nested class, but it might be useful to make swappable to enable different behaviors
          Hide
          Arpit Agarwal added a comment -

          Hi Chris, are there any prerequisites for this Jira? I guess I am not clear on how this class is intended to be used. Will callers hardcode usage of WeightedRoundRobinMultiplexer?

          Also could you please add Javadocs for the public methods?

          Show
          Arpit Agarwal added a comment - Hi Chris, are there any prerequisites for this Jira? I guess I am not clear on how this class is intended to be used. Will callers hardcode usage of WeightedRoundRobinMultiplexer ? Also could you please add Javadocs for the public methods?
          Hide
          Chris Li added a comment -

          Upload latest version of patch for reference

          Show
          Chris Li added a comment - Upload latest version of patch for reference
          Hide
          Chris Li added a comment -

          Adding a preview of the new concurrent version of the multiplexer.

          Show
          Chris Li added a comment - Adding a preview of the new concurrent version of the multiplexer.
          Hide
          Chris Li added a comment -

          Hi zhaoyunjiong, thanks for taking a look.

          This is handled in the FairCallQueue: it treats the index from the Multiplexer as a suggestion and works on the first non-empty queue, starting from this suggestion. So if the Mux suggests queue 2 but queues 2, 3, and 4 are empty, the FairCallQueue will use queue 5, or whatever queue is not empty.

          This way, the Mux doesn't need to know anything about what queues are.

          Show
          Chris Li added a comment - Hi zhaoyunjiong , thanks for taking a look. This is handled in the FairCallQueue: it treats the index from the Multiplexer as a suggestion and works on the first non-empty queue, starting from this suggestion. So if the Mux suggests queue 2 but queues 2, 3, and 4 are empty, the FairCallQueue will use queue 5, or whatever queue is not empty. This way, the Mux doesn't need to know anything about what queues are.
          Hide
          zhaoyunjiong added a comment -

          What if when queue0 didn't have any calls in queue and requestsLeft still > 0, while other queues have calls waiting in the queue?
          I think when this happens, it'll be better moveToNextQueue immediately.

          Show
          zhaoyunjiong added a comment - What if when queue0 didn't have any calls in queue and requestsLeft still > 0, while other queues have calls waiting in the queue? I think when this happens, it'll be better moveToNextQueue immediately.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12625146/subtask2_add_mux.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3466//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12625146/subtask2_add_mux.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3466//console This message is automatically generated.
          Hide
          Chris Li added a comment -

          Depends on subtask1: This multiplexer enables takers of the FairCallQueue to withdraw in a weighted round-robin fashion to combat starvation of low-priority Schedulables.

          Weights are configurable as follows:
          ipc.8020.wrr-multiplexer.weights = 30, 20, 5

          Means queue0 will be drawn from thirty times, queue1 twenty times, queue2 five times, and repeat.

          Show
          Chris Li added a comment - Depends on subtask1: This multiplexer enables takers of the FairCallQueue to withdraw in a weighted round-robin fashion to combat starvation of low-priority Schedulables. Weights are configurable as follows: ipc.8020.wrr-multiplexer.weights = 30, 20, 5 Means queue0 will be drawn from thirty times, queue1 twenty times, queue2 five times, and repeat.

            People

            • Assignee:
              Chris Li
              Reporter:
              Chris Li
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development