Details

    • Type: Sub-task Sub-task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      It will be useful if we can support RPC min fair share on a per user or group basis. That will be useful for SLA jobs in a shared cluster. It will be complementary to the history-based soft policy defined in fair queue's history RPC server.

        Issue Links

          Activity

          Hide
          Yong Zhang added a comment -

          Hi Ming Ma, I would like work on this jira. Can I take it?

          Show
          Yong Zhang added a comment - Hi Ming Ma , I would like work on this jira. Can I take it?
          Hide
          Yong Zhang added a comment -

          Is it in this way, for example, if RPC server receive 10 calls as A, B, B, A, C, C, D, A, B, C.
          then server side will create 4 queues for A, B, C and D.
          Q-A: 3 calls
          Q-B: 3 calls
          Q-C: 3 calls
          Q-D: 1 calls
          Then RPC server will try to get call in fair way: Q-A, Q-B, Q-C,Q-D, Q-A, Q-B, Q-C, Q-A, Q-B, Q-C.
          Q-A, Q-B, Q-C,Q-D will be kept for a short time (can be configured), then will be deleted.

          Show
          Yong Zhang added a comment - Is it in this way, for example, if RPC server receive 10 calls as A, B, B, A, C, C, D, A, B, C. then server side will create 4 queues for A, B, C and D. Q-A: 3 calls Q-B: 3 calls Q-C: 3 calls Q-D: 1 calls Then RPC server will try to get call in fair way: Q-A, Q-B, Q-C,Q-D, Q-A, Q-B, Q-C, Q-A, Q-B, Q-C. Q-A, Q-B, Q-C,Q-D will be kept for a short time (can be configured), then will be deleted.
          Hide
          Ming Ma added a comment -

          Thanks, Yong Zhang. You are welcome to take it over. Couple notes:

          • As you can tell from the main task HADOOP-9640, the majority of RPC fair callqueue functionality have been completed and running in large production clusters. This specific jira is to make the fair share configurable.
          • In addition to user based fair share, it might be useful to evaluate NN method based fair share, e.g., getContentSummary has lower share than addBLock.
          • Implementation wise, you might want to investigate how to leverage the RPC scheduler abstraction.
          • I did some prototype a while back. The effectiveness of configurable fair share depends on the work load. It is useful if you have extremely high write volume. It will be good you can do additional benchmarking.
          Show
          Ming Ma added a comment - Thanks, Yong Zhang . You are welcome to take it over. Couple notes: As you can tell from the main task HADOOP-9640 , the majority of RPC fair callqueue functionality have been completed and running in large production clusters. This specific jira is to make the fair share configurable. In addition to user based fair share, it might be useful to evaluate NN method based fair share, e.g., getContentSummary has lower share than addBLock. Implementation wise, you might want to investigate how to leverage the RPC scheduler abstraction. I did some prototype a while back. The effectiveness of configurable fair share depends on the work load. It is useful if you have extremely high write volume. It will be good you can do additional benchmarking.

            People

            • Assignee:
              Unassigned
              Reporter:
              Ming Ma
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Development