Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently if FairCallQueue and backoff are enabled, backoff will kick in as soon as the assigned sub queue is filled up.

        /**
         * Put and offer follow the same pattern:
         * 1. Get the assigned priorityLevel from the call by scheduler
         * 2. Get the nth sub-queue matching this priorityLevel
         * 3. delegate the call to this sub-queue.
         *
         * But differ in how they handle overflow:
         * - Put will move on to the next queue until it lands on the last queue
         * - Offer does not attempt other queues on overflow
         */
      

      Seems it is better to try lower priority sub queues when the assigned sub queue is full, just like the case when backoff is disabled. This will give regular users more opportunities and allow the cluster to be configured with smaller call queue length. Chris Li, Arpit Agarwal, what do you think?

        Activity

        Hide
        daryn Daryn Sharp added a comment -

        I've been pondering this very question since upgrading to 2.7. I'm not sure it's a good idea unless well thought out since the current impl suffers from some unexpected behaviors.

        In our experience an abusive user floods the 1st queue and backoff kicks in for everyone. Agreed, less than desirable. Bad user decays to a lower queue, probably floods it too. Depending on job width all queues may fill but decay is a dampener. Allowing simple spillover for all will allow the bad user to rapidly fill queues. On the flip-side, w/o spillover, nobody was going to get those queue slots anyway.

        Additionally, total call count is driven up by the backoff rejected calls. Generally shrinking call percentages of all other users into highest priority queue. Once the bad users' tasks are all blocked in queues, the incoming call rate appears lower. They decay back into a lower queue and clog it back up again.

        Show
        daryn Daryn Sharp added a comment - I've been pondering this very question since upgrading to 2.7. I'm not sure it's a good idea unless well thought out since the current impl suffers from some unexpected behaviors. In our experience an abusive user floods the 1st queue and backoff kicks in for everyone. Agreed, less than desirable. Bad user decays to a lower queue, probably floods it too. Depending on job width all queues may fill but decay is a dampener. Allowing simple spillover for all will allow the bad user to rapidly fill queues. On the flip-side, w/o spillover, nobody was going to get those queue slots anyway. Additionally, total call count is driven up by the backoff rejected calls. Generally shrinking call percentages of all other users into highest priority queue. Once the bad users' tasks are all blocked in queues, the incoming call rate appears lower. They decay back into a lower queue and clog it back up again.
        Hide
        mingma Ming Ma added a comment -

        Thanks Daryn Sharp! Here is the issue we had that motivates this jira, but after offline discussion with Chris Li and team members, we feel like tuning FairCallQueue configs should achieve the same result.

        With FariCallQueue and backoff, we don't get much complaints regarding one abusive user's impact on other users. The main issue we currently have is a heavy user's impact on datanode service rpc requests which has been increasing as we continue to expand our cluster size. FairCallQueue is only for client RPC, not for datanode RPC. There was some discussion in HADOOP-10599 about this. Specifically:

        • A heavy user generates lots of rpc requests, but it only filled up 1/4 of the lowest priority sub queue. However that is enough to cause lock contention with DN RPC requests.
        • So to have backoff kick in sooner for the heavy user, we can reduce the rpc sub queue length. But that will impact all rpc sub queues.
        • After the call queue length reduction, if lots of light users belonging to p0 come in at the same time, some light users will get backed off, given p0 sub queue is much smaller than before. Thus if it can overflow to the next queue, light users at least won't get backed off.

        However, several configs tuning including client and service rpc handler count and FairCallQueue weight adjustment should be able to achieve the same result.

        On a related note, if FairCallQueue is used but backoff is disabled, as mentioned in the description, put method will move on to the next queue until it lands on the last queue. It isn't clear why it can't just block on the corresponding sub queue instead. In other words, what is the reason overflow is useful for the block case, to reduce the chance that the reader threads being blocked? Still, it seems configs tuning can also achieve that, similar to the argument for the backoff case.

        Show
        mingma Ming Ma added a comment - Thanks Daryn Sharp ! Here is the issue we had that motivates this jira, but after offline discussion with Chris Li and team members, we feel like tuning FairCallQueue configs should achieve the same result. With FariCallQueue and backoff, we don't get much complaints regarding one abusive user's impact on other users. The main issue we currently have is a heavy user's impact on datanode service rpc requests which has been increasing as we continue to expand our cluster size. FairCallQueue is only for client RPC, not for datanode RPC. There was some discussion in HADOOP-10599 about this. Specifically: A heavy user generates lots of rpc requests, but it only filled up 1/4 of the lowest priority sub queue. However that is enough to cause lock contention with DN RPC requests. So to have backoff kick in sooner for the heavy user, we can reduce the rpc sub queue length. But that will impact all rpc sub queues. After the call queue length reduction, if lots of light users belonging to p0 come in at the same time, some light users will get backed off, given p0 sub queue is much smaller than before. Thus if it can overflow to the next queue, light users at least won't get backed off. However, several configs tuning including client and service rpc handler count and FairCallQueue weight adjustment should be able to achieve the same result. On a related note, if FairCallQueue is used but backoff is disabled, as mentioned in the description, put method will move on to the next queue until it lands on the last queue. It isn't clear why it can't just block on the corresponding sub queue instead. In other words, what is the reason overflow is useful for the block case, to reduce the chance that the reader threads being blocked? Still, it seems configs tuning can also achieve that, similar to the argument for the backoff case.
        Hide
        arpitagarwal Arpit Agarwal added a comment - - edited

        Hi Ming Ma, backoff has two good properties:

        1. Prevents stalling the IPC reader thread indefinitely.
        2. Throttles clients when the NameNode is under load by signaling congestion.

        Spilling to lower priority queue with FairCallQueue delays the 'congestion' signal so it makes (2) less effective. The networking world seems to think earlier notification of congestion is better e.g. TCP RED, ECN and delay-based congestion control.

        A heavy user generates lots of rpc requests, but it only filled up 1/4 of the lowest priority sub queue. However that is enough to cause lock contention with DN RPC requests.

        Xiaoyu Yao recently introduced HADOOP-12916 with the goal of addressing the same problem.

        Show
        arpitagarwal Arpit Agarwal added a comment - - edited Hi Ming Ma , backoff has two good properties: Prevents stalling the IPC reader thread indefinitely. Throttles clients when the NameNode is under load by signaling congestion. Spilling to lower priority queue with FairCallQueue delays the 'congestion' signal so it makes (2) less effective. The networking world seems to think earlier notification of congestion is better e.g. TCP RED , ECN and delay-based congestion control. A heavy user generates lots of rpc requests, but it only filled up 1/4 of the lowest priority sub queue. However that is enough to cause lock contention with DN RPC requests. Xiaoyu Yao recently introduced HADOOP-12916 with the goal of addressing the same problem.

          People

          • Assignee:
            Unassigned
            Reporter:
            mingma Ming Ma
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development