Details

      Description

      My company has gone through a fault in production, in which a critical switch causes unstable network for a set of machines with package loss rate of 30%-50%. In such fault, the supervisors and workers on the machines are not definitely dead, which is easy to handle. Instead they are still alive but very unstable. They lost heartbeat to the nimbus occasionally. The nimbus, in such circumstance, will still assign jobs to these machines, but will soon find them invalid again, result in a very slow convergence to stable status.
      To deal with such unstable cases, we intend to implement a blacklist scheduler, which will add the unstable nodes (supervisors, slots) to the blacklist temporarily, and resume them later.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                howardhl Howard Lee
                Reporter:
                howardhl Howard Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 17h 50m
                  17h 50m