@k82cn There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
(2) specifying the label manually: /sig <label>

Note: method (1) will trigger a notification to the team. You can find the team list here.

/sig apps

One option is similar with addNode, get all DS and check it; the performance is a concern: pod deletion is more frequent than node addition.

I'm thinking to handle it by #42002; let scheduler help to place the Pod when there's enough resource.

@kubernetes/sig-apps-bugs @kubernetes/sig-scheduling-bugs

/assign

We'd love to see this fixed in 1.8. It's unlikely we'll have #42002 done in 1.8, so @k82cn would you be willing to fix it with the naive solution first?

@janetkuo , sure, np. let's fix it before #42002.

I've always thought that there is some kind of periodic resync which checks all daemons and should handle this issue. It's even mentioned in some code comments.
Does it even work? Or how does it work?

@lukaszo wrote:

I've always thought that there is some kind of periodic resync [...]

I thought so too, but as far as I can tell, the default period is 12 hours and the randomized value could be up to 2x that, so we can't really rely on it for this kind of problem.

@lukaszo , @enisoc , I think this's for reflector sync. In DaemonSet, there are runWorkers running to sync up DaemonSet; but it's triggered by event, e.g. AddNode, DaemonSet's Pod deleted. If any misunderstanding, please correct me :).

As far as I can tell, the min-resync-period flag (default 12hrs) ends up being used as the defaultEventHandlerResyncPeriod, which is the resync period you get if you use AddEventHandler() (like DaemonSet does) instead of AddEventHandlerWithResyncPeriod() (like StatefulSet does).

So I think this explains why DaemonSet does not appear to resync. It's using the default period which is a random value between 12h and 24h.

Eventually, statefulsets should also be pushed to use AddEventHandler. We don't want controller loops to depend on tight resync intervals. The resource handlers on secondary caches are paramount for complementing the resync logic of the main controllers, eg. once a Node has more resources (status change on the Node?), a watch event in the secondary (node) cache of the DS controller should trigger a resync of all the DaemonSets that want to run on that node.

once a Node has more resources (status change on the Node?)

nop, it dependents on pod's event (e.g. add, update, delete) :(. It avoid too many info in Node object. In scheduler, it cache those info in schedulercache; and uses those info by PodFitResource predicate.

In DaemonSet's deletePod handler, we only care about the DS pod and enqueue DS; if other pod in Node is deleted, DS controller will not do anything.