This prevents the mistake from 1.34 where the default-on
DRAResourceClaimDeviceStatus feature caused the use of the experimental
allocator implementation. The test fails without a fix for that.
In 1.34, the default feature gate selection picked the "experimental" allocator
implementation when it should have used the "incubating" allocator. No harm
came from that because the experimental allocator has all the necessary if
checks to disable the extra code and no bugs were introduced when implementing
it, but it means that our safety net wasn't there when we expected it to be.
The reason is that the "DRAResourceClaimDeviceStatus" feature gate is on by
default and was only listed as supported by the experimental implementation.
This could be fixed by listing it as supported also by the other
implementation, but that would be a bit odd because there is nothing to support
for it (the reason why this was missed in 1.34!). Instead, the allocator
features are now only indirectly related to feature gates, with a single
boolean controlling the implementation of binding conditions.
Copying from feature.Features to new fields in the plugin got a bit silly with
the long list of features that we have now. Embedding feature.Features is
simpler.
Two fields in feature.Features weren't named according to the feature gate, now
they are named consistently and the fields are sorted.
Currently volume and dynamic-resource plugin shares an AssumeCache
implementation. However, they features significantly different use case. DRA
call Assume() on objects returned by APIServer, but volume call Assume() on
objects yet to be sent to the APIServer.
VolumeBinding plugin only makes one update request, while DynamicResource makes
2 requests (add finalizer then update allocation status). Taking advantage of
this, currently the volume cache is simpler:
1. Reserve: assume PV/PVC will be updated
2. PreBind: really send the update request
3. AssumeCache receives an update from informer and overwrite the assumed state.
a. if Prebind succeeded, this will surely include the update from step 2.
b. if PreBind is not finished yet, and this is an irrelevant update, it is safe to
overwrite the assumed state, because our update in PreBind will surely fail with Conflict.
While for DynamicResource:
1. Reserve: add devices to inFlightAllocations
2. PreBind:
a. send the 2 update requests
b. add the returned object into AssumeCache
c. AssumeCache dispatch events synchronously to update allocatedDevices
d. remove devices from inFlightAllocations
DynamicResource needs some features from AssumeCache that is not necessary for VolumeBinding:
1. DynamicResource needs strictly ordered update events to update allocatedDevices,
including those from Assume() and Restore()
2. DynamicResource needs to compare ResourceVersion to prevent the assumed state from being
overwritten by older version from informer. While this works, the doc[1] says:
"you must not compare resource versions for greater-than or less-than relationships".
Given so much difference, it can be beneficial to fork another simpler
AssumeCache for VolumeBinding plugin. Because of no need to send events, the
lite AssumeCache is a passive component. It only record the assumed version
without copying all objects from informer into its local cache. When reading,
we read from both informer and local cache. So it will always be up-to-date
with informer, no need to wait for event handler.
This resolves a race condition where AssumeCache and scheduler queue both
receive events from informer. When a pod is scheduling due to PV update event,
the PVCache may be not updated yet because it has not processed the relevant
event.
The passive version still listens events from informer, but only for cleaning up
its local cache to save memory.
[1]: https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions
TestPlugin/multi-claims-binding-conditions-all-success/PreEnqueue
flakes due to the assumed cache not been synced with the initial
store. The test waits until the registered handler used by the
assumed cache has synced before proceeding with the test
Moving Scheduler interfaces to staging: Move PodInfo and NodeInfo interfaces (together with related types) to staging repo, leaving internal implementation in kubernetes/kubernetes/pkg/scheduler
As before when adding v1beta2, DRA drivers built using the
k8s.io/dynamic-resource-allocation helper packages remain compatible with all
Kubernetes release >= 1.32. The helper code picks whatever API version is
enabled from v1beta1/v1beta2/v1.
However, the control plane now depends on v1, so a cluster configuration where
only v1beta1 or v1beta2 are enabled without the v1 won't work.
Added a skipOnWindows flag to DynamicResources scheduler test case
to skip test that relies on nanosecond timer precision.
Windows timer granularity is much coarser than Linux, which causes
the test to fail often.
The intent is to catch abnormal runtimes with the generously large default
timeout of 10 seconds.
We have to set up a context with the configured timeout (optional!), then
ensure that both CEL evaluation and the allocation logic itself properly
returns the context error. The scheduler plugin then can convert that into
"unschedulable".
The allocator and thus Filter now also check for context cancellation by the
scheduler. This happens when enough nodes have been found.
Initializing the scheduler Features struct will be needed in different places,
therefore NewSchedulerFeaturesFromGates gets introduced. Besides, having it
next to the struct makes it easier to add new features.
The DRASchedulerFilterTimeout feature gate simplifies disabling the timeout
because setting a feature gate is often easier than modifying the scheduler
configuration with a zero timeout value.
The timeout and feature gate are new. The gate starts as beta and enabled by
default, which is consistent with the "smaller changes with low enough risk
that still may need to be disabled..." guideline.
This is meant for simple changes, like code cleanup or API changes of the
allocator code. For more complex changes and new features, SIG Scheduling
approvers will be required to approve, as before.
The goal is to maintain different version of the allocator logic. We already
had one incidence where adding an alpha feature caused a regression also when
it was disabled. Not everything can be implemented within obviously correct if
branches.
This also opens the door for implementing different alternatives.
The code just gets moved around for now.
