If someone gains the ability to create static pods, they might try to use that
ability to run code which gets access to the resources associated with some
existing claim which was previously allocated for some other pod. Such an
attempt already fails because the claim status tracks which pods are allowed to
use the claim, the static pod is not in that list, the node is not authorized
to add it, and the kubelet checks that list before starting the pod in
195803cde5/pkg/kubelet/cm/dra/manager.go (L218-L222).
Even if the pod were started, DRA drivers typically manage node-local resources
which can already be accessed via such an attack without involving DRA. DRA
drivers which manage non-node-local resources have to consider access by a
compromised node as part of their threat model.
Nonetheless, it is better to not accept static pods which reference
ResourceClaims or ResourceClaimTemplates in the first place because there
is no valid use case for it.
This is done at different levels for defense in depth:
- configuration validation in the kubelet
- admission checking of node restrictions
- API validation
Co-authored-by: Jordan Liggitt <liggitt@google.com>
Code changes by Jordan, with one small change (resourceClaims -> resourceclaims).
Unit tests by Patrick.
Thanks to the tracker, the plugin sees all taints directly in the device
definition and can compare it against the tolerations of a request while
trying to find a device for the request.
When the feature is turnedd off, taints are ignored during scheduling.
The controller is derived from the node taint eviction controller.
In contrast to that controller it tracks the UID of pods to prevent
deleting the wrong pod when it got replaced.
SELinuxMount stays off by default, because it changes the default
kubelet behavior. SELinuxChangePolicy is on by default and notifies users
on Pods that could get broken by SELinuxMount feature gate.
* Add Watch to controller roles
Starting from version 1.32, the client feature `WatchListClient` has been
set to `true` in `kube-controller-manager`.
(commit 06a15c5cf9)
As a result, when the `kube-controller-manager` executes the `List` method,
it utilizes `Watch`. However, there are some existing controller roles that
include `List` but do not include `Watch`. Therefore, when processes using
these controller roles execute the `List` method, `Watch` is executed first,
but due to permission errors, it falls back to `List`.
This PR adds `Watch` to the controller roles that include `List` but do not
include `Watch`.
The affected roles are as follows (prefixed with `system:controller:`):
- `cronjob-controller`
- `endpoint-controller`
- `endpointslice-controller`
- `endpointslicemirroring-controller`
- `horizontal-pod-autoscaler`
- `node-controller`
- `pod-garbage-collector`
- `storage-version-migrator-controller`
Signed-off-by: Mitsuru Kariya <mitsuru.kariya@nttdata.com>
* Fix Fixture Data
I apologize, the Fixture Data modifications were missed.
Signed-off-by: Mitsuru Kariya <mitsuru.kariya@nttdata.com>
* Add ControllerRoles Test
Added a test to check that if a controller role includes `List`, it also includes `Watch`.
Signed-off-by: Mitsuru Kariya <mitsuru.kariya@nttdata.com>
* Fix typo
Co-authored-by: Jordan Liggitt <jordan@liggitt.net>
* Add Additional Tests
Added tests to check that if NodeRules, ClusterRoles, and NamespaceRoles
include `List`, it also include `Watch`.
Signed-off-by: Mitsuru Kariya <mitsuru.kariya@nttdata.com>
---------
Signed-off-by: Mitsuru Kariya <mitsuru.kariya@nttdata.com>
Co-authored-by: Jordan Liggitt <jordan@liggitt.net>
The WatchListClient feature is enabled for kube-controller-manager, but
namespace-controller misses the necessary "watch" permission, which
results in 30 error logs being generated every time a namespace is
deleted and falling back to the standard LIST semantics.
Signed-off-by: Quan Tian <quan.tian@broadcom.com>
The "// import <path>" comment has been superseded by Go modules.
We don't have to remove them, but doing so has some advantages:
- They are used inconsistently, which is confusing.
- We can then also remove the (currently broken) hack/update-vanity-imports.sh.
- Last but not least, it would be a first step towards avoiding the k8s.io domain.
This commit was generated with
sed -i -e 's;^package \(.*\) // import.*;package \1;' $(git grep -l '^package.*// import' | grep -v 'vendor/')
Everything was included, except for
package labels // import k8s.io/kubernetes/pkg/util/labels
because that package is marked as "read-only".
The SELinuxWarningController does not necessarily need permissions to read
the objects, because it gets them through a shared informer instantiated by
KCM itself, but let's list the permissions for completeness.
This removes the DRAControlPlaneController feature gate, the fields controlled
by it (claim.spec.controller, claim.status.deallocationRequested,
claim.status.allocation.controller, class.spec.suitableNodes), the
PodSchedulingContext type, and all code related to the feature.
The feature gets removed because there is no path towards beta and GA and DRA
with "structured parameters" should be able to replace it.
Having a dedicated ActionType which only gets used when the scheduler itself
already detects some change in the list of generated ResourceClaims of a pod
avoids calling the DRA plugin for unrelated Pod changes.
Retitle the feature to the affirmative ("AllowInsecure...=false") instead of a
double-negative ("Disable$NEWTHING...=false") for clarity
Signed-off-by: Micah Hausler <mhausler@amazon.com>
This is a complete revamp of the original API. Some of the key
differences:
- refocused on structured parameters and allocating devices
- support for constraints across devices
- support for allocating "all" or a fixed amount
of similar devices in a single request
- no class for ResourceClaims, instead individual
device requests are associated with a mandatory
DeviceClass
For the sake of simplicity, optional basic types (ints, strings) where the null
value is the default are represented as values in the API types. This makes Go
code simpler because it doesn't have to check for nil (consumers) and values
can be set directly (producers). The effect is that in protobuf, these fields
always get encoded because `opt` only has an effect for pointers.
The roundtrip test data for v1.29.0 and v1.30.0 changes because of the new
"request" field. This is considered acceptable because the entire `claims`
field in the pod spec is still alpha.
The implementation is complete enough to bring up the apiserver.
Adapting other components follows.
This is in preparation for revamping the resource.k8s.io completely. Because
there will be no support for transitioning from v1alpha2 to v1alpha3, the
roundtrip test data for that API in 1.29 and 1.30 gets removed.
Repeating the version in the import name of the API packages is not really
required. It was done for a while to support simpler grepping for usage of
alpha APIs, but there are better ways for that now. So during this transition,
"resourceapi" gets used instead of "resourcev1alpha3" and the version gets
dropped from informer and lister imports. The advantage is that the next bump
to v1beta1 will affect fewer source code lines.
Only source code where the version really matters (like API registration)
retains the versioned import.
If the node authorizer is active, RBAC rules are not needed. But if it's
disabled, kubelet needs to get permission through RBAC. In contrast to the
authorizer code which is a bit more flexible and isn't directly tied to the
current kubelet implementation (i.e. it allows list+delete instead of just
deletecollection), the RBAC entry is just for what the current kubelet does
because it's a bit easier to change.
When a resource gets deleted during migration, the SVM SSA patch
calls are interpreted as a logical create request. Since the object
from storage is nil, the merged result is just a type meta object,
which lacks a name in the body. This fails when the API server
checks that the name from the request URL and the body are the same.
Note that a create request is something that SVM controller should
never do.
Once the UID is set on the patch, the API server will fail the
request at a slightly earlier point with an "uid mismatch" conflict
error, which the SVM controller can handle gracefully.
Setting UID by itself is not sufficient. When a resource gets
deleted and recreated, if RV is not set but UID is set, we would get
an immutable field validation error for attempting to update the
UID. To address this, we set the resource version on the SSA patch
as well. This will cause that update request to also fail with a
conflict error.
Added the create verb on all resources for SVM controller RBAC as
otherwise the API server will reject the request before it fails
with a conflict error.
The change addresses a host of other issues with the SVM controller:
1. Include failure message in SVM resource
2. Do not block forever on unsynced GC monitor
3. Do not immediately fail on GC monitor being missing, allow for
a grace period since discovery may be out of sync
4. Set higher QPS and burst to handle large migrations
Test changes:
1. Clean up CRD webhook convertor logs
2. Allow SVM tests to be run multiple times to make finding flakes easier
3. Create and delete CRs during CRD test to force out any flakes
4. Add a stress test with multiple parallel migrations
5. Enable RBAC on KAS
6. Run KCM directly to exercise wiring and RBAC
7. Better logs during CRD migration
8. Scan audit logs to confirm SVM controller never creates
Signed-off-by: Monis Khan <mok@microsoft.com>
Adding the required Kubernetes API so that the kubelet can start using
it. This patch also adds the corresponding alpha feature gate as
outlined in KEP 4639.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
2e34e187c9 enabled kubelet to do List and Watch
requests with the caveat that kubelet should better use a field selector (which
it does). The same is now also needed for DeleteCollection because kubelet will
use that to clean up in one operation instead of using multiple.
