DRA: ClusterResourceClaimTemplate

[johnbelamaric]

Enhancement Description

One of the challenges with using DRA is that it requires the creation of a separate ResourceClaimTemplate object in addition to the standard workload definition. Often, the ResourceClaimTemplate represents a common pattern that could be used across many workloads. But since it is a namespaced object, a copy must be populated in every namespace that might need to use that pattern. Instead, a cluster-scoped object could be used to allow administrators to capture common and complex patterns once, and let them be re-used across namespaces.

/assign @everpeace
/cc @pohly @klueska @andreyvelich

• One-line enhancement description (can be used as a release note): ClusterResourceClaimTemplates can be defined by the administrator once in the cluster scope, and used by workloads to create namespaced ResourceClaims.
• Kubernetes Enhancement Proposal: TBD
• Discussion Link:
• PRs by stage and milestone:
  • Alpha - v1.xx
    • KEP (k/enhancements) update PR(s):
    • Code (k/k) update PR(s):
    • Docs (k/website) update PR(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[johnbelamaric]

/sg device-management
/sig scheduling

[johnbelamaric]

/wg device-management

[johnbelamaric]

@liggitt Ideally, I'd like to be able to use the existing Pod and PodGroup fields that refer to a ResourceClaimTemplateName and have the controller check for a ClusterResourceClaimTemplate of that name if a namespace-local ResourceClaimTemplate does not exist. Is that a pattern with precedence in our APIs?

[everpeace]

@johnbelamaric Thank you for opening the issue!! This enhancement focuses on something like Day2-ish operational issues. But, I feel this is an important UX/Admin-experience improvement.

For the API design, my initial API idea is:

• keeping ResourceClaimTemplateName field for backward compatibility
• introducing a new ResourceClaimTemplateRef field in the neighbor of ResourceClaimTemplateName so that it can refer to both cluster-scoped and namespace-scoped ones like this:

  • # Case 1: namespace-scoped
    resourceClaimTemplateRef:
      kind: ResourceClaimTemplate
      name: namespaced-resource-claim-template
    # Case 2: cluster-scoped
    resourceClaimTemplateRef:
      kind: ClusterResourceClaimTemplate
      name: cluster-resource-claim-template

And, I'm wondering how we should authorize the ClusterResourceClaimTemplate reference usage. I'm considering we authorize the usage if the pod's serviceaccount can get the ClusterResourceClaimTemplate.

By the way, currently, do we do authorization checks for ResourceClaimTemplateName reference usage??

[liggitt]

@liggitt Ideally, I'd like to be able to use the existing Pod and PodGroup fields that refer to a ResourceClaimTemplateName and have the controller check for a ClusterResourceClaimTemplate of that name if a namespace-local ResourceClaimTemplate does not exist. Is that a pattern with precedence in our APIs?

mmm... not really... ambiguous reference resolution from an object depending on what else exists at a given point in time is pretty hard to reason about (both at admission time, and throughout time if the namespaced referred-to object gets deleted or created in a way that would shadow or unshadow the cluster-scoped object)... that seems likely to lead to race conditions where the namespaced object was not created in the right order (or was not observed by the controller in time) and accidentally resolved to the cluster-scoped object instead

it also would change the meaning of the field in ways that would mislead other existing clients who expected a resourceClaimTemplate pointing at a name which does not exist as a local resourceclaimtemplate is invalid and will not work

[pohly]

I concur with @liggitt .

If for whatever reason someone wants to use their own variant of a cluster-scoped ResourceClaimTemplate, they cannot reliably ensure that the ResourceClaim controller uses "their" namespaced override: if they create all of their objects together, it's racy whether the controller sees the namespaced ResourceClaimTemplate soon enough to use it.

A separate API field (clusterResourceClaimTemplateName) seems safer:

• Dropping the field because support is disabled will lead to a validation errors (empty pod claim struct).
• n-1 clients not knowing or not supporting the field will error out.

Both are desirable properties.