Using SkyPilot with KAI Scheduler#
KAI Scheduler is a Kubernetes scheduler with hierarchical queues, quotas and gang scheduling. SkyPilot can submit each cluster to a KAI queue as a gang, so all of its pods are scheduled together.
Prerequisites#
KAI Scheduler, with its default scheduler name
kai-scheduler.A leaf KAI
Queueto submit to. KAI never schedules aPodGroupin a parent queue, so a launch into one waits until it times out.Permission to get, create and delete
podgroups.scheduling.run.aiin the namespace SkyPilot launches into. If you set a topology, also permission to gettopologies.kai.scheduler, which needs aClusterRolebecause Topologies are cluster-scoped.
Setting a queue#
Set the queue in the API server’s ~/.sky/config.yaml, for one context or
for all contexts:
kubernetes:
context_configs:
my-context:
kai:
queue: team-a
preemptibility: preemptible # Optional.
A task can also set the queue under config.kubernetes.kai.queue. It wins
over a queue set for all contexts. It loses to one set for its context, unless
its workspace has an entry under workspaces. To fix a team’s queue, set it
per workspace, as below.
Giving each team its own queue#
Set the queue for a context in each workspace. A task cannot override it, so the workspace’s members launch into that queue:
workspaces:
team-a:
kubernetes:
context_configs:
my-context:
kai:
queue: team-a
This chooses the queue SkyPilot submits to. It does not restrict access to
other queues: a task’s pod_config can still change the pods’ scheduling
fields. To enforce queue access, use a policy on the cluster or an
admin policy.
Scheduling shards#
If your KAI installation runs scheduling shards, give the cluster the shard’s label with custom_metadata.
A shard only schedules
PodGroupobjects that carry its label.KAI’s default shard only schedules
PodGroupobjects without a shard label.The queue must be in the same shard.
A PodGroup that no shard selects is never evaluated. Its launch waits for
the full provision_timeout,
which is 24 hours by default with a KAI queue.
Keeping pods close together#
To keep a cluster’s pods close together in the network, name a KAI
Topology and one of its levels:
kubernetes:
context_configs:
my-context:
kai:
queue: team-a
topology: fabric
preferred_topology_level: leaf
preferred_topology_level: KAI places all of the cluster’s pods in one domain at that level if it can. Otherwise it uses one domain at the lowest broader level that has room, up to the whole cluster.required_topology_level: KAI places all of the cluster’s pods in one domain at that level, and the cluster stays queued until a domain has room. KAI does not tell a full domain apart from one too small for the cluster, so a cluster larger than every domain at that level stays queued until its launch times out.
Name a level by its node label or alias. The preferred level must be at or
below the required one. SkyPilot checks both against the Topology before
it creates the PodGroup. A cluster with a topology only uses nodes that
carry a label for every level of the Topology.
How a launch works#
SkyPilot creates a
PodGroupfor the cluster before its pods, withminMemberset to the number of nodes. ThePodGrouphas the same labels as the pods, so labels set with custom_metadata apply to both.The launch waits while KAI keeps the cluster queued. If KAI has not scheduled it within provision_timeout, the launch fails over to the next allowed context, if any.
sky downdeletes the cluster’s pods and itsPodGroup.
Relaunching onto a running cluster cannot change its queue, priority class,
preemptibility or topology. Run sky down first.
sky launch --resize changes the PodGroup’s minMember with the
cluster. KAI evicts a running gang that falls below minMember, so a
scale-down lowers it before removing workers, and a scale-up raises it once the
new pods are scheduled. Until then, KAI schedules the new pods individually.
Jobs and serve controllers are not submitted to KAI, so their pods are not subject to queue limits or preemption.
Priority and preemption#
KAI orders and preempts the gang by the PodGroup’s PriorityClass.
SkyPilot sets it from resources.priority_class, on the PodGroup and on
the pods, so the PriorityClass must exist. Unless
kubernetes.kai.preemptibility
is set, a priority below 100 is preemptible.
Eviction and recovery#
When KAI evicts a gang, by reclaim, preemption or consolidation, it deletes the cluster’s pods.
sky status --refreshthen removes the cluster, and a managed job recovers onto a new one.A pod that SkyPilot finds while KAI deletes it is reported as evicted by KAI, with KAI’s reason.
If a pod of a running cluster fails, KAI evicts the rest of the gang after its staleness grace period, 60 seconds by default.
Cancelling a
sky launchthat KAI keeps queued leaves the cluster’s pods in KAI’s queue, where KAI can still schedule them. Runsky downto remove them. Cancelling a managed job removes them.
Troubleshooting#
kubectl describe podgroup <name>shows KAI’s full scheduling conditions. ThePodGroupis named like the cluster’s pods, without the-headsuffix.A queue’s GPU
limitof-1is unlimited. An unset limit is 0, so KAI never schedules a GPU cluster in that queue.
Limitations#
KAI and Kueue cannot be used for the same cluster.