One of the important aspects of Kubernetes is its scalability and performance characteristic. As Kubernetes user or operator/administrator of a cluster you would expect to have some guarantees in those areas.

The goal of this doc is to organize the guarantees that Kubernetes provides in these areas.

Our scalability definition is built on two concepts:

• Service Level Indicators
• Service Level Objectives

We require our SLIs/SLOs to have the following properties:

• They are precise and well-defined
  It's extremely important to ensure that both users and us have exactly the same understanding of what we guarantee.
• They are consistent with each other
  This is mostly about using the same terminology, same concepts, etc.
• They are user-oriented
  First, the SLOs we provide need to be things users really care about. Second, they need to be understandable for people not familiar with the system internals (e.g. their formulation can't depend on some arcane knowledge or implementation details of the system).
• They are testable
  Ideally, SLIs/SLOs should be measurable in all running clusters, but if measuring some metrics isn't possible or would be extremely expensive (e.g. in terms of resource overhead for the system), benchmarks sometimes may be enough. That means that not every SLO may be translatable to SLA (Service Level Agreement).

While SLIs are generic (they just define what and how we measure), SLOs provide specific guarantees and satisfying them may depend on meeting some specific requirements. Specific examples that may visibly affect ability to satisfy them are:

• cluster configuration
• user of Kubernetes extensibility features
• load on the cluster.

As a result, we define Kubernetes scalability using "you promise, we promise" framework, as following:

If you promise to:

• all the SLOs are satisfied.

We are in the process of extending coverage of the system with SLIs and SLOs to better reflect user expectations.

Note that may also introduce internal (for developers only) SLIs, that may be useful for understanding performance characteristic of the system, but for which we will not provide any guarantees for users.

In order to meet SLOs, system must run in the environment satisfying the following criteria:

• Runs a single or more appropriately sized master machines
• Events are stored in a separate etcd instance (or cluster)
• All etcd instances are running on master machine(s)
• Kubernetes version is at least X.Y.Z
• ...

TODO: Document other necessary configuration.

To make the cluster eligible for SLO, users also can't have too many objects in their clusters. More concretely, the number of different objects in the cluster MUST satisfy thresholds defined in thresholds file.

In order to meet SLOs, you have to use extensibility features "wisely". The more precise formulation is to-be-defined, but this includes things like:

• webhooks have to provide high availability and low latency
• CRDs and CRs have to be kept within thresholds
• ...

The currently existing SLIs/SLOs are enough to guarantee that cluster isn't completely dead. However, they are not meeting user expectations in many areas of the system and we are actively working on extending their coverage.

We are also introducing two more prerequisites which have to be met to ensure that SLOs can be satisfied:

Prerequisites:
   1. Kubernetes cluster is available and serving.
   2. Cluster churn is <= 20, where churn is defined as:
     churn = #(Pod spec creations/updates/deletions) + #(user originated requests) in a given second

TODO: Cluster churn should be moved to scalability thresholds.

[1] For the purpose of visualization it will be a sliding window. However, for the purpose of SLO itself, it basically means "fraction of good minutes per day" being within threshold.