summaryrefslogtreecommitdiff
path: root/node-performance-testing.md
diff options
context:
space:
mode:
authorMike Brown <brownwm@us.ibm.com>2016-05-04 16:14:05 -0500
committerMike Brown <brownwm@us.ibm.com>2016-05-04 17:05:44 -0500
commit59f159feb47c309c45beff80c45521f813b51c3a (patch)
tree7f9d4031d5facce1c5e1710534267d037bfcaeae /node-performance-testing.md
parent13977a3612956ba155a91f43f3f03adfe7a883e8 (diff)
devel/ tree 80col wrap and other minor edits
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
Diffstat (limited to 'node-performance-testing.md')
-rw-r--r--node-performance-testing.md89
1 files changed, 49 insertions, 40 deletions
diff --git a/node-performance-testing.md b/node-performance-testing.md
index ae8789a7..54c15dee 100644
--- a/node-performance-testing.md
+++ b/node-performance-testing.md
@@ -34,69 +34,76 @@ Documentation for other releases can be found at
# Measuring Node Performance
-This document outlines the issues and pitfalls of measuring Node performance, as well as the tools
-available.
+This document outlines the issues and pitfalls of measuring Node performance, as
+well as the tools available.
## Cluster Set-up
-There are lots of factors which can affect node performance numbers, so care must be taken in
-setting up the cluster to make the intended measurements. In addition to taking the following steps
-into consideration, it is important to document precisely which setup was used. For example,
-performance can vary wildly from commit-to-commit, so it is very important to **document which commit
+There are lots of factors which can affect node performance numbers, so care
+must be taken in setting up the cluster to make the intended measurements. In
+addition to taking the following steps into consideration, it is important to
+document precisely which setup was used. For example, performance can vary
+wildly from commit-to-commit, so it is very important to **document which commit
or version** of Kubernetes was used, which Docker version was used, etc.
### Addon pods
-Be aware of which addon pods are running on which nodes. By default Kubernetes runs 8 addon pods,
-plus another 2 per node (`fluentd-elasticsearch` and `kube-proxy`) in the `kube-system`
-namespace. The addon pods can be disabled for more consistent results, but doing so can also have
-performance implications.
+Be aware of which addon pods are running on which nodes. By default Kubernetes
+runs 8 addon pods, plus another 2 per node (`fluentd-elasticsearch` and
+`kube-proxy`) in the `kube-system` namespace. The addon pods can be disabled for
+more consistent results, but doing so can also have performance implications.
-For example, Heapster polls each node regularly to collect stats data. Disabling Heapster will hide
-the performance cost of serving those stats in the Kubelet.
+For example, Heapster polls each node regularly to collect stats data. Disabling
+Heapster will hide the performance cost of serving those stats in the Kubelet.
#### Disabling Add-ons
-Disabling addons is simple. Just ssh into the Kubernetes master and move the addon from
-`/etc/kubernetes/addons/` to a backup location. More details [here](../../cluster/addons/).
+Disabling addons is simple. Just ssh into the Kubernetes master and move the
+addon from `/etc/kubernetes/addons/` to a backup location. More details
+[here](../../cluster/addons/).
### Which / how many pods?
-Performance will vary a lot between a node with 0 pods and a node with 100 pods. In many cases
-you'll want to make measurements with several different amounts of pods. On a single node cluster
-scaling a replication controller makes this easy, just make sure the system reaches a steady-state
-before starting the measurement. E.g. `kubectl scale replicationcontroller pause --replicas=100`
+Performance will vary a lot between a node with 0 pods and a node with 100 pods.
+In many cases you'll want to make measurements with several different amounts of
+pods. On a single node cluster scaling a replication controller makes this easy,
+just make sure the system reaches a steady-state before starting the
+measurement. E.g. `kubectl scale replicationcontroller pause --replicas=100`
-In most cases pause pods will yield the most consistent measurements since the system will not be
-affected by pod load. However, in some special cases Kubernetes has been tuned to optimize pods that
-are not doing anything, such as the cAdvisor housekeeping (stats gathering). In these cases,
-performing a very light task (such as a simple network ping) can make a difference.
+In most cases pause pods will yield the most consistent measurements since the
+system will not be affected by pod load. However, in some special cases
+Kubernetes has been tuned to optimize pods that are not doing anything, such as
+the cAdvisor housekeeping (stats gathering). In these cases, performing a very
+light task (such as a simple network ping) can make a difference.
-Finally, you should also consider which features yours pods should be using. For example, if you
-want to measure performance with probing, you should obviously use pods with liveness or readiness
-probes configured. Likewise for volumes, number of containers, etc.
+Finally, you should also consider which features yours pods should be using. For
+example, if you want to measure performance with probing, you should obviously
+use pods with liveness or readiness probes configured. Likewise for volumes,
+number of containers, etc.
### Other Tips
-**Number of nodes** - On the one hand, it can be easier to manage logs, pods, environment etc. with
- a single node to worry about. On the other hand, having multiple nodes will let you gather more
- data in parallel for more robust sampling.
+**Number of nodes** - On the one hand, it can be easier to manage logs, pods,
+environment etc. with a single node to worry about. On the other hand, having
+multiple nodes will let you gather more data in parallel for more robust
+sampling.
## E2E Performance Test
-There is an end-to-end test for collecting overall resource usage of node components:
-[kubelet_perf.go](../../test/e2e/kubelet_perf.go). To
-run the test, simply make sure you have an e2e cluster running (`go run hack/e2e.go -up`) and
-[set up](#cluster-set-up) correctly.
+There is an end-to-end test for collecting overall resource usage of node
+components: [kubelet_perf.go](../../test/e2e/kubelet_perf.go). To
+run the test, simply make sure you have an e2e cluster running (`go run
+hack/e2e.go -up`) and [set up](#cluster-set-up) correctly.
Run the test with `go run hack/e2e.go -v -test
---test_args="--ginkgo.focus=resource\susage\stracking"`. You may also wish to customise the number of
-pods or other parameters of the test (remember to rerun `make WHAT=test/e2e/e2e.test` after you do).
+--test_args="--ginkgo.focus=resource\susage\stracking"`. You may also wish to
+customise the number of pods or other parameters of the test (remember to rerun
+`make WHAT=test/e2e/e2e.test` after you do).
## Profiling
-Kubelet installs the [go pprof handlers](https://golang.org/pkg/net/http/pprof/), which can be
-queried for CPU profiles:
+Kubelet installs the [go pprof handlers]
+(https://golang.org/pkg/net/http/pprof/), which can be queried for CPU profiles:
```console
$ kubectl proxy &
@@ -109,13 +116,15 @@ $ go tool pprof -web $KUBELET_BIN $OUTPUT
`pprof` can also provide heap usage, from the `/debug/pprof/heap` endpoint
(e.g. `http://localhost:8001/api/v1/proxy/nodes/${NODE}:10250/debug/pprof/heap`).
-More information on go profiling can be found [here](http://blog.golang.org/profiling-go-programs).
+More information on go profiling can be found
+[here](http://blog.golang.org/profiling-go-programs).
## Benchmarks
-Before jumping through all the hoops to measure a live Kubernetes node in a real cluster, it is
-worth considering whether the data you need can be gathered through a Benchmark test. Go provides a
-really simple benchmarking mechanism, just add a unit test of the form:
+Before jumping through all the hoops to measure a live Kubernetes node in a real
+cluster, it is worth considering whether the data you need can be gathered
+through a Benchmark test. Go provides a really simple benchmarking mechanism,
+just add a unit test of the form:
```go
// In foo_test.go