diff options
| author | Mike Brown <brownwm@us.ibm.com> | 2016-05-04 16:14:05 -0500 |
|---|---|---|
| committer | Mike Brown <brownwm@us.ibm.com> | 2016-05-04 17:05:44 -0500 |
| commit | 59f159feb47c309c45beff80c45521f813b51c3a (patch) | |
| tree | 7f9d4031d5facce1c5e1710534267d037bfcaeae /node-performance-testing.md | |
| parent | 13977a3612956ba155a91f43f3f03adfe7a883e8 (diff) | |
devel/ tree 80col wrap and other minor edits
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
Diffstat (limited to 'node-performance-testing.md')
| -rw-r--r-- | node-performance-testing.md | 89 |
1 files changed, 49 insertions, 40 deletions
diff --git a/node-performance-testing.md b/node-performance-testing.md index ae8789a7..54c15dee 100644 --- a/node-performance-testing.md +++ b/node-performance-testing.md @@ -34,69 +34,76 @@ Documentation for other releases can be found at # Measuring Node Performance -This document outlines the issues and pitfalls of measuring Node performance, as well as the tools -available. +This document outlines the issues and pitfalls of measuring Node performance, as +well as the tools available. ## Cluster Set-up -There are lots of factors which can affect node performance numbers, so care must be taken in -setting up the cluster to make the intended measurements. In addition to taking the following steps -into consideration, it is important to document precisely which setup was used. For example, -performance can vary wildly from commit-to-commit, so it is very important to **document which commit +There are lots of factors which can affect node performance numbers, so care +must be taken in setting up the cluster to make the intended measurements. In +addition to taking the following steps into consideration, it is important to +document precisely which setup was used. For example, performance can vary +wildly from commit-to-commit, so it is very important to **document which commit or version** of Kubernetes was used, which Docker version was used, etc. ### Addon pods -Be aware of which addon pods are running on which nodes. By default Kubernetes runs 8 addon pods, -plus another 2 per node (`fluentd-elasticsearch` and `kube-proxy`) in the `kube-system` -namespace. The addon pods can be disabled for more consistent results, but doing so can also have -performance implications. +Be aware of which addon pods are running on which nodes. By default Kubernetes +runs 8 addon pods, plus another 2 per node (`fluentd-elasticsearch` and +`kube-proxy`) in the `kube-system` namespace. The addon pods can be disabled for +more consistent results, but doing so can also have performance implications. -For example, Heapster polls each node regularly to collect stats data. Disabling Heapster will hide -the performance cost of serving those stats in the Kubelet. +For example, Heapster polls each node regularly to collect stats data. Disabling +Heapster will hide the performance cost of serving those stats in the Kubelet. #### Disabling Add-ons -Disabling addons is simple. Just ssh into the Kubernetes master and move the addon from -`/etc/kubernetes/addons/` to a backup location. More details [here](../../cluster/addons/). +Disabling addons is simple. Just ssh into the Kubernetes master and move the +addon from `/etc/kubernetes/addons/` to a backup location. More details +[here](../../cluster/addons/). ### Which / how many pods? -Performance will vary a lot between a node with 0 pods and a node with 100 pods. In many cases -you'll want to make measurements with several different amounts of pods. On a single node cluster -scaling a replication controller makes this easy, just make sure the system reaches a steady-state -before starting the measurement. E.g. `kubectl scale replicationcontroller pause --replicas=100` +Performance will vary a lot between a node with 0 pods and a node with 100 pods. +In many cases you'll want to make measurements with several different amounts of +pods. On a single node cluster scaling a replication controller makes this easy, +just make sure the system reaches a steady-state before starting the +measurement. E.g. `kubectl scale replicationcontroller pause --replicas=100` -In most cases pause pods will yield the most consistent measurements since the system will not be -affected by pod load. However, in some special cases Kubernetes has been tuned to optimize pods that -are not doing anything, such as the cAdvisor housekeeping (stats gathering). In these cases, -performing a very light task (such as a simple network ping) can make a difference. +In most cases pause pods will yield the most consistent measurements since the +system will not be affected by pod load. However, in some special cases +Kubernetes has been tuned to optimize pods that are not doing anything, such as +the cAdvisor housekeeping (stats gathering). In these cases, performing a very +light task (such as a simple network ping) can make a difference. -Finally, you should also consider which features yours pods should be using. For example, if you -want to measure performance with probing, you should obviously use pods with liveness or readiness -probes configured. Likewise for volumes, number of containers, etc. +Finally, you should also consider which features yours pods should be using. For +example, if you want to measure performance with probing, you should obviously +use pods with liveness or readiness probes configured. Likewise for volumes, +number of containers, etc. ### Other Tips -**Number of nodes** - On the one hand, it can be easier to manage logs, pods, environment etc. with - a single node to worry about. On the other hand, having multiple nodes will let you gather more - data in parallel for more robust sampling. +**Number of nodes** - On the one hand, it can be easier to manage logs, pods, +environment etc. with a single node to worry about. On the other hand, having +multiple nodes will let you gather more data in parallel for more robust +sampling. ## E2E Performance Test -There is an end-to-end test for collecting overall resource usage of node components: -[kubelet_perf.go](../../test/e2e/kubelet_perf.go). To -run the test, simply make sure you have an e2e cluster running (`go run hack/e2e.go -up`) and -[set up](#cluster-set-up) correctly. +There is an end-to-end test for collecting overall resource usage of node +components: [kubelet_perf.go](../../test/e2e/kubelet_perf.go). To +run the test, simply make sure you have an e2e cluster running (`go run +hack/e2e.go -up`) and [set up](#cluster-set-up) correctly. Run the test with `go run hack/e2e.go -v -test ---test_args="--ginkgo.focus=resource\susage\stracking"`. You may also wish to customise the number of -pods or other parameters of the test (remember to rerun `make WHAT=test/e2e/e2e.test` after you do). +--test_args="--ginkgo.focus=resource\susage\stracking"`. You may also wish to +customise the number of pods or other parameters of the test (remember to rerun +`make WHAT=test/e2e/e2e.test` after you do). ## Profiling -Kubelet installs the [go pprof handlers](https://golang.org/pkg/net/http/pprof/), which can be -queried for CPU profiles: +Kubelet installs the [go pprof handlers] +(https://golang.org/pkg/net/http/pprof/), which can be queried for CPU profiles: ```console $ kubectl proxy & @@ -109,13 +116,15 @@ $ go tool pprof -web $KUBELET_BIN $OUTPUT `pprof` can also provide heap usage, from the `/debug/pprof/heap` endpoint (e.g. `http://localhost:8001/api/v1/proxy/nodes/${NODE}:10250/debug/pprof/heap`). -More information on go profiling can be found [here](http://blog.golang.org/profiling-go-programs). +More information on go profiling can be found +[here](http://blog.golang.org/profiling-go-programs). ## Benchmarks -Before jumping through all the hoops to measure a live Kubernetes node in a real cluster, it is -worth considering whether the data you need can be gathered through a Benchmark test. Go provides a -really simple benchmarking mechanism, just add a unit test of the form: +Before jumping through all the hoops to measure a live Kubernetes node in a real +cluster, it is worth considering whether the data you need can be gathered +through a Benchmark test. Go provides a really simple benchmarking mechanism, +just add a unit test of the form: ```go // In foo_test.go |
