diff options
| author | Qiang Wu <qiang.q.wu@oracle.com> | 2018-01-09 13:04:08 +0800 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2018-01-09 13:04:08 +0800 |
| commit | ca6d798a2171dad8283b437755a0184948a3ea03 (patch) | |
| tree | e65b6f3dad7c4b51db848c793ca4568dcae41434 | |
| parent | 121935fa24e31786816181d4bc0426dffd22ddb6 (diff) | |
| parent | ab26472187fa8a0b02b80e67e71c6ffe7a6d9647 (diff) | |
Merge pull request #3 from wu-qiang/kms-plugin-grpc-api
gRPC-based KMS plugin service
583 files changed, 33237 insertions, 7184 deletions
diff --git a/.generated_files b/.generated_files index a4349240..454f7dd5 100644 --- a/.generated_files +++ b/.generated_files @@ -1,7 +1,7 @@ # Files that should be ignored by tools which do not want to consider generated # code. # -# https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/size.go +# https://git.k8s.io/contrib/mungegithub/mungers/size.go # # This file is a series of lines, each of the form: # <type> <name> diff --git a/.github/PULL_REQUEST_TEMPLATE b/.github/PULL_REQUEST_TEMPLATE new file mode 100644 index 00000000..ee56fb31 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE @@ -0,0 +1,7 @@ +<!-- Thanks for sending a pull request! Here are some tips for you: +- If this is your first contribution, read our Getting Started guide https://github.com/kubernetes/community#your-first-contribution +- If you are editing SIG information, please follow these instructions: https://git.k8s.io/community/generator + You will need to follow these steps: + 1. Edit sigs.yaml with your change + 2. Generate docs with `make generate`. To build docs for one sig, run `make WHAT=sig-apps generate` +-->
\ No newline at end of file diff --git a/.gitignore b/.gitignore new file mode 100644 index 00000000..5e438b15 --- /dev/null +++ b/.gitignore @@ -0,0 +1,39 @@ +# OSX leaves these everywhere on SMB shares +._* + +# OSX trash +.DS_Store + +# Eclipse files +.classpath +.project +.settings/** + +# Files generated by JetBrains IDEs, e.g. IntelliJ IDEA +.idea/ +*.iml + +# Vscode files +.vscode + +# Emacs save files +*~ +\#*\# +.\#* + +# Vim-related files +[._]*.s[a-w][a-z] +[._]s[a-w][a-z] +*.un~ +Session.vim +.netrwhist + +# JUnit test output from ginkgo e2e tests +/junit*.xml + +# Mercurial files +**/.hg +**/.hg* + +# direnv .envrc files +.envrc @@ -1,42 +1,68 @@ # The Contributor License Agreement -The [Cloud Native Computing Foundation][CNCF] defines the legal status of the -contributed code in a _Contributor License Agreement_ (CLA). +The [Cloud Native Computing Foundation](https://www.cncf.io/community) defines +the legal status of the contributed code in a _Contributor License Agreement_ +(CLA). Only original source code from CLA signatories can be accepted into kubernetes. -This policy does not apply to [third_party] and [vendor]. +This policy does not apply to [third_party](https://git.k8s.io/kubernetes/third_party) +and [vendor](https://git.k8s.io/kubernetes/vendor). -## How do I sign? +## What am I agreeing to? + +There are two versions of the CLA: -#### 1. Read +1. One for [individual contributors](https://github.com/cncf/cla/blob/master/individual-cla.pdf) +submitting contributions on their own behalf. +1. One for [corporations](https://github.com/cncf/cla/blob/master/corporate-cla.pdf) +to sign for contributions submitted by their employees. - * [CLA for individuals] to sign up as an individual or as an employee of a signed organization. - * [CLA for corporations] to sign as a corporation representative and manage signups from your organization. - -#### 2. Sign in with GitHub. +It is important to read and understand this legal agreement. + +## How do I sign? -Click - * [Individual signup] to sign up as an individual or as an employee of a signed organization. - * [Corp signup] to sign as a corporation representative and manage signups from your organization. +#### 1. Log into the Linux Foundation ID Portal with Github -Either signup form looks like this: +Click one of: + * [Individual signup](https://identity.linuxfoundation.org/projects/cncf) to + sign up as an individual or as an employee of a signed organization. + * [Corporation signup](https://identity.linuxfoundation.org/node/285/organization-signup) + to sign as a corporation representative and manage signups from your organization. + +Once you get to the sign in form, click "Log in with Github":  -#### 3. Enter the correct E-mail address to validate! +#### 2. Create Linux Foundation ID Portal account with correct e-mail address -The address entered on the form must meet two constraints: - - * It __must match__ your [git email] (the output of `git config user.email`) - or your PRs will not be approved! +Ensure that the e-mail address you use when completing this form matches the one +you will use for your commits. - * It must be your official `person@organization.com` address if you signed up - as an employee of said organization. +If you are signing up as an employee, you must use your official +person@organization.domain email address in the CNCF account registration page.  -#### 4. Look for an email indicating successful signup. +#### 3. Complete signing process + +Once you have created your account, follow the instructions to complete the +signing process via Hellosign. + +#### 4. Ensure your Github e-mail address matches address used to sign CLA + +Your Github email address __must match__ the same address you use when signing +the CLA. Github has [documentation](https://help.github.com/articles/setting-your-commit-email-address-on-github/) +on setting email addresses. + +You must also set your [git e-mail](https://help.github.com/articles/setting-your-email-in-git) +to match this e-mail address as well. + +If you've already submitted a PR you can correct your user.name and user.email +and then use use `git commit --ammend --reset-author` and then `git push` to +correct the PR. + +#### 5. Look for an email indicating successful signup. > The Linux Foundation > @@ -50,21 +76,8 @@ Once you have this, the CLA authorizer bot will authorize your PRs.  - ## Troubleshooting -If you have signup trouble, please explain your case on -the [CLA signing issue] and we (@sarahnovotny and @foxish), -along with the [CNCF] will help sort it out. - -Another option: ask for help at `helpdesk@rt.linuxfoundation.org`. - -[CNCF]: https://www.cncf.io/community -[CLA signing issue]: https://github.com/kubernetes/kubernetes/issues/27796 -[CLA for individuals]: https://github.com/cncf/cla/blob/master/individual-cla.pdf -[CLA for corporations]: https://github.com/cncf/cla/blob/master/corporate-cla.pdf -[Corp signup]: https://identity.linuxfoundation.org/node/285/organization-signup -[Individual signup]: https://identity.linuxfoundation.org/projects/cncf -[git email]: https://help.github.com/articles/setting-your-email-in-git -[third_party]: https://github.com/kubernetes/kubernetes/tree/master/third_party -[vendor]: https://github.com/kubernetes/kubernetes/tree/master/vendor
\ No newline at end of file +If you are having problems with signed the CLA send a mail to: `helpdesk@rt.linuxfoundation.org`. + +Someone from the CNCF will respond to your ticket to help. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0932ac2d..102ff49f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,17 +1,39 @@ -# Contributing to the community repo +# Contributing to the Community Repo -Make a [pull request](https://help.github.com/articles/using-pull-requests) (PR). +Welcome to the Kubernetes Community contributing guide. We are excited about the prospect of you joining our [community](https://github.com/kubernetes/community)! -Upon successful review, someone will give the PR -a __LGTM__ (_looks good to me_) in the review thread. +## Finding Something to Work On + +Before you begin, you can look at some places where you can help out: + +- [Existing open issues](https://github.com/kubernetes/community/issues) +- [Open Pull Requests](https://github.com/kubernetes/community/pulls) - Most of the pull requests in this repository are not code, but documentation and process. This is a great place to get started. Even if you do not have the rights to merge you can still help by doing a review. -A [SIG lead](sig-list.md) (or someone with approval powers +## Steps + +1. Make a [Pull Request](https://help.github.com/articles/using-pull-requests) (PR). +2. Upon successful review, someone will give the PR +a __LGTM__ (_looks good to me_) in the review thread. +3. A [SIG lead](sig-list.md) (or someone with approval powers as specified in an OWNERS file) may merge the PR immediately with or without an LGTM from someone else. Or they may wait a business day to get further feedback from other reviewers. +### Trivial Edits + +Each incoming Pull Request needs to be reviewed, checked, and then merged. +While automation helps with this, each contribution also has an engineering cost. Therefore it is appreciated if you do NOT make trivial edits and fixes, but instead focus on giving the entire file a review. +If you find one grammatical or spelling error, it is likely there are more in that file, you can really make your Pull Request count by checking formatting, checking for broken links, and fixing errors and then submitting all the fixes at once to that file. +Some questions to consider: +- Can the file be improved further? +- Does the trivial edit greatly improve the quality of the content? + +## Contributing to Individual SIGs + +Each SIG may or may not have it's own policies for editing their section of this repository. + Edits in SIG sub-directories should follow any additional guidelines described by the respective SIG leads in the sub-directory's `CONTRIBUTING` file (e.g. [sig-cli/CONTRIBUTING](sig-cli/CONTRIBUTING.md)). -
\ No newline at end of file +Attending a [SIG meeting](/sig-list.md) or posting on their mailing list might be prudent if you want to make extensive contributions. @@ -1,21 +1,24 @@ -IMAGE_NAME=kube-communitydocs +IMAGE_NAME=golang:1.9 default: \ generate \ reset-docs: - git checkout HEAD -- sig-list.md sig-*/README.md + git checkout HEAD -- ./sig-list.md ./sig-*/README.md ./wg-*/README.md -build-image: - docker build -q -t $(IMAGE_NAME) -f generator/Dockerfile generator +generate: + go run ./generator/app.go -generate: build-image - docker run --rm -e WG -e SIG -v $(shell pwd):/go/src/app/generated:Z $(IMAGE_NAME) app +generate-dockerized: + docker run --rm -e WHAT -v $(shell pwd):/go/src/app:Z $(IMAGE_NAME) make -C /go/src/app generate verify: @hack/verify.sh -test: build-image - docker run --rm $(IMAGE_NAME) go test -v ./... +test: + go test -v ./generator/... -.PHONY: default reset-docs build-image generate verify test +test-dockerized: + docker run --rm -v $(shell pwd):/go/src/app:Z $(IMAGE_NAME) make -C /go/src/app test + +.PHONY: default reset-docs generate generate-dockerized verify test test-dockerized @@ -1,18 +1,22 @@ reviewers: - - sarahnovotny - - idvoretskyi - calebamiles + - castrojo + - cblecker - grodrigues3 -approvers: - - sarahnovotny - idvoretskyi - - calebamiles - - grodrigues3 + - sarahnovotny +approvers: - brendandburns + - calebamiles + - castrojo + - cblecker - dchen1107 + - grodrigues3 + - idvoretskyi - jbeda - lavalamp + - sarahnovotny - smarterclayton + - spiffxp - thockin - wojtek-t - diff --git a/OWNERS_ALIASES b/OWNERS_ALIASES new file mode 100644 index 00000000..242d2bc5 --- /dev/null +++ b/OWNERS_ALIASES @@ -0,0 +1,117 @@ +aliases: + sig-api-machinery-leads: + - lavalamp + - deads2k + sig-apps-leads: + - michelleN + - mattfarina + - prydonius + sig-architecture-leads: + - bgrant0607 + - jdumars + sig-auth-leads: + - ericchiang + - liggitt + - deads2k + sig-autoscaling-leads: + - mwielgus + - directxman12 + sig-aws-leads: + - justinsb + - kris-nova + - chrislovecnm + - mfburnett + sig-azure-leads: + - slack + - colemickens + - jdumars + sig-big-data-leads: + - foxish + - erikerlandson + sig-cli-leads: + - fabianofranz + - pwittrock + - AdoHe + sig-cluster-lifecycle-leads: + - lukemarsden + - jbeda + - roberthbailey + - luxas + sig-cluster-ops-leads: + - zehicle + - jdumars + sig-contributor-experience-leads: + - grodrigues3 + - Phillels + sig-docs-leads: + - devin-donnelly + - jaredbhatti + sig-gcp-leads: + - abgworrall + sig-multicluster-leads: + - csbell + - quinton-hoole + sig-instrumentation-leads: + - piosz + - fabxc + sig-network-leads: + - thockin + - dcbw + - caseydavenport + sig-node-leads: + - dchen1107 + - derekwaynecarr + sig-on-premise-leads: + - marcoceppi + - dghubble + sig-openstack-leads: + - idvoretskyi + - xsgordon + sig-product-management-leads: + - apsinha + - idvoretskyi + - calebamiles + sig-release-leads: + - pwittrock + - calebamiles + sig-scalability-leads: + - wojtek-t + - countspongebob + - jbeda + sig-scheduling-leads: + - davidopp + - timothysc + sig-service-catalog-leads: + - pmorie + - arschles + - vaikas-google + - duglin + sig-storage-leads: + - saad-ali + - childsb + sig-testing-leads: + - spiffxp + - fejta + - stevekuznetsov + - timothysc + sig-ui-leads: + - danielromlein + - floreks + sig-windows-leads: + - michmike + wg-resource-management-leads: + - vishh + - derekwaynecarr + wg-container-identity-leads: + - smarterclayton + - destijl + wg-kubeadm-adoption-leads: + - luxas + - justinsb + wg-cluster-api-leads: + - kris-nova + - pipejakob + - roberthbailey + wg-app-def-leads: + - ant31 + - sebgoa @@ -14,21 +14,24 @@ For more specific topics, try a SIG. ## SIGs Kubernetes is a set of projects, each shepherded by a special interest group (SIG). - + A first step to contributing is to pick from the [list of kubernetes SIGs](sig-list.md). -A SIG can have its own policy for contribution, +A SIG can have its own policy for contribution, described in a `README` or `CONTRIBUTING` file in the SIG folder in this repo (e.g. [sig-cli/CONTRIBUTING](sig-cli/CONTRIBUTING.md)), and its own mailing list, slack channel, etc. - + +If you want to edit details about a SIG (e.g. its weekly meeting time or its leads), +please follow [these instructions](./generator) that detail how our docs are auto-generated. + ## How Can I Help? Documentation (like the text you are reading now) can always use improvement! -There's a [semi-curated list of issues][help-wanted] -that should not need deep knowledge of the system. +There's a [semi-curated list of issues][help wanted] +that should not need deep knowledge of the system. To dig deeper, read a design doc, e.g. [architecture]. @@ -52,14 +55,14 @@ lead to many relevant topics, including ## Your First Contribution We recommend that you work on an existing issue before attempting -to [develop a new feature]. +to [develop a new feature]. -Start by finding an existing issue with the [help-wanted] label; -these are issues we've deemed are well suited for new contributors. -Alternatively, if there is a specific area you are interested in, +Start by finding an existing issue with the [help wanted] label; +these issues we've deemed are well suited for new contributors. +Alternatively, if there is a specific area you are interested in, ask a [SIG lead](sig-list.md) for suggestions), and respond on the -issue thread expressing interest in working on it. - +issue thread expressing interest in working on it. + This helps other people know that the issue is active, and hopefully prevents duplicated efforts. @@ -75,15 +78,14 @@ If you want to work on a new idea of relatively small scope: 1. Submit a [pull request] containing a tested change. -[architecture]: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture.md -[cmd]: https://github.com/kubernetes/kubernetes/tree/master/cmd +[architecture]: /contributors/design-proposals/architecture/architecture.md +[cmd]: https://git.k8s.io/kubernetes/cmd [CLA]: CLA.md [Collaboration Guide]: contributors/devel/collab.md [Developer's Guide]: contributors/devel/development.md [develop a new feature]: https://github.com/kubernetes/features [expectations]: contributors/devel/community-expectations.md -[help-wanted]: https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Ahelp-wanted +[help wanted]: https://go.k8s.io/help-wanted [pull request]: contributors/devel/pull-requests.md []() - diff --git a/code-of-conduct.md b/code-of-conduct.md new file mode 100644 index 00000000..79b33b4d --- /dev/null +++ b/code-of-conduct.md @@ -0,0 +1,3 @@ +# Kubernetes Community Code of Conduct + +Kubernetes follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md). diff --git a/communication.md b/communication.md index 34d7a74a..37c3c2af 100644 --- a/communication.md +++ b/communication.md @@ -20,18 +20,15 @@ and meetings devoted to Kubernetes. ## Social Media * [Twitter] -* [Google+] -* [blog] -* Pose questions and help answer them on [Slack][slack.k8s.io] or [Stack Overflow]. +* [Blog] +* Pose questions and help answer them on [Stack Overflow]. +* [Slack] - sign up -Most real time discussion happens at [kubernetes.slack.com]; -you can sign up at [slack.k8s.io]. - +Real time discussion at kubernetes.slack.io: Discussions on most channels are archived at [kubernetes.slackarchive.io]. Start archiving by inviting the _slackarchive_ bot to a channel via `/invite @slackarchive`. -To add new channels, contact one of the admins -(briangrant, goltermann, jbeda, sarahnovotny and thockin). +To add new channels, contact one of the admins in the #slack-admins channel. Our guidelines are [here](/communication/slack-guidelines.md). ## Issues @@ -50,10 +47,13 @@ Development announcements and discussions appear on the Google group Users trade notes on the Google group [kubernetes-users] (send mail to `kubernetes-users@googlegroups.com`). +## Office Hours + +Office hours are held once a month. Please refer to [this document](events/office-hours.md) to learn more. ## Weekly Meeting -We have PUBLIC and RECORDED [weekly meeting] every Thursday at 10am US Pacific Time. +We have PUBLIC and RECORDED [weekly meeting] every Thursday at 10am US Pacific Time over Zoom. Map that to your local time with this [timezone table]. @@ -71,11 +71,11 @@ please propose a specific date on the [Kubernetes Community Meeting Agenda]. Kubernetes is the main focus of CloudNativeCon/KubeCon, held every spring in Europe and winter in North America. Information about these and other community events is available on the CNCF [events] pages. -[blog]: http://blog.kubernetes.io +[Blog]: http://blog.kubernetes.io [calendar.google.com]: https://calendar.google.com/calendar/embed?src=cgnt364vd8s86hr2phapfjc6uk%40group.calendar.google.com&ctz=America/Los_Angeles [CNCF code of conduct]: https://github.com/cncf/foundation/blob/master/code-of-conduct.md -[communication]: https://github.com/kubernetes/community/blob/master/communication.md -[community meeting]: https://github.com/kubernetes/community/blob/master/communication.md#weekly-meeting +[communication]: /communication.md +[community meeting]: /communication.md#weekly-meeting [events]: https://www.cncf.io/events/ [file an issue]: https://github.com/kubernetes/kubernetes/issues/new [Google+]: https://plus.google.com/u/0/b/116512812300813784482/116512812300813784482 @@ -84,10 +84,10 @@ Kubernetes is the main focus of CloudNativeCon/KubeCon, held every spring in Eur [kubernetes-community-video-chat]: https://groups.google.com/forum/#!forum/kubernetes-community-video-chat [kubernetes-dev]: https://groups.google.com/forum/#!forum/kubernetes-dev [kubernetes-users]: https://groups.google.com/forum/#!forum/kubernetes-users -[kubernetes.slackarchive.io]: http://kubernetes.slackarchive.io -[kubernetes.slack.com]: http://kubernetes.slack.com -[Special Interest Group]: https://github.com/kubernetes/community/blob/master/README.md#SIGs -[slack.k8s.io]: http://slack.k8s.io +[kubernetes.slackarchive.io]: https://kubernetes.slackarchive.io +[kubernetes.slack.com]: https://kubernetes.slack.com +[Slack]: slack.k8s.io +[Special Interest Group]: /README.md#SIGs [Stack Overflow]: http://stackoverflow.com/questions/tagged/kubernetes [timezone table]: https://www.google.com/search?q=1000+am+in+pst [troubleshooting guide]: http://kubernetes.io/docs/troubleshooting diff --git a/community/K8sYoutubeCollaboration.md b/communication/K8sYoutubeCollaboration.md index d46c21a2..d46c21a2 100644 --- a/community/K8sYoutubeCollaboration.md +++ b/communication/K8sYoutubeCollaboration.md diff --git a/communication/OWNERS b/communication/OWNERS new file mode 100644 index 00000000..30847803 --- /dev/null +++ b/communication/OWNERS @@ -0,0 +1,12 @@ +reviewers: + - parispittman + - castrojo + - jdumars + - idvoretskyi + - cblecker +approvers: + - cblecker + - castrojo + - sig-contributor-experience-leads +labels: + - sig/contributor-experience diff --git a/communication/README.md b/communication/README.md new file mode 100644 index 00000000..a577084c --- /dev/null +++ b/communication/README.md @@ -0,0 +1 @@ +[placeholder for the future home of communication.md] diff --git a/communication/slack-guidelines.md b/communication/slack-guidelines.md new file mode 100644 index 00000000..5a869da8 --- /dev/null +++ b/communication/slack-guidelines.md @@ -0,0 +1,85 @@ +# SLACK GUIDELINES + +Slack is the main communication platform for Kubernetes outside of our mailing lists. It’s important that conversation stays on topic in each channel, and that everyone abides by the Code of Conduct. We have over 30,000 members who should all expect to have a positive experience. + +Chat is searchable and public. Do not make comments that you would not say on a video recording or in another public space. Please be courteous to others. + +`@here` and `@channel` should be used rarely. Members will receive notifications from these commands and we are a global project - please be kind. Note: `@all` is only to be used by admins. + +## CODE OF CONDUCT +Kubernetes adheres to Cloud Native Compute Foundation's [Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md) throughout the project, and includes all communication mediums. + +## ADMINS +(by Slack ID and timezone) +* caniszczyk - CT +* ihor - CET +* jdumars - ET +* jorge - CT +* paris - PT + +Slack Admins should make sure to mention this in the “What I do” section of their Slack profile, as well as for which time zone. + +To connect: please reach out in the #slack-admins channel, mention one of us in the specific channel where you have a question, or DM (Direct Message) one of us privately. + +### ADMIN EXPECTATIONS AND GUIDELINES +* Adhere to Code of Conduct +* Take care of spam as soon as possible, which may mean taking action by making members inactive +* Moderating and fostering a safe environment for conversations +* Bring Code of Conduct issues to the Steering Committee +* Create relevant channels and list Code of Conduct in new channel welcome message +* Help troubleshoot Slack issues +* Review bot, token, and webhook requests +* Be helpful! + +## CREATING CHANNELS +Please reach out to the #slack-admins group with your request to create a new channel. + +Channels are dedicated to [SIGs, WGs](/sig-list.md), sub-projects, community topics, and related Kubernetes programs/projects. +Channels are not: +* company specific; cloud providers are ok with product names as the channel. Discourse will be about Kubernetes-related topics and not proprietary information of the provider. +* private unless there is an exception: code of conduct matters, mentoring, security/vulnerabilities, or steering committee. + +Typical naming conventions: +#kubernetes-foo #sig-foo #meetup-foo #location-users #projectname + +All channels need a documented purpose. Use this space to welcome the targeted community: promote your meetings, post agendas, etc. + +We may make special accommodations where necessary. + +## ESCALATING and/or REPORTING A PROBLEM +Join the #slack-admins channel or contact one of the admins in the closest timezone via DM directly and describe the situation. If the issue can be documented, please take a screenshot to include in your message. + +What if you have a problem with an admin? +Send a DM to another listed Admin and describe the situation OR +If it’s a code of conduct issue, please send an email to steering-private@kubernetes.io and describe the situation + +## BOTS, TOKENS, WEBHOOKS, OH MY + +Bots, tokens, and webhooks are reviewed on a case-by-case basis with most requests being rejected due to security, privacy, and usability concerns.. Bots and the like tend to make a lot of noise in channels. Our Slack instance has over 30,000 people and we want everyone to have a great experience. Please join #Slack-admins and have a discussion about your request before requesting the access. GitHub workflow alerts into certain channels and requests from CNCF are typically OK. + +## ADMIN MODERATION + +Be mindful of how you handle communication during stressful interactions. Administrators act as direct representatives of the community, and need to maintain a very high level of professionalism at all times. If you feel too involved in the situation to maintain impartiality or professionalism, that’s a great time to enlist the help of another admin. + +Try to take any situations that involve upset or angry members to DM or video chat. Please document these interactions for other Slack admins to review. + +Content will be automatically removed if it violates code of conduct or is a sales pitch. Admins will take a screenshot of such behavior in order to document the situation. The community takes such violations extremely seriously, and they will be handled swiftly. + +## INACTIVATING ACCOUNTS + +For reasons listed below, admins may inactivate individual Slack accounts. Due to Slack’s framework, it does not allow for an account to be banned or suspended in the traditional sense. [Visit Slack’s policy on this.](https://get.Slack.help/hc/en-us/articles/204475027-Deactivate-a-member-s-account) + +* Spreading spam content in DMs and/or channels +* Not adhering to the code of conduct set forth in DMs and/or channels +* Overtly selling products, related or unrelated to Kubernetes + +## SPECIFIC CHANNEL RULES + +In the case that certain channels have rules or guidelines, they will be listed in the purpose or pinned docs of that channel. + +#kubernetes-dev = questions and discourse around upstream contributions and development to kubernetes +#kubernetes-careers = job openings for positions working with/on/around Kubernetes. Postings should include contact details. + +## DM (Direct Message) Conversations + +Please do not engage in proprietary company specific conversations in the Kubernetes Slack instance. This is meant for conversations around related Kubernetes open source topics and community. Proprietary conversations should occur in your company Slack and/or communication platforms. As with all communication, please be mindful of appropriateness, professionalism, and applicability to the Kubernetes community. diff --git a/community-membership.md b/community-membership.md index 0a50f23f..772dedcb 100644 --- a/community-membership.md +++ b/community-membership.md @@ -26,7 +26,7 @@ but will not allow tests to be run against their PRs automatically nor allow the ### Requirements for outside collaborators - Working on some contribution to the project that would benefit from - the abillity to have PRs or Issues to be assigned to the contributor + the ability to have PRs or Issues to be assigned to the contributor - Have the support of 1 member - Find a member who will sponsor you - Send an email to kubernetes-membership@googlegroups.com @@ -64,7 +64,7 @@ Members are expected to remain active contributors to the community. - Sponsored by 2 reviewers. **Note the following requirements for sponsors**: - Sponsors must have close interactions with the prospective member - e.g. code/design/proposal review, coordinating on issues, etc. - Sponsors must be reviewers or approvers in at least 1 OWNERS file (in any repo in the Kubernetes GitHub organization) - - Not a requirement, but having sponsorship from a reviewer from another company is encouraged (you get a gold star). + - Sponsors must be from multiple member companies to demonstrate integration across community. - Send an email to *kubernetes-membership@googlegroups.com* with: - CC: your sponsors on the message - Subject: `REQUEST: New membership for <your-GH-handle>` @@ -229,7 +229,7 @@ TODO: Determine if this role is outdated and needs to be redefined or merged int - Primary reviewer for 20 substantial PRs - Reviewed or merged at least 50 PRs - Apply to [`kubernetes-maintainers`](https://github.com/orgs/kubernetes/teams/kubernetes-maintainers), with: - - A [Champion](https://github.com/kubernetes/community/blob/master/incubator.md#faq) from the existing + - A [Champion](/incubator.md#faq) from the existing kubernetes-maintainers members - A Sponsor from Project Approvers - Summary of contributions to the project diff --git a/community/2017-events/12-dev-summit/information.md b/community/2017-events/12-dev-summit/information.md deleted file mode 100644 index 8c4be385..00000000 --- a/community/2017-events/12-dev-summit/information.md +++ /dev/null @@ -1 +0,0 @@ -More info forthcoming about the 2017 Kubernetes Developer's Summit! diff --git a/community/OWNERS b/community/OWNERS deleted file mode 100644 index ff16061d..00000000 --- a/community/OWNERS +++ /dev/null @@ -1,12 +0,0 @@ -reviewers: - - czahedi - - nyener - - zehicle - - sarahnovotny - - jberkus -approvers: - - czahedi - - nyener - - zehicle - - sarahnovotny - - jberkus diff --git a/community/elections/2017/README.md b/community/elections/2017/README.md deleted file mode 100644 index dfd0ff3b..00000000 --- a/community/elections/2017/README.md +++ /dev/null @@ -1,14 +0,0 @@ -# Kubernetes Steering Committee Election 2017 - -The following is the platform statement from the candidates for the Steering Committee election. - -For more information see the -[Steering Committee Charter](https://docs.google.com/document/d/1LelgXRs6mjYzaZOzh4X-4uqhHGsFHYD3dbDRYVYml-0/edit#) - -## Candidates: - -<!-- Alphabetical order, add the link to your statement like this: - -- [Jane Podlet](jane-podlet.md) - ---> diff --git a/contributors/design-proposals/sig-cli/template.md b/contributors/design-proposals/Design_Proposal_TEMPLATE.md index 9f3a683b..9f3a683b 100644 --- a/contributors/design-proposals/sig-cli/template.md +++ b/contributors/design-proposals/Design_Proposal_TEMPLATE.md diff --git a/contributors/design-proposals/OWNERS b/contributors/design-proposals/OWNERS new file mode 100644 index 00000000..ace398b5 --- /dev/null +++ b/contributors/design-proposals/OWNERS @@ -0,0 +1,18 @@ +reviewers: + - brendandburns + - dchen1107 + - jbeda + - lavalamp + - smarterclayton + - thockin + - wojtek-t + - bgrant0607 +approvers: + - brendandburns + - dchen1107 + - jbeda + - lavalamp + - smarterclayton + - thockin + - wojtek-t + - bgrant0607 diff --git a/contributors/design-proposals/README.md b/contributors/design-proposals/README.md index f165d05d..dbdfb6c9 100644 --- a/contributors/design-proposals/README.md +++ b/contributors/design-proposals/README.md @@ -2,14 +2,10 @@ This directory contains Kubernetes design documents and accepted design proposals. -For a design overview, please see [the architecture document](architecture.md). +For a design overview, please see [the architecture document](architecture/architecture.md). Note that a number of these documents are historical and may be out of date or unimplemented. TODO: Add the current status to each document and clearly indicate which are up to date. TODO: Document the [proposal process](../devel/faster_reviews.md#1-dont-build-a-cathedral-in-one-pr). - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/api-machinery/OWNERS b/contributors/design-proposals/api-machinery/OWNERS new file mode 100644 index 00000000..0df76e64 --- /dev/null +++ b/contributors/design-proposals/api-machinery/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-api-machinery-leads +approvers: + - sig-api-machinery-leads +labels: + - sig/api-machinery diff --git a/contributors/design-proposals/add-new-patchStrategy-to-clear-fields-not-present-in-patch.md b/contributors/design-proposals/api-machinery/add-new-patchStrategy-to-clear-fields-not-present-in-patch.md index 2a9f660b..5f035f9b 100644 --- a/contributors/design-proposals/add-new-patchStrategy-to-clear-fields-not-present-in-patch.md +++ b/contributors/design-proposals/api-machinery/add-new-patchStrategy-to-clear-fields-not-present-in-patch.md @@ -44,7 +44,7 @@ that does not contain a discriminator. |---|---| | non-inlined non-discriminated union | Yes | | non-inlined discriminated union | Yes | -| inlined union with [patchMergeKey](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#strategic-merge-patch) only | Yes | +| inlined union with [patchMergeKey](/contributors/devel/api-conventions.md#strategic-merge-patch) only | Yes | | other inlined union | No | For the inlined union with patchMergeKey, we move the tag to the parent struct's instead of @@ -299,7 +299,7 @@ Each field present in the request will be merged with the live config. There are 2 reasons of avoiding this logic: - Using `$patch` as directive key will break backward compatibility. -But can easily fixed by using a different key, e.g. `retainKeys: true`. +But can easily be fixed by using a different key, e.g. `retainKeys: true`. Reason is that `$patch` has been used in earlier releases. If we add new value to this directive, the old server will reject the new patch due to not knowing the new value. diff --git a/contributors/design-proposals/api-machinery/admission-control-webhooks.md b/contributors/design-proposals/api-machinery/admission-control-webhooks.md new file mode 100644 index 00000000..100c27fa --- /dev/null +++ b/contributors/design-proposals/api-machinery/admission-control-webhooks.md @@ -0,0 +1,960 @@ +# Webhooks Beta + + +## PUBLIC +Authors: @erictune, @caesarxuchao, @enisoc +Thanks to: {@dbsmith, @smarterclayton, @deads2k, @cheftako, @jpbetz, @mbohlool, @mml, @janetkuo} for comments, data, prior designs, etc. + + +[TOC] + + +# Summary + +This document proposes a detailed plan for bringing Webhooks to Beta. Highlights include (incomplete, see rest of doc for complete list) : + + + +* Adding the ability for webhooks to mutate. +* Bootstrapping +* Monitoring +* Versioned rather than Internal data sent on hook +* Ordering behavior within webhooks, and with other admission phases, is better defined + +This plan is compatible with the [original design doc](/contributors/design-proposals/api-machinery/admission_control_extension.md). + + +# Definitions + +**Mutating Webhook**: Webhook that can change a request as well as accept/reject. + +**Non-Mutating Webhook**: Webhook that cannot change request, but can accept or reject. + +**Webhook**: encompasses both Mutating Webhook and/or Non-mutating Webhook. + +**Validating Webhook**: synonym for Non-Mutating Webhook + +**Static Admission Controller**: Compiled-in Admission Controllers, (in plugin/pkg/admission). + +**Webhook Host**: a process / binary hosting a webhook. + +# Naming + +Many names were considered before settling on mutating. None of the names +considered were completely satisfactory. The following are the names which were +considered and a brief explanation of the perspectives on each. + +* Mutating: Well defined meaning related to mutable and immutable. Some + negative connotations related to genetic mutation. Might be too specifically + as CS term. +* Defaulting: Clearly indicates a create use case. However implies a lack of + functionality for the update use case. +* Modifying: Similar issues to mutating but not as well defined. +* Revising: Less clear what it does. Does it imply it works only on updates? +* Transforming: Some concern that it might have more to do with changing the + type or shape of the related object. +* Adjusting: Same general perspective as modifying. +* Updating: Nice clear meaning. However it seems to easy to confuse update with + the update operation and intuit it does not apply to the create operation. + + +# Development Plan + +Google able to staff development, test, review, and documentation. Community help welcome, too, esp. Reviewing. + +Intent is Beta of Webhooks (**both** kinds) in 1.9. + +Not in scope: + + + +* Initializers remains Alpha for 1.9. (See [Comparison of Webhooks and Initializers](#comparison-of-webhooks-and-initializers) section). No changes to it. Will revisit its status post-1.9. +* Converting static admission controllers is out of scope (but some investigation has been done, see Moving Built-in Admission Controllers section). + + +## Work Items + +* Add API for registering mutating webhooks. See [API Changes](#api-changes) +* Copy the non-mutating webhook admission controller code and rename it to be for mutating. (Splitting into two registration APIs make ordering clear.) Add changes to handle mutating responses. See [Responses for Mutations](#responses-for-mutations). +* Document recommended flag order for admission plugins. See [Order of Admission](#order-of-admission). +* In kube-up.sh and other installers, change flag per previous item. +* Ensure able to monitor latency and rejection from webhooks. See [Monitorability](#monitorability). +* Don't send internal objects. See [#49733](https://github.com/kubernetes/kubernetes/issues/49733) +* Serialize mutating Webhooks into order in the apiregistration. Leave non-mutating in parallel. +* Good Error Messages. See [Good Error Messages](#good-error-messages) +* Conversion logic in GenericWebhook to send converted resource to webhook. See [Conversion](#conversion) and [#49733](https://github.com/kubernetes/kubernetes/issues/49733). +* Schedule discussion around resiliency to down webhooks and bootstrapping +* Internal Go interface refactor (e.g. along the lines suggested #[1137](https://github.com/kubernetes/community/pull/1137)). + + +# Design Discussion + + +## Why Webhooks First + +We will do webhooks beta before initializers beta because: + + + +1. **Serves Most Use Cases**: We reviewed code of all current use cases, namely: Kubernetes Built-in Admission Controllers, OpenShift Admission Controllers, Istio & Service Catalog. (See also [Use Cases Detailed Descriptions](#use-cases-detailed-descriptions).) All of those use cases are well served by mutating and non-mutating webhooks. (See also [Comparison of Webhooks and Initializers](#comparison-of-webhooks-and-initializers)). +1. **Less Work**: An engineer quite experienced with both code bases estimated that it is less work to adding Mutating Webhooks and bring both kinds of webhooks to beta; than to bring non-mutating webhooks and initializers to Beta. Some open issues with Initializers with long expected development time include quota replenishment bug, and controller awareness of uninitialized objects. +1. **API Consistency**: Prefer completing one related pair of interfaces (both kinds of webhooks) at the same time. + + +## Why Support Mutation for Beta + +Based on experience and feedback from the alpha phase of both Webhooks and Initializers, we believe Webhooks Beta should support mutation because: + + + +1. We have lots of use cases to inform this (both from Initializers, and Admission Controllers) to ensure we have needed features +1. We have experience with Webhooks API already to give confidence in the API. The registration API will be quite similar except in the responses. +1. There is a strong community demand for something that satisfies a mutating case. + + +## Plan for Existing Initializer-Clients + +After the release of 1.9, we will advise users who currently use initializers to: + + + +* Move to Webhooks if their use case fits that model well. +* Provide SIG-API-Machinery with feedback if Initializers is a better fit. + +We will continue to support Initializers as an Alpha API in 1.9. + +We will make a user guide and extensively document these webhooks. We will update some existing examples, maybe https://github.com/caesarxuchao/example-webhook-admission-controller (since the initializer docs point to it, e.g. https://github.com/kelseyhightower/kubernetes-initializer-tutorial), or maybe https://github.com/openshift/generic-admission-server. + +We will clearly document the reasons for each and how users should decide which to use. + + +## Monitorability + +There should be prometheus variables to show: + + + +* API operation latency + * Overall + * By webhook name +* API response codes + * Overall + * By webhook name. + +Adding a webhook dynamically adds a key to a map-valued prometheus metric. Webhook host process authors should consider how to make their webhook host monitorable: while eventually we hope to offer a set of best practices around this, for the initial release we won't have requirements here. + + +## API Changes + +GenericAdmissionWebhook Admission Controller is split and renamed. + + + +* One is called `MutatingAdmissionWebhook` +* The other is called `ValidatingAdmissionWebhook` +* Splitting them allows them to appear in different places in the `--admission-control` flag's order. + +ExternalAdmissionHookConfiguration API is split and renamed. + + + +* One is called `MutatingAdmissionWebhookConfiguration` +* The other is called `ValidatingAdmissionWebhookConfiguration` +* Splitting them: + * makes it clear what the order is when some items don't have both flavors, + * enforces mutate-before-validate, + * better allows declarative update of the config than one big list with an implied partition point + +The `ValidatingAdmissionWebhookConfiguration` stays the same as `ExternalAdmissionHookConfiguration` except it moves to v1beta1. + +The `MutatingAdmissionWebhookConfiguration` is the same API as `ValidatingAdmissionWebhookConfiguration`. It is only visible via the v1beta1 version. + +We will change from having a Kubernetes service object to just accepting a DNS +name for the location of the webhook. + +The Group/Version called + +`admissionregistration.k8s.io/v1alpha1` with kinds + +InitializerConfiguration and ExternalAdmissionHookConfiguration. + +InitializerConfiguration will not join `admissionregistration.k8s.io/v1beta1` at this time. + +Any webhooks that register with v1alpha1 may or may not be surprised when they start getting versioned data. But we don't make any promises for Alpha, and this is a very important bug to fix. + + +## Order of Admission + +At kubernetes.io, we will document the ordering requirements or just recommend a particular order for `--admission-control`. A starting point might be `MutatingAdmissionWebhook,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,ValidatingAdmissionWebhook,ResourceQuota`. + + There might be other ordering dependencies that we will document clearly, but some important properties of a valid ordering: + +* ResourceQuota comes last, so that if prior ones reject a request, it won't increment quota. +* All other Static ones are in the order recommended by [the docs](https://kubernetes.io/docs/admin/admission-controllers/#is-there-a-recommended-set-of-plug-ins-to-use). (which variously do mutation and validation) Preserves the behavior when there are no webhooks. +* Ensures dynamic mutations happen before all validations. +* Ensures dynamic validations happen after all mutations. +* Users don't need to reason about the static ones, just the ones they add. + +System administrators will likely need to know something about the webhooks they +intend to run in order to make the best ordering, but we will try to document a +good "first guess". + +Validation continues to happen after all the admission controllers (e.g. after mutating webhooks, static admission controllers, and non-mutating admission controllers.) + +**TODO**: we should move ResourceQuota after Validation, e.g. as described in #1137. However, this is a longstanding bug and likely a larger change than can be done in 1.9--a larger quota redesign is out of scope. But we will likely make an improvement in the current ordering. + + +## Parallel vs Serial + +The main reason for parallel is reducing latency due to round trip and conversion. We think this can often mitigated by consolidating multiple webhooks shared by the same project into one. + +Reasons not to allow parallel are complexity of reasoning about concurrent patches, and CRD not supporting PATCH. + +`ValidatingAdmissionWebhook `is already parallel, and there are no responses to merge. Therefore, it stays parallel. + +`MutatingAdmissionWebhook `will run in serial, to ensure conflicts are resolved deterministically. + +The order is the sort order of all the WebhookConfigs, by name, and by index within the Webhooks list. + +We don't plan to make mutating webhooks parallel at this time, but we will revisit the question in the future and decide before going to GA. + +## Good Error Messages + +When a webhook is persistently failing to allow e.g. pods to be created, then the error message from the apiserver must show which webhook failed. + +When a core controller, e.g. ReplicaSet, fails to make a resources, it must send a helpful event that is visible in `kubectl describe` for the controlling resources, saying the reason create failed. + +## Registering for all possible representations of the same object + +Some Kubernetes resources are mounted in the api type system at multiple places +(e.g., during a move between groups). Additionally, some resources have multiple +active versions. There's not currently a way to easily tell which of the exposed +resources map to the same "storage location". We will not try to solve that +problem at the moment: if the system administrator wishes to hook all +deployments, they must (e.g.) make sure their hook is registered for both +deployments.v1beta1.extensions AND deployments.v1.apps. + +This is likely to be error-prone, especially over upgrades. For GA, we may +consider mechanisms to make this easier. We expect to gather user feedback +before designing this. + + +## Conversion and Versioning + +Webhooks will receive the admission review subject in the exact version which +the user sent it to the control plane. This may require the webhook to +understand multiple versions of those types. + +All communication to webhooks will be JSON formatted, with a request body of +type admission.k8s.io/v1beta1. For GA, we will likely also allow proto, via a +TBD mechanism. + +We will not take any particular steps to make it possible to know whether an +apiserver is safe to upgrade, given the webhooks it is running. System +administrators must understand the stack of webhooks they are running, watch the +Kubernetes release notes, and look to the webhook authors for guidance about +whether the webhook supports Kubernetes version N. We may choose to address this +deficency in future betas. + +To follow the debate that got us to this position, you can look at this +potential design for the next steps: https://docs.google.com/document/d/1BT8mZaT42jVxtC6l14YMXpUq0vZc6V5MPf_jnzDMMcg/edit + + +## Mutations + +The Response for `MutatingAdmissionWebhook` must have content-type, and it must be one of: + +* `application/json` +* `application/protobuf` +* `application/strategic-merge-patch+json` +* `application/json-patch+json` +* `application/merge-json-patch+json` + +If the response is a patch, it is merged with the versioned response from the previous webhook, where possible without Conversion. + +We encourage the use of patch to avoid the "old clients dropping new fields" problem. + + +## Bootstrapping + +Bootstrapping (both turning on a cluster for the first time and making sure a +cluster can boot from a cold start) is made more difficult by having webhooks, +which are a dependency of the control plane. This is covered in its [own design +doc](./admission-webhook-bootstrapping.md). + +## Upgrading the control plane + +There are two categories of webhooks: security critical (e.g., scan images for +vulnerabilities) and nice-to-have (set labels). + +Security critical webhooks cannot work with Kubernetes types they don't have +built-in knowledge of, because they can't know if e.g. Kubernetes 1.11 adds a +backwards-compatible `v1.Pod.EvilField` which will defeat their functionality. + +They therefore need to be updated before any apiserver. It is the responsibility +of the author of such a webhook to release new versions in response to new +Kubernetes versions in a timely manner. Webhooks must support two consecutive +Kubernetes versions so that rollback/forward is possible. When/if Kubernetes +introduces LTS versions, webhook authors will have to also support two +consecutive LTS versions. + +Non-security-critical webhooks can either be turned off to perform an upgrade, +or can just continue running the old webhook version as long as a completely new +version of an object they want to hook is not added. If they are metadata-only +hooks, then they should be able to run until we deprecate meta/v1. Such webhooks +should document that they don't consider themselves security critical, aren't +obligated to follow the above requirements for security-critical webhooks, and +therefore do not guarantee to be updated for every Kubernetes release. + +It is expected that webhook authors will distribute config for each Kubernetes +version that registers their webhook for all the necessary types, since it would +be unreasonable to make system administrators understand all of the webhooks +they run to that level of detail. + +## Support for Custom Resources + +Webhooks should work with Custom Resources created by CRDs. + +They are particularly needed for Custom Resources, where they can supplement the validation and defaulting provided by OpenAPI. Therefore, the webhooks will be moved or copied to genericapiserver for 1.9. + + +## Support for Aggregated API Servers + +Webhooks should work with Custom Resources on Aggregated API Servers. + +Aggregated API Servers should watch apiregistraton on the main APIserver, and should identify webhooks with rules that match any of their resources, and call those webhooks. + +For example a user might install a Webhook that adds a certain annotation to every single object. Aggregated APIs need to support this use case. + +We will build the dynamic admission stack into the generic apiserver layer to support this use case. + + +## Moving Built-in Admission Controllers + +This section summarizes recommendations for Posting static admission controllers to Webhooks. + +See also [Details of Porting Admission Controllers](#details-of-porting-admission-controllers) and this [Backup Document](https://docs.google.com/spreadsheets/d/1zyCABnIzE7GiGensn-KXneWrkSJ6zfeJWeLaUY-ZmM4/edit#gid=0). + +Here is an estimate of how each kind of admission controller would be moved (or not). This is to see if we can cover the use cases we currently have, not necessarily a promise that all of these will or should be move into another process. + + + +* Leave static: + * OwnerReferencesPermissionEnforcement + * GC is a core feature of Kubernetes. Move to required. + * ResourceQuota + * May [redesign](https://github.com/kubernetes/kubernetes/issues/51820) + * Original design doc says it remains static. +* Divide into Mutating and non-mutating Webhooks + * PodSecurityPolicy + * NamespaceLifecycle +* Use Mutating Webhook + * AlwaysPullImages + * ServiceAccount + * StorageClass +* Use non-mutating Webhook + * Eventratelimit + * DenyEscalatingExec + * ImagePolicy + * Need to standardize the webhook format + * NodeRestriction + * Needs to be admission to access User.Info + * PodNodeSelector + * PodTolerationRestriction +* Move to resource's validation or defaulting + * AntiAffinity + * DefaultTolerationSeconds + * PersistentVolumeClaimResize + * Initializers are reasonable to consider moving into the API machinery + +For "Divide", the backend may well be different port of same binary, sharing a SharedInformer, so data is not cached twice. + +For all Kubernetes built-in webhooks, the backend will likely be compiled into kube-controller-manager and share the SharedInformer. + + +# Use Case Analysis + + +## Use Cases Detailed Descriptions + +Mutating Webhooks, Non-mutating webhooks, Initializers, and Finalizers (collectively, Object Lifecycle Extensions) serve to: + + + +* allow policy and behavioral changes to be developed independently of the control loops for individual Resources. These might include company specific rules, or a PaaS that layers on top of Kubernetes. +* implement business logic for Custom Resource Definitions +* separate Kubernetes business logic from the core Apiserver logic, which increases reusability, security, and reliability of the core. + +Specific Use cases: + + + +* Kubernetes static Admission Controllers + * Documented [here](https://kubernetes.io/docs/admin/admission-controllers/) + * Discussed [here](/contributors/design-proposals/api-machinery/admission_control_extension.md) + * All are highly reliable. Most are simple. No external deps. + * Many need update checks. + * Can be separated into mutation and validate phases. +* OpenShift static Admission Controllers + * Discussed [here](/contributors/design-proposals/api-machinery/admission_control_extension.md) + * Similar to Kubernetes ones. +* Istio, Case 1: Add Container to all Pods. + * Currently uses Initializer but can use Mutating Webhook. + * Simple, can be highly reliable and fast. No external deps. + * No current use case for updates. +* Istio, Case 2: Validate Mixer CRDs + * Checking cached values from other CRD objects. + * No external deps. + * Must check updates. +* Service Catalog + * Watch PodPreset and edit Pods. + * Simple, can be highly reliable and fast. No external deps. + * No current use case for updates. + +Good further discussion of use cases [here](/contributors/design-proposals/api-machinery/admission_control_extension.md) + + +## Details of Porting Admission Controllers + +This section summarizes which Kubernetes static admission controllers can readily be ported to Object Lifecycle Extensions. + + +### Static Admission Controllers + + +<table> + <tr> + <td>Admission Controller + </td> + <td>How + </td> + <td>Why + </td> + </tr> + <tr> + <td>PodSecurityPolicy + </td> + <td>Use Mutating Webhook and Non-Mutating Webhook. + </td> + <td>Requires User.Info, so needs webhook. +<p> +Mutating will set SC from matching PSP. +<p> +Non-Mutating will check again in case any other mutators or initializers try to change it. + </td> + </tr> + <tr> + <td>ResourceQuota + </td> + <td>Leave static + </td> + <td>A Redesign for Resource Quota has been proposed, to allow at least object count quota for other objects as well. This suggests that Quota might need to remain compiled in like authn and authz are. + </td> + </tr> + <tr> + <td>AlwaysPullImages + </td> + <td>Use Mutating Webhook (could implement using initializer since the thing is it validating is forbidden to change by Update Validation of the object) + </td> + <td>Needs to + </td> + </tr> + <tr> + <td>AntiAffinity + </td> + <td>Move to pod validation + </td> + <td>Since this is provided by the core project, which also manages the pod business logic, it isn't clear why this is even an admission controller. Ask Scheduler people. + </td> + </tr> + <tr> + <td>DefaultTolerationSeconds + </td> + <td>Move to pod defaulting or use a Mutating Webhook. + </td> + <td>It is very simple. + </td> + </tr> + <tr> + <td>eventratelimit + </td> + <td>Non-mutating webhook + </td> + <td>Simple logic, does not mutate. Alternatively, have rate limit be a built-in of api server. + </td> + </tr> + <tr> + <td>DenyEscalatingExec + </td> + <td>Non-mutating Webhook. + </td> + <td>It is very simple. It is optional. + </td> + </tr> + <tr> + <td>OwnerReferences- PermissionEnforcement (gc) + </td> + <td>Leave compiled in + </td> + <td>Garbage collection is core to Kubernetes. Main and all aggregated apiservers should enforce it. + </td> + </tr> + <tr> + <td>ImagePolicy + </td> + <td>Non-mutating webhook + </td> + <td>Must use webhook since image can be updated on pod, and that needs to be checked. + </td> + </tr> + <tr> + <td>LimitRanger + </td> + <td>Mutating Webhook + </td> + <td>Fast + </td> + </tr> + <tr> + <td>NamespaceExists + </td> + <td>Leave compiled in + </td> + <td>This has been on by default for years, right? + </td> + </tr> + <tr> + <td>NamespaceLifecycle + </td> + <td>Split: +<p> + +<p> +Cleanup, leave compiled in. +<p> + +<p> +Protection of system namespaces: use non-mutating webhook + </td> + <td> + </td> + </tr> + <tr> + <td>NodeRestriction + </td> + <td>Use a non-mutating webhook + </td> + <td>Needs webhook so it can use User.Info. + </td> + </tr> + <tr> + <td>PersistentVolumeClaimResize + </td> + <td>Move to validation + </td> + <td>This should be in the validation logic for storage class. + </td> + </tr> + <tr> + <td>PodNodeSelector + </td> + <td>Move to non-mutating webhook + </td> + <td>Already compiled in, so fast enough to use webhook. Does not mutate. + </td> + </tr> + <tr> + <td>podtolerationrestriction + </td> + <td>Move to non-mutating webhook + </td> + <td>Already compiled in, so fast enough to use webhook. Does not mutate. + </td> + </tr> + <tr> + <td>serviceaccount + </td> + <td>Move to mutating webhook. + </td> + <td>Already compiled in, so fast enough to use webhook. Does mutate by defaulting the service account. + </td> + </tr> + <tr> + <td>storageclass + </td> + <td>Move to mutating webhook. + </td> + <td> + </td> + </tr> +</table> + + +[Backup Document](https://docs.google.com/spreadsheets/d/1zyCABnIzE7GiGensn-KXneWrkSJ6zfeJWeLaUY-ZmM4/edit#gid=0) + + +### OpenShift Admission Controllers + + +<table> + <tr> + <td>Admission Controller + </td> + <td>How + </td> + <td>Why + </td> + </tr> + <tr> + <td>pkg/authorization/admission/restrictusers" + </td> + <td>Non-mutating Webhook or leave static + </td> + <td>Verification only. But uses a few loopback clients to check other resources. + </td> + </tr> + <tr> + <td>pkg/build/admission/jenkinsbootstrapper + </td> + <td>Non-mutating Webhook or leave static + </td> + <td>Doesn't mutate Build or BuildConfig, but creates Jenkins instances. + </td> + </tr> + <tr> + <td>pkg/build/admission/secretinjector + </td> + <td>Mutating webhook or leave static + </td> + <td>uses a few loopback clients to check other resources. + </td> + </tr> + <tr> + <td>pkg/build/admission/strategyrestrictions + </td> + <td>Non-mutating Webhook or leave static + </td> + <td>Verifications only. But uses a few loopback clients, and calls subjectAccessReview + </td> + </tr> + <tr> + <td>pkg/image/admission + </td> + <td>Non-Mutating Webhook + </td> + <td>Fast, checks image size + </td> + </tr> + <tr> + <td>pkg/image/admission/imagepolicy + </td> + <td>Mutating and non-mutating webhooks + </td> + <td>Rewriting image pull spec is mutating. +<p> +acceptor.Accepts is non-Mutating + </td> + </tr> + <tr> + <td>pkg/ingress/admission + </td> + <td>Non-mutating webhook, or leave static. + </td> + <td>Simple, but calls to authorizer. + </td> + </tr> + <tr> + <td>pkg/project/admission/lifecycle + </td> + <td>Initializer or Non-mutating webhook? + </td> + <td>Needs to update another resource: Namespace + </td> + </tr> + <tr> + <td>pkg/project/admission/nodeenv + </td> + <td>Mutating webhook + </td> + <td>Fast + </td> + </tr> + <tr> + <td>pkg/project/admission/requestlimit + </td> + <td>Non-mutating webhook + </td> + <td>Fast, verification only + </td> + </tr> + <tr> + <td>pkg/quota/admission/clusterresourceoverride + </td> + <td>Mutating webhook + </td> + <td>Updates container resource request and limit + </td> + </tr> + <tr> + <td>pkg/quota/admission/clusterresourcequota + </td> + <td>Leave static. + </td> + <td>Refactor with the k8s quota + </td> + </tr> + <tr> + <td>pkg/quota/admission/runonceduration + </td> + <td>Mutating webhook + </td> + <td>Fast. Needs a ProjectCache though. Updates pod.Spec.ActiveDeadlineSeconds + </td> + </tr> + <tr> + <td>pkg/scheduler/admission/podnodeconstraints + </td> + <td>Non-mutating webhook or leave static + </td> + <td>Verification only. But calls to authorizer. + </td> + </tr> + <tr> + <td>pkg/security/admission + </td> + <td>Use Mutating Webhook and Non-Mutating Webhook. + </td> + <td>Similar to PSP in k8s + </td> + </tr> + <tr> + <td>pkg/service/admission/externalip + </td> + <td>Non-mutating webhook + </td> + <td>Fast and verification only + </td> + </tr> + <tr> + <td>pkg/service/admission/endpoints + </td> + <td>Non-mutating webhook or leave static + </td> + <td>Verification only. But calls to authorizer. + </td> + </tr> +</table> + + + +### Other Projects + +Istio Pod Injector: + + + +* Injects Sidecar Container, Init Container, adds a volume for Istio config, and changes the Security Context +* Source: + * https://github.com/istio/pilot/blob/master/platform/kube/inject/inject.go#L278 + * https://github.com/istio/pilot/blob/master/cmd/sidecar-initializer/main.go + +<table> + <tr> + <td> +Function + </td> + <td>How + </td> + <td>Why + </td> + </tr> + <tr> + <td>Istio Pod Injector + </td> + <td>Mutating Webhook + </td> + <td>Containers can only be added at pod creation time. +<p> +Because the change is complex, showing intermediate state may help debugging. +<p> +Fast, so could also use webhook. + </td> + </tr> + <tr> + <td>Istio Mixer CRD Validation + </td> + <td>Non-Mutating Webhook + </td> + <td> + </td> + </tr> + <tr> + <td>Service Catalog PodPreset + </td> + <td>Initializer + </td> + <td>Containers can only be added at pod creation time. +<p> +Because the change is complex, showing intermediate state may help debugging. +<p> +Fast, so could also use webhook. + </td> + </tr> + <tr> + <td>Allocate Cert for Service + </td> + <td>Initializer + </td> + <td>Longer duration operation which might fail, with external dependency, so don't use webhook. +<p> +Let user see initializing state. +<p> +Don't let controllers that depend on services see the service before it is ready. + </td> + </tr> +</table> + + + +## Comparison of Webhooks and Initializers + + +<table> + <tr> + <td>Mutating and Non-Mutating Webhooks + </td> + <td>Initializers (and Finalizers) + </td> + </tr> + <tr> + <td><ul> + +<li>Act on Create, update, or delete +<li>Reject Create, Update or delete</li></ul> + + </td> + <td><ul> + +<li>Act on Create and delete +<li>Reject Create.</li></ul> + + </td> + </tr> + <tr> + <td><ul> + +<li>Clients never see pre-created state. <ul> + + <li>Good for enforcement. + <li>Simple invariants.</li> </ul> +</li> </ul> + + </td> + <td><ul> + +<li>Clients can see pre-initialized state. <ul> + + <li>Let clients see progress + <li>Debuggable</li> </ul> +</li> </ul> + + </td> + </tr> + <tr> + <td><ul> + +<li>Admin cannot easily override broken webhook. <ul> + + <li>Must be highly reliable code + <li>Avoid deps on external systems.</li> </ul> +</li> </ul> + + </td> + <td><ul> + +<li>Admin can easily fix a "stuck" object by "manually" initializing (or finalizing). <ul> + + <li>Can be <em>slightly</em> less reliable. + <li>Prefer when there are deps on external systems.</li> </ul> +</li> </ul> + + </td> + </tr> + <tr> + <td><ul> + +<li>Synchronous <ul> + + <li>Apiserver uses a go routine + <li>TCP connection open + <li>Should be very low latency</li> </ul> +</li> </ul> + + </td> + <td><ul> + +<li>Asynchronous <ul> + + <li>Can be somewhat higher latency</li> </ul> +</li> </ul> + + </td> + </tr> + <tr> + <td><ul> + +<li>Does not persist intermediate state <ul> + + <li>Should happen very quickly. + <li>Does not increase etcd traffic.</li> </ul> +</li> </ul> + + </td> + <td><ul> + +<li>Persist intermediate state <ul> + + <li>Longer ops can persist across apiserver upgrades/failures + <li>Does increase etcd traffic.</li> </ul> +</li> </ul> + + </td> + </tr> + <tr> + <td><ul> + +<li>Webhook does not know if later webhooks fail <ul> + + <li>Must not have side effects, + <li>Or have a really good GC plan.</li> </ul> +</li> </ul> + + </td> + <td><ul> + +<li>Initializer does not know if later initializers fail, but if paired with a finalizer, it could see the resource again. <ul> + + <li>This is not implemented + <li>TODO: initializers: have a way to ensure finalizer runs even if later initializers reject?</li> </ul> +</li> </ul> + + </td> + </tr> + <tr> + <td> + Use Examples:<ul> + +<li>checking one field on an object, and setting another field on the same object</li></ul> + + </td> + <td> + Use Examples:<ul> + +<li>Allocate (and deallocate) external resource in parallel with a Kubernetes resource.</li></ul> + + </td> + </tr> +</table> + + +Another [Detailed Comparison of Initializers and Webhooks](https://docs.google.com/document/d/17P_XjXDpxDC5xSD0nMT1W18qE2AlCMkVJcV6jXKlNIs/edit?ts=59d5683b#heading=h.5irk4csrpu0y) diff --git a/contributors/design-proposals/api-machinery/admission-webhook-bootstrapping.md b/contributors/design-proposals/api-machinery/admission-webhook-bootstrapping.md new file mode 100644 index 00000000..190a538a --- /dev/null +++ b/contributors/design-proposals/api-machinery/admission-webhook-bootstrapping.md @@ -0,0 +1,98 @@ +# Webhook Bootstrapping + +## Background +[Admission webhook](./admission-control-webhooks.md) is a feature that +dynamically extends Kubernetes admission chain. Because the admission webhooks +are in the critical path of admitting REST requests, broken webhooks could block +the entire cluster, even blocking the reboot of the webhooks themselves. This +design presents a way to avoid such bootstrap deadlocks. + +## Objective +- If one or more webhooks are down, it should be able restart them automatically. +- If a core system component that supports webhooks is down, the component + should be able to restart. + +## Design idea +We add a selector to the admission webhook configuration, which will be compared +to the labels of namespaces. Only objects in the matching namespaces are +subjected to the webhook admission. A cluster admin will want to exempt these +namespaces from webhooks: +- Namespaces where this webhook and other webhooks are deployed in; +- Namespaces where core system components are deployed in. + +## API Changes +`ExternalAdmissionHook` is the dynamic configuration API of an admission webhook. +We will add a new field `NamespaceSelector` to it: + +```golang +type ExternalAdmissionHook struct { + Name string + ClientConfig AdmissionHookClientConfig + Rules []RuleWithOperations + FailurePolicy *FailurePolicyType + // Only objects in matching namespaces are subjected to this webhook. + // LabelSelector.MatchExpressions allows exclusive as well as inclusive + // matching, so you can use this // selector as a whitelist or a blacklist. + // For example, to apply the webhook to all namespaces except for those have + // labels with key "runlevel" and value equal to "0" or "1": + // metav1.LabelSelctor{MatchExpressions: []LabelSelectorRequirement{ + // { + // Key: "runlevel", + // Operator: metav1.LabelSelectorOpNotIn, + // Value: []string{"0", "1"}, + // }, + // }} + // As another example, to only apply the webhook to the namespaces that have + // labels with key “environment” and value equal to “prod” and “staging”: + // metav1.LabelSelctor{MatchExpressions: []LabelSelectorRequirement{ + // { + // Key: "environment", + // Operator: metav1.LabelSelectorOpIn, + // Value: []string{"prod", "staging"}, + // }, + // }} + // See https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ for more examples of label selectors. + NamespaceSelector *metav1.LabelSelector +} +``` + +## Guidelines on namespace labeling +The mechanism depends on cluster admin properly labelling the namespaces. We +will provide guidelines on the labelling scheme. One suggestion is labelling +namespaces with runlevels. The design of runlevels is out of the scope of this +document (tracked in +[#54522](https://github.com/kubernetes/kubernetes/issues/54522)), a strawman +runlevel scheme is: + +- runlevel 0: namespaces that host core system components, like kube-apiserver + and kube-controller-manager. +- runlevel 1: namespaces that host add-ons that are part of the webhook serving + stack, e.g., kube-dns. +- runlevel 2: namespaces that host webhooks deployments and services. + +`ExternalAdmissionHook.NamespaceSelector` should be configured to skip all the +above namespaces. In the case where some webhooks depend on features offered by +other webhooks, the system administrator could extend this concept further (run +level 3, 4, 5, …) to accommodate them. + +## Security implication +The mechanism depends on namespaces being properly labelled. We assume only +highly privileged users can modify namespace labels. Note that the system +already relies on correct namespace annotations, examples include the +podNodeSelector admission plugin, and the podTolerationRestriction admission +plugin etc. + +# Considered Alternatives +- Allow each webhook to exempt one namespace + - Doesn’t work: if there are two webhooks in two namespaces both blocking pods + startup, they will block each other. +- Put all webhooks in a single namespace and let webhooks exempt that namespace, + e.g., deploy webhooks in the “kube-system” namespace and exempt the namespace. + - It doesn’t provide sufficient isolation. Not all objects in the + “kube-system” namespace should bypass webhooks. +- Add namespace selector to webhook configuration, but use the selector to match + the name of namespaces + ([#1191](https://github.com/kubernetes/community/pull/1191)). + - Violates k8s convention. The matching label (key=name, value=<namespace’s + name>) is imaginary. + - Hard to manage. Namespace’s name is arbitrary. diff --git a/contributors/design-proposals/admission_control.md b/contributors/design-proposals/api-machinery/admission_control.md index a7330104..dec92334 100644 --- a/contributors/design-proposals/admission_control.md +++ b/contributors/design-proposals/api-machinery/admission_control.md @@ -99,8 +99,3 @@ following: - If operation=connect, exec If at any step, there is an error, the request is canceled. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/api-machinery/admission_control_event_rate_limit.md b/contributors/design-proposals/api-machinery/admission_control_event_rate_limit.md new file mode 100644 index 00000000..af25e4bf --- /dev/null +++ b/contributors/design-proposals/api-machinery/admission_control_event_rate_limit.md @@ -0,0 +1,176 @@ +# Admission control plugin: EventRateLimit + +## Background + +This document proposes a system for using an admission control to enforce a limit +on the number of event requests that the API Server will accept in a given time +slice. In a large cluster with many namespaces managed by disparate administrators, +there may be a small percentage of namespaces that have pods that are always in +some type of error state, for which the kubelets and controllers in the cluster +are producing a steady stream of error event requests. Each individual namespace +may not be causing a large amount of event requests on its own, but taken +collectively the errors from this small percentage of namespaces can have a +significant impact on the performance of the cluster overall. + +## Use cases + +1. Ability to protect the API Server from being flooded by event requests. +2. Ability to protect the API Server from being flooded by event requests for + a particular namespace. +3. Ability to protect the API Server from being flooded by event requests for + a particular user. +4. Ability to protect the API Server from being flooded by event requests from + a particular source+object. + +## Data Model + +### Configuration + +```go +// LimitType is the type of the limit (e.g., per-namespace) +type LimitType string + +const ( + // ServerLimitType is a type of limit where there is one bucket shared by + // all of the event queries received by the API Server. + ServerLimitType LimitType = "server" + // NamespaceLimitType is a type of limit where there is one bucket used by + // each namespace + NamespaceLimitType LimitType = "namespace" + // UserLimitType is a type of limit where there is one bucket used by each + // user + UserLimitType LimitType = "user" + // SourceAndObjectLimitType is a type of limit where there is one bucket used + // by each combination of source and involved object of the event. + SourceAndObjectLimitType LimitType = "sourceAndObject" +) + +// Configuration provides configuration for the EventRateLimit admission +// controller. +type Configuration struct { + metav1.TypeMeta `json:",inline"` + + // limits are the limits to place on event queries received. + // Limits can be placed on events received server-wide, per namespace, + // per user, and per source+object. + // At least one limit is required. + Limits []Limit `json:"limits"` +} + +// Limit is the configuration for a particular limit type +type Limit struct { + // type is the type of limit to which this configuration applies + Type LimitType `json:"type"` + + // qps is the number of event queries per second that are allowed for this + // type of limit. The qps and burst fields are used together to determine if + // a particular event query is accepted. The qps determines how many queries + // are accepted once the burst amount of queries has been exhausted. + QPS int32 `json:"qps"` + + // burst is the burst number of event queries that are allowed for this type + // of limit. The qps and burst fields are used together to determine if a + // particular event query is accepted. The burst determines the maximum size + // of the allowance granted for a particular bucket. For example, if the burst + // is 10 and the qps is 3, then the admission control will accept 10 queries + // before blocking any queries. Every second, 3 more queries will be allowed. + // If some of that allowance is not used, then it will roll over to the next + // second, until the maximum allowance of 10 is reached. + Burst int32 `json:"burst"` + + // cacheSize is the size of the LRU cache for this type of limit. If a bucket + // is evicted from the cache, then the allowance for that bucket is reset. If + // more queries are later received for an evicted bucket, then that bucket + // will re-enter the cache with a clean slate, giving that bucket a full + // allowance of burst queries. + // + // The default cache size is 4096. + // + // If limitType is 'server', then cacheSize is ignored. + // +optional + CacheSize int32 `json:"cacheSize,omitempty"` +} +``` + +### Validation + +Validation of a **Configuration** enforces that the following rules apply: + +* There is at least one item in **Limits**. +* Each item in **Limits** has a unique **Type**. + +Validation of a **Limit** enforces that the following rules apply: + +* **Type** is one of "server", "namespace", "user", and "source+object". +* **QPS** is positive. +* **Burst** is positive. +* **CacheSize** is non-negative. + +### Default Value Behavior + +If there is no item in **Limits** for a particular limit type, then no limits +will be enforced for that type of limit. + +## AdmissionControl plugin: EventRateLimit + +The **EventRateLimit** plug-in introspects all incoming event requests and +determines whether the event fits within the rate limits configured. + +To enable the plug-in and support for EventRateLimit, the kube-apiserver must +be configured as follows: + +```console +$ kube-apiserver --admission-control=EventRateLimit --admission-control-config-file=$ADMISSION_CONTROL_CONFIG_FILE +``` + +## Example + +An example EventRateLimit configuration: + +| Type | RequestBurst | RequestRefillRate | CacheSize | +| ---- | ------------ | ----------------- | --------- | +| Server | 1000 | 100 | | +| Namespace | 100 | 10 | 50 | + +The API Server starts with an allowance to accept 1000 event requests. Each +event request received counts against that allowance. The API Server refills +the allowance at a rate of 100 per second, up to a maximum allowance of 1000. +If the allowance is exhausted, then the API Server will respond to subsequent +event requests with 429 Too Many Requests, until the API Server adds more to +its allowance. + +For example, let us say that at time t the API Server has a full allowance to +accept 1000 event requests. At time t, the API Server receives 1500 event +requests. The first 1000 to be handled are accepted. The last 500 are rejected +with a 429 response. At time t + 1 second, the API Server has refilled its +allowance with 100 tokens. At time t + 1 second, the API Server receives +another 500 event requests. The first 100 to be handled are accepted. The last +400 are rejected. + +The API Server also starts with an allowance to accept 100 event requests from +each namespace. This allowance works in parallel with the server-wide +allowance. An accepted event request will count against both the server-side +allowance and the per-namespace allowance. An event request rejected by the +server-side allowance will still count against the per-namespace allowance, +and vice versa. The API Server tracks the allowances for at most 50 namespaces. +The API Server will stop tracking the allowance for the least-recently-used +namespace if event requests from more than 50 namespaces are received. If an +event request for namespace N is received after the API Server has stop +tracking the allowance for namespace N, then a new, full allowance will be +created for namespace N. + +In this example, the API Server will track any allowances for neither the user +nor the source+object in an event request because both the user and the +source+object details have been omitted from the configuration. The allowance +mechanisms for per-user and per-source+object rate limiting works identically +to the per-namespace rate limiting, with the exception that the former consider +the user of the event request or source+object of the event and the latter +considers the namespace of the event request. + +## Client Behavior + +Currently, the Client event recorder treats a 429 response as an http transport +type of error, which warrants retrying the event request. Instead, the event +recorder should abandon the event. Additionally, the event recorder should +abandon all future events for the period of time specified in the +Retry-After header of the 429 response. diff --git a/contributors/design-proposals/admission_control_extension.md b/contributors/design-proposals/api-machinery/admission_control_extension.md index a3be032f..c7e7f1b7 100644 --- a/contributors/design-proposals/admission_control_extension.md +++ b/contributors/design-proposals/api-machinery/admission_control_extension.md @@ -68,7 +68,7 @@ Name | Code | Description AlwaysPullImages | alwayspullimages/admission.go | Forces the Kubelet to pull images to prevent pods from accessing private images that another user with credentials has already pulled to the node. LimitPodHardAntiAffinityTopology | antiaffinity/admission.go | Defended the cluster against abusive anti-affinity topology rules that might hang the scheduler. DenyEscalatingExec | exec/admission.go | Prevent users from executing into pods that have higher privileges via their service account than allowed by their policy (regular users can't exec into admin pods). -DenyExecOnPrivileged | exec/admission.go | Blanket ban exec access to pods with host level security. Superceded by DenyEscalatingExec +DenyExecOnPrivileged | exec/admission.go | Blanket ban exec access to pods with host level security. Superseded by DenyEscalatingExec OwnerReferencesPermissionEnforcement | gc/gc_admission.go | Require that a user who sets a owner reference (which could result in garbage collection) has permission to delete the object, to prevent abuse. ImagePolicyWebhook | imagepolicy/admission.go | Invoke a remote API to determine whether an image is allowed to run on the cluster. PodNodeSelector | podnodeselector/admission.go | Default and limit what node selectors may be used within a namespace by reading a namespace annotation and a global configuration. @@ -632,9 +632,4 @@ Some options: It should be easy for a novice Kubernetes administrator to apply simple policy rules to the cluster. In the future it is desirable to have many such policy engines enabled via extension to enable quick policy -customization to meet specific needs. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> +customization to meet specific needs.
\ No newline at end of file diff --git a/contributors/design-proposals/aggregated-api-servers.md b/contributors/design-proposals/api-machinery/aggregated-api-servers.md index db939f1d..d1299a19 100644 --- a/contributors/design-proposals/aggregated-api-servers.md +++ b/contributors/design-proposals/api-machinery/aggregated-api-servers.md @@ -52,8 +52,8 @@ clients can always use the proxy and don't need to know that under the hood multiple apiservers are running. Wording note: When we say "API servers" we really mean groups of apiservers, -since any individual apiserver is horizontally replicatable. Similarly, -kube-aggregator itself is horizontally replicatable. +since any individual apiserver is horizontally replicable. Similarly, +kube-aggregator itself is horizontally replicable. ## Operational configurations @@ -80,9 +80,9 @@ There are two configurations in which it makes sense to run `kube-aggregator`. `api.mycompany.com/v1/grobinators` from different apiservers. This restriction allows us to limit the scope of `kube-aggregator` to a manageable level. * Follow API conventions: APIs exposed by every API server should adhere to [kubernetes API - conventions](../devel/api-conventions.md). + conventions](../../devel/api-conventions.md). * Support discovery API: Each API server should support the kubernetes discovery API - (list the suported groupVersions at `/apis` and list the supported resources + (list the supported groupVersions at `/apis` and list the supported resources at `/apis/<groupVersion>/`) * No bootstrap problem: The core kubernetes apiserver must not depend on any other aggregated server to come up. Non-core apiservers may use other non-core @@ -140,7 +140,7 @@ complete user information, including user, groups, and "extra" for backing API s Each API server is responsible for storing their resources. They can have their own etcd or can use kubernetes server's etcd using [third party -resources](../design-proposals/extending-api.md#adding-custom-resources-to-the-kubernetes-api-server). +resources](../design-proposals/api-machinery/extending-api.md#adding-custom-resources-to-the-kubernetes-api-server). ### Health check @@ -268,8 +268,3 @@ There were other alternatives that we had discussed. providing a centralised authentication and authorization service which all of the servers can use. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/api-machinery/alternate-api-representations.md b/contributors/design-proposals/api-machinery/alternate-api-representations.md new file mode 100644 index 00000000..e7635bc3 --- /dev/null +++ b/contributors/design-proposals/api-machinery/alternate-api-representations.md @@ -0,0 +1,433 @@ +# Alternate representations of API resources + +## Abstract + +Naive clients benefit from allowing the server to returning resource information in a form +that is easy to represent or is more efficient when dealing with resources in bulk. It +should be possible to ask an API server to return a representation of one or more resources +of the same type in a way useful for: + +* Retrieving a subset of object metadata in a list or watch of a resource, such as the + metadata needed by the generic Garbage Collector or the Namespace Lifecycle Controller +* Dealing with generic operations like `Scale` correctly from a client across multiple API + groups, versions, or servers +* Return a simple tabular representation of an object or list of objects for naive + web or command-line clients to display (for `kubectl get`) +* Return a simple description of an object that can be displayed in a wide range of clients + (for `kubectl describe`) +* Return the object with fields set by the server cleared (as `kubectl export`) which + is dependent on the schema, not on user input. + +The server should allow a common mechanism for a client to request a resource be returned +in one of a number of possible forms. In general, many of these forms are simply alternate +versions of the existing content and are not intended to support arbitrary parameterization. + +Also, the server today contains a number of objects which are common across multiple groups, +but which clients must be able to deal with in a generic fashion. These objects - Status, +ListMeta, ObjectMeta, List, ListOptions, ExportOptions, and Scale - are embedded into each +group version but are actually part of a a shared API group. It must be possible for a naive +client to translate the Scale response returned by two different API group versions. + + +## Motivation + +Currently it is difficult for a naive client (dealing only with the list of resources +presented by API discovery) to properly handle new and extended API groups, especially +as versions of those groups begin to evolve. It must be possible for a naive client to +perform a set of common operations across a wide range of groups and versions and leverage +a predictable schema. + +We also foresee increasing difficulty in building clients that must deal with extensions - +there are at least 6 known web-ui or CLI implementations that need to display some +information about third party resources or additional API groups registered with a server +without requiring each of them to change. Providing a server side implementation will +allow clients to retrieve meaningful information for the `get` and `describe` style +operations even for new API groups. + + +## Implementation + +The HTTP spec and the common REST paradigm provide mechanisms for clients to [negotiate +alternative representations of objects (RFC2616 14.1)](http://www.w3.org/Protocols/rfc2616/rfc2616.txt) +and for the server to correctly indicate a requested mechanism was chosen via the `Accept` +and `Content-Type` headers. This is a standard request response protocol intended to allow +clients to request the server choose a representation to return to the client based on the +server's capabilities. In RESTful terminology, a representation is simply a known schema that +the client is capable of handling - common schemas are HTML, JSON, XML, or protobuf, with the +possibility of the client and server further refining the requested output via either query +parameters or media type parameters. + +In order to ensure that generic clients can properly deal with many different group versions, +we introduce the `meta.k8s.io` group with version `v1` that grandfathers all existing resources +currently described as "unversioned". A generic client may request that responses be applied +in this version. The contents of a particular API group version would continue to be bound into +other group versions (`status.v1.meta.k8s.io` would be bound as `Status` into all existing +API groups). We would remove the `unversioned` package and properly home these resources in +a real API group. + + +### Considerations around choosing an implementation + +* We wish to avoid creating new resource *locations* (URLs) for existing resources + * New resource locations complicate access control, caching, and proxying + * We are still retrieving the same resource, just in an alternate representation, + which matches our current use of the protobuf, JSON, and YAML serializations + * We do not wish to alter the mechanism for authorization - a user with access + to a particular resource in a given namespace should be limited regardless of + the representation in use. + * Allowing "all namespaces" to be listed would require us to create "fake" resources + which would complicate authorization +* We wish to support retrieving object representations in multiple schemas - JSON for + simple clients and Protobuf for clients concerned with efficiency. +* Most clients will wish to retrieve a newer format, but for older servers will desire + to fall back to the implicit resource represented by the endpoint. + * Over time, clients may need to request results in multiple API group versions + because of breaking changes (when we introduce v2, clients that know v2 will want + to ask for v2, then v1) + * The Scale resource is an example - a generic client may know v1 Scale, but when + v2 Scale is introduced the generic client will still only request v1 Scale from + any given resource, and the server that no longer recognizes v1 Scale must + indicate that to the client. +* We wish to preserve the greatest possible query parameter space for sub resources + and special cases, which encourages us to avoid polluting the API with query + parameters that can be otherwise represented as alternate forms. +* We do not wish to allow deep orthogonal parameterization - a list of pods is a list + of pods regardless of the form, and the parameters passed to the JSON representation + should not vary significantly to the tabular representation. +* Because we expect not all extensions will implement protobuf, an efficient client + must continue to be able to "fall-back" to JSON, such as for third party + resources. +* We do not wish to create fake content-types like `application/json+kubernetes+v1+meta.k8s.io` + because the list of combinations is unbounded and our ability to encode specific values + (like slashes) into the value is limited. + +### Client negotiation of response representation + +When a client wishes to request an alternate representation of an object, it should form +a valid `Accept` header containing one or more accepted representations, where each +representation is represented by a media-type and [media-type parameters](https://tools.ietf.org/html/rfc6838#section-4.3). +The server should omit representations that are unrecognized or in error - if no representations +are left after omission the server should return a `406 Not Acceptable` HTTP response. + +The supported parameters are: + +| Name | Value | Default | Description | +| ---- | ----- | ------- | ----------- | +| g | The group name of the desired response | Current group | The group the response is expected in. | +| v | The version of the desired response | Current version | The version the response is expected in. Note that this is separate from Group because `/` is not a valid character in Accept headers. | +| as | Kind name | None | If specified, transform the resource into the following kind (including the group and version parameters). | +| sv | The server group (`meta.k8s.io`) version that should be applied to generic resources returned by this endpoint | Matching server version for the current group and version | If specified, the server should transform generic responses into this version of the server API group. | +| export | `1` | None | If specified, transform the resource prior to returning to omit defaulted fields. Additional arguments allowed in the query parameter. For legacy reasons, `?export=1` will continue to be supported on the request | +| pretty | `0`/`1` | `1` | If specified, apply formatting to the returned response that makes the serialization readable (for JSON, use indentation) | + +Examples: + +``` +# Request a PodList in an alternate form +GET /v1/pods +Accept: application/json;as=Table;g=meta.k8s.io;v=v1 + +# Request a PodList in an alternate form, with pretty JSON formatting +GET /v1/pods +Accept: application/json;as=Table;g=meta.k8s.io;v=v1;pretty=1 + +# Request that status messages be of the form meta.k8s.io/v2 on the response +GET /v1/pods +Accept: application/json;sv=v2 +{ + "kind": "Status", + "apiVersion": "meta.k8s.io/v2", + ... +} +``` + +For both export and the more complicated server side `kubectl get` cases, it's likely that +more parameters are required and should be specified as query parameters. However, the core +behavior is best represented as a variation on content-type. Supporting both is not limiting +in the short term as long as we can validate correctly. + +As a simplification for common use, we should create **media-type aliases** which may show up in lists of mime-types supported +and simplify use for clients. For example, the following aliases would be reasonable: + +* `application/json+vnd.kubernetes.export` would return the requested object in export form +* `application/json+vnd.kubernetes.as+meta.k8s.io+v1+TabularOutput` would return the requested object in a tabular form +* `text/csv` would return the requested object in a tabular form in the comma-separated-value (CSV) format + +### Example: Partial metadata retrieval + +The client may request to the server to return the list of namespaces as a +`PartialObjectMetadata` kind, which is an object containing only `ObjectMeta` and +can be serialized as protobuf or JSON. This is expected to be significantly more +performant when controllers like the Garbage collector retrieve multiple objects. + + GET /api/v1/namespaces + Accept: application/json;g=meta.k8s.io,v=v1,as=PartialObjectMetadata, application/json + +The server would respond with + + 200 OK + Content-Type: application/json;g=meta.k8s.io,v=v1,as=PartialObjectMetadata + { + "apiVersion": "meta.k8s.io/v1", + "kind": "PartialObjectMetadataList", + "items": [ + { + "apiVersion": "meta.k8s.io/v1", + "kind": "PartialObjectMetadata", + "metadata": { + "name": "foo", + "resourceVersion": "10", + ... + } + }, + ... + ] + } + +In this example PartialObjectMetadata is a real registered type, and each API group +provides an efficient transformation from their schema to the partial schema directly. +The client upon retrieving this type can act as a generic resource. + +Note that the `as` parameter indicates to the server the Kind of the resource, but +the Kubernetes API convention of returning a List with a known schema continues. An older +server could ignore the presence of the `as` parameter on the media type and merely return +a `NamespaceList` and the client would either use the content-type or the object Kind +to distinguish. Because all responses are expected to be self-describing, an existing +Kubernetes client would be expected to differentiate on Kind. + +An old server, not recognizing these parameters, would respond with: + + 200 OK + Content-Type: application/json + { + "apiVersion": "v1", + "kind": "NamespaceList", + "items": [ + { + "apiVersion": "v1", + "kind": "Namespace", + "metadata": { + "name": "foo", + "resourceVersion": "10", + ... + } + }, + ... + ] + } + + +### Example: Retrieving a known version of the Scale resource + +Each API group that supports resources that can be scaled must expose a subresource on +their object that accepts GET or PUT with a `Scale` kind resource. This subresource acts +as a generic interface that a client that knows nothing about the underlying object can +use to modify the scale value of that resource. However, clients *must* be able to understand +the response the server provides, and over time the response may change and should therefore +be versioned. Our current API provides no way for a client to discover whether a `Scale` +response returned by `batch/v2alpha1` is the same as the `Scale` resource returned by +`autoscaling/v1`. + +Under this proposal, to scale a generic resource a client would perform the following +operations: + + GET /api/v1/namespace/example/replicasets/test/scale + Accept: application/json;g=meta.k8s.io,v=v1,as=Scale, application/json + + 200 OK + Content-Type: application/json;g=meta.k8s.io,v=v1,as=Scale + { + "apiVersion": "meta.k8s.io/v1", + "kind": "Scale", + "spec": { + "replicas": 1 + } + ... + } + +The client, seeing that a generic response was returned (`meta.k8s.io/v1`), knows that +the server supports accepting that resource as well, and performs a PUT: + + PUT /apis/extensions/v1beta1/namespace/example/replicasets/test/scale + Accept: application/json;g=meta.k8s.io,v=v1,as=Scale, application/json + Content-Type: application/json + { + "apiVersion": "meta.k8s.io/v1", + "kind": "Scale", + "spec": { + "replicas": 2 + } + } + + 200 OK + Content-Type: application/json;g=meta.k8s.io,v=v1,as=Scale + { + "apiVersion": "meta.k8s.io/v1", + "kind": "Scale", + "spec": { + "replicas": 2 + } + ... + } + +Note that the client still asks for the common Scale as the response so that it +can access the value it wants. + + +### Example: Retrieving an alternative representation of the resource for use in `kubectl get` + +As new extension groups are added to the server, all clients must implement simple "view" logic +for each resource. However, these views are specific to the resource in question, which only +the server is aware of. To make clients more tolerant of extension and third party resources, +it should be possible for clients to ask the server to present a resource or list of resources +in a tabular / descriptive format rather than raw JSON. + +While the design of serverside tabular support is outside the scope of this proposal, a few +knows apply. The server must return a structured resource usable by both command line and +rich clients (web or IDE), which implies a schema, which implies JSON, and which means the +server should return a known Kind. For this example we will call that kind `TabularOutput` +to demonstrate the concept. + +A server side resource would implement a transformation from their resource to `TabularOutput` +and the API machinery would translate a single item or a list of items (or a watch) into +the tabular resource. + +A generic client wishing to display a tabular list for resources of type `v1.ReplicaSets` would +make the following call: + + GET /api/v1/namespaces/example/replicasets + Accept: application/json;g=meta.k8s.io,v=v1,as=TabularOutput, application/json + + 200 OK + Content-Type: application/json;g=meta.k8s.io,v=v1,as=TabularOutput + { + "apiVersion": "meta.k8s.io/v1", + "kind": "TabularOutput", + "columns": [ + {"name": "Name", "description": "The name of the resource"}, + {"name": "Resource Version", "description": "The version of the resource"}, + ... + ], + "items": [ + {"columns": ["name", "10", ...]}, + ... + ] + } + +The client can then present that information as necessary. If the server returns the +resource list `v1.ReplicaSetList` the client knows that the server does not support tabular +output and so must fall back to a generic output form (perhaps using the existing +compiled in listers). + +Note that `kubectl get` supports a number of parameters for modifying the response, +including whether to filter resources, whether to show a "wide" list, or whether to +turn certain labels into columns. Those options are best represented as query parameters +and transformed into a known type. + + +### Example: Versioning a ListOptions call to a generic API server + +When retrieving lists of resources, the server transforms input query parameters like +`labels` and `fields` into a `ListOptions` type. It should be possible for a generic +client dealing with the server to be able to specify the version of ListOptions it +is sending to detect version skew. + +Since this is an input and list is implemented with GET, it is not possible to send +a body and no Content-Type is possible. For this approach, we recommend that the kind +and API version be specifiable via the GET call for further clarification: + +New query parameters: + +| Name | Value | Default | Description | +| ---- | ----- | ------- | ----------- | +| kind | The kind of parameters being sent | `ListOptions` (GET), `DeleteOptions` (DELETE) | The kind of the serialized struct, defaults to ListOptions on GET and DeleteOptions on DELETE. | +| queryVersion / apiVersion | The API version of the parameter struct | `meta.k8s.io/v1` | May be altered to match the expected version. Because we have not yet versioned ListOptions, this is safe to alter. | + +To send ListOptions in the v2 future format, where the serialization of `resourceVersion` +is changed to `rv`, clients would provide: + + GET /api/v1/namespaces/example/replicasets?apiVersion=meta.k8s.io/v2&rv=10 + +Before we introduce a second API group version, we would have to ensure old servers +properly reject apiVersions they do not understand. + + +### Impact on web infrastructure + +In the past, web infrastructure and old browsers have coped poorly with the `Accept` +header. However, most modern caching infrastructure properly supports `Vary: Accept` +and caching of responses has not been a significant requirement for Kubernetes APIs +to this point. + + +### Considerations for discoverability + +To ensure clients can discover these endpoints, the Swagger and OpenAPI documents +should also include a set of example mime-types for each endpoint that are supported. +Specifically, the `produces` field on an individual operation can be used to list a +set of well known types. The description of the operation can include a stanza about +retrieving alternate representations. + + +## Alternatives considered + +* Implement only with query parameters + + To properly implement alternative resource versions must support multiple version + support (ask for v2, then v1). The Accept mechanism already handles this sort of + multi-version negotiation, while any approach based on query parameters would + have to implement this option as well. In addition, some serializations may not + be valid in all content types, so the client asking for TabularOutput in protobuf + may also ask for TabularOutput in JSON - if TabularOutput is not valid in protobuf + the server call fall back to JSON. + +* Use new resource paths - `/apis/autoscaling/v1/namespaces/example/horizontalpodautoscalermetadata` + + This leads to a proliferation of paths which will confuse automated tools and end + users. Authorization, logging, audit may all need a way to map the two resources + as equivalent, while clients would need a discovery mechanism that identifies a + "same underlying object" relationship that is different from subresources. + +* Use a special HTTP header to denote the alternative representation + + Given the need to support multiple versions, this would be reimplementing Accept + in a slightly different way, so we prefer to reuse Accept. + +* For partial object retrieval, support complex field selectors + + From an efficiency perspective, calculating subpaths and filtering out sub fields + from the underlying object is complex. In practice, almost all filtering falls into + a few limited subsets, and thus retrieving an object into a few known schemas can be made + much more efficient. In addition, arbitrary transformation of the object provides + opportunities for supporting forward "partial" migration - for instance, returning a + ReplicationController as a ReplicaSet to simplify a transition across resource types. + While this is not under explicit consideration, allowing a caller to move objects across + schemas will eventually be a required behavior when dramatic changes occur in an API + schema. + +## Backwards Compatibility + +### Old clients + +Old clients would not be affected by the new Accept path. + +If servers begin returning Status in version `meta.k8s.io/v1`, old clients would likely error +as that group has never been used. We would continue to return the group version of the calling +API group on server responses unless the `sv` mime-type parameter is set. + + +### Old servers + +Because old Kubernetes servers are not selective about the content type parameters they +accept, we may wish to patch server versions to explicitly bypass content +types they do not recognize the parameters to. As a special consideration, this would allow +new clients to more strictly handle Accept (so that the server returns errors if the content +type is not recognized). + +As part of introducing the new API group `meta.k8s.io`, some opaque calls where we assume the +empty API group-version for the resource (GET parameters) could be defaulted to this group. + + +## Future items + +* ??? diff --git a/contributors/design-proposals/api-machinery/api-chunking.md b/contributors/design-proposals/api-machinery/api-chunking.md new file mode 100644 index 00000000..4930192a --- /dev/null +++ b/contributors/design-proposals/api-machinery/api-chunking.md @@ -0,0 +1,177 @@ +# Allow clients to retrieve consistent API lists in chunks + +On large clusters, performing API queries that return all of the objects of a given resource type (GET /api/v1/pods, GET /api/v1/secrets) can lead to significant variations in peak memory use on the server and contribute substantially to long tail request latency. + +When loading very large sets of objects -- some clusters are now reaching 100k pods or equivalent numbers of supporting resources -- the system must: + +* Construct the full range description in etcd in memory and serialize it as protobuf in the client + * Some clusters have reported over 500MB being stored in a single object type + * This data is read from the underlying datastore and converted to a protobuf response + * Large reads to etcd can block writes to the same range (https://github.com/coreos/etcd/issues/7719) +* The data from etcd has to be transferred to the apiserver in one large chunk +* The `kube-apiserver` also has to deserialize that response into a single object, and then re-serialize it back to the client + * Much of the decoded etcd memory is copied into the struct used to serialize to the client +* An API client like `kubectl get` will then decode the response from JSON or protobuf + * An API client with a slow connection may not be able to receive the entire response body within the default 60s timeout + * This may cause other failures downstream of that API client with their own timeouts + * The recently introduced client compression feature can assist + * The large response will also be loaded entirely into memory + +The standard solution for reducing the impact of large reads is to allow them to be broken into smaller reads via a technique commonly referred to as paging or chunking. By efficiently splitting large list ranges from etcd to clients into many smaller list ranges, we can reduce the peak memory allocation on etcd and the apiserver, without losing the consistent read invariant our clients depend on. + +This proposal does not cover general purpose ranging or paging for arbitrary clients, such as allowing web user interfaces to offer paged output, but does define some parameters for future extension. To that end, this proposal uses the phrase "chunking" to describe retrieving a consistent snapshot range read from the API server in distinct pieces. + +Our primary consistent store etcd3 offers support for efficient chunking with minimal overhead, and mechanisms exist for other potential future stores such as SQL databases or Consul to also implement a simple form of consistent chunking. + +Relevant issues: + +* https://github.com/kubernetes/kubernetes/issues/2349 + +## Terminology + +**Consistent list** - A snapshot of all resources at a particular moment in time that has a single `resourceVersion` that clients can begin watching from to receive updates. All Kubernetes controllers depend on this semantic. Allows a controller to refresh its internal state, and then receive a stream of changes from the initial state. + +**API paging** - API parameters designed to allow a human to view results in a series of "pages". + +**API chunking** - API parameters designed to allow a client to break one large request into multiple smaller requests without changing the semantics of the original request. + + +## Proposed change: + +Expose a simple chunking mechanism to allow large API responses to be broken into consistent partial responses. Clients would indicate a tolerance for chunking (opt-in) by specifying a desired maximum number of results to return in a `LIST` call. The server would return up to that amount of objects, and if more exist it would return a `continue` parameter that the client could pass to receive the next set of results. The server would be allowed to ignore the limit if it does not implement limiting (backward compatible), but it is not allowed to support limiting without supporting a way to continue the query past the limit (may not implement `limit` without `continue`). + +``` +GET /api/v1/pods?limit=500 +{ + "metadata": {"continue": "ABC...", "resourceVersion": "147"}, + "items": [ + // no more than 500 items + ] +} +GET /api/v1/pods?limit=500&continue=ABC... +{ + "metadata": {"continue": "DEF...", "resourceVersion": "147"}, + "items": [ + // no more than 500 items + ] +} +GET /api/v1/pods?limit=500&continue=DEF... +{ + "metadata": {"resourceVersion": "147"}, + "items": [ + // no more than 500 items + ] +} +``` + +The token returned by the server for `continue` would be an opaque serialized string that would contain a simple serialization of a version identifier (to allow future extension), and any additional data needed by the server storage to identify where to start the next range. + +The continue token is not required to encode other filtering parameters present on the initial request, and clients may alter their filter parameters on subsequent chunk reads. However, the server implementation **may** reject such changes with a `400 Bad Request` error, and clients should consider this behavior undefined and left to future clarification. Chunking is intended to return consistent lists, and clients **should not** alter their filter parameters on subsequent chunk reads. + +If the resource version parameter specified on the request is inconsistent with the `continue` token, the server **must** reject the request with a `400 Bad Request` error. + +The schema of the continue token is chosen by the storage layer and is not guaranteed to remain consistent for clients - clients **must** consider the continue token as opaque. Server implementations **should** ensure that continue tokens can persist across server restarts and across upgrades. + +Servers **may** return fewer results than `limit` if server side filtering returns no results such as when a `label` or `field` selector is used. If the entire result set is filtered, the server **may** return zero results with a valid `continue` token. A client **must** use the presence of a `continue` token in the response to determine whether more results are available, regardless of the number of results returned. A server that supports limits **must not** return more results than `limit` if a `continue` token is also returned. If the server does not return a `continue` token, the server **must** return all remaining results. The server **may** return zero results with no `continue` token on the last call. + +The server **may** limit the amount of time a continue token is valid for. Clients **should** assume continue tokens last only a few minutes. + +The server **must** support `continue` tokens that are valid across multiple API servers. The server **must** support a mechanism for rolling restart such that continue tokens are valid after one or all API servers have been restarted. + + +### Proposed Implementations + +etcd3 is the primary Kubernetes store and has been designed to support consistent range reads in chunks for this use case. The etcd3 store is an ordered map of keys to values, and Kubernetes places all keys within a resource type under a common prefix, with namespaces being a further prefix of those keys. A read of all keys within a resource type is an in-order scan of the etcd3 map, and therefore we can retrieve in chunks by defining a start key for the next chunk that skips the last key read. + +etcd2 will not be supported as it has no option to perform a consistent read and is on track to be deprecated in Kubernetes. Other databases that might back Kubernetes could either choose to not implement limiting, or leverage their own transactional characteristics to return a consistent list. In the near term our primary store remains etcd3 which can provide this capability at low complexity. + +Implementations that cannot offer consistent ranging (returning a set of results that are logically equivalent to receiving all results in one response) must not allow continuation, because consistent listing is a requirement of the Kubernetes API list and watch pattern. + +#### etcd3 + +For etcd3 the continue token would contain a resource version (the snapshot that we are reading that is consistent across the entire LIST) and the start key for the next set of results. Upon receiving a valid continue token the apiserver would instruct etcd3 to retrieve the set of results at a given resource version, beginning at the provided start key, limited by the maximum number of requests provided by the continue token (or optionally, by a different limit specified by the client). If more results remain after reading up to the limit, the storage should calculate a continue token that would begin at the next possible key, and the continue token set on the returned list. + +The storage layer in the apiserver must apply consistency checking to the provided continue token to ensure that malicious users cannot trick the server into serving results outside of its range. The storage layer must perform defensive checking on the provided value, check for path traversal attacks, and have stable versioning for the continue token. + +#### Possible SQL database implementation + +A SQL database backing a Kubernetes server would need to implement a consistent snapshot read of an entire resource type, plus support changefeed style updates in order to implement the WATCH primitive. A likely implementation in SQL would be a table that stores multiple versions of each object, ordered by key and version, and filters out all historical versions of an object. A consistent paged list over such a table might be similar to: + + SELECT * FROM resource_type WHERE resourceVersion < ? AND deleted = false AND namespace > ? AND name > ? LIMIT ? ORDER BY namespace, name ASC + +where `namespace` and `name` are part of the continuation token and an index exists over `(namespace, name, resourceVersion, deleted)` that makes the range query performant. The highest returned resource version row for each `(namespace, name)` tuple would be returned. + + +### Security implications of returning last or next key in the continue token + +If the continue token encodes the next key in the range, that key may expose info that is considered security sensitive, whether simply the name or namespace of resources not under the current tenant's control, or more seriously the name of a resource which is also a shared secret (for example, an access token stored as a kubernetes resource). There are a number of approaches to mitigating this impact: + +1. Disable chunking on specific resources +2. Disable chunking when the user does not have permission to view all resources within a range +3. Encrypt the next key or the continue token using a shared secret across all API servers +4. When chunking, continue reading until the next visible start key is located after filtering, so that start keys are always keys the user has access to. + +In the short term we have no supported subset filtering (i.e. a user who can LIST can also LIST ?fields= and vice versa), so 1 is sufficient to address the sensitive key name issue. Because clients are required to proceed as if limiting is not possible, the server is always free to ignore a chunked request for other reasons. In the future, 4 may be the best option because we assume that most users starting a consistent read intend to finish it, unlike more general user interface paging where only a small fraction of requests continue to the next page. + + +### Handling expired resource versions + +If the required data to perform a consistent list is no longer available in the storage backend (by default, old versions of objects in etcd3 are removed after 5 minutes), the server **must** return a `410 Gone ResourceExpired` status response (the same as for watch), which means clients must start from the beginning. + +``` +# resourceVersion is expired +GET /api/v1/pods?limit=500&continue=DEF... +{ + "kind": "Status", + "code": 410, + "reason": "ResourceExpired" +} +``` + +Some clients may wish to follow a failed paged list with a full list attempt. + +The 5 minute default compaction interval for etcd3 bounds how long a list can run. Since clients may wish to perform processing over very large sets, increasing that timeout may make sense for large clusters. It should be possible to alter the interval at which compaction runs to accomodate larger clusters. + + +#### Types of clients and impact + +Some clients such as controllers, receiving a 410 error, may instead wish to perform a full LIST without chunking. + +* Controllers with full caches + * Any controller with a full in-memory cache of one or more resources almost certainly depends on having a consistent view of resources, and so will either need to perform a full list or a paged list, without dropping results +* `kubectl get` + * Most administrators would probably prefer to see a very large set with some inconsistency rather than no results (due to a timeout under load). They would likely be ok with handling `410 ResourceExpired` as "continue from the last key I processed" +* Migration style commands + * Assuming a migration command has to run on the full data set (to upgrade a resource from json to protobuf, or to check a large set of resources for errors) and is performing some expensive calculation on each, very large sets may not complete over the server expiration window. + +For clients that do not care about consistency, the server **may** return a `continue` value on the `ResourceExpired` error that allows the client to restart from the same prefix key, but using the latest resource version. This would allow clients that do not require a fully consistent LIST to opt in to partially consistent LISTs but still be able to scan the entire working set. It is likely this could be a sub field (opaque data) of the `Status` response under `statusDetails`. + + +### Rate limiting + +Since the goal is to reduce spikiness of load, the standard API rate limiter might prefer to rate limit page requests differently from global lists, allowing full LISTs only slowly while smaller pages can proceed more quickly. + + +### Chunk by default? + +On a very large data set, chunking trades total memory allocated in etcd, the apiserver, and the client for higher overhead per request (request/response processing, authentication, authorization). Picking a sufficiently high chunk value like 500 or 1000 would not impact smaller clusters, but would reduce the peak memory load of a very large cluster (10k resources and up). In testing, no significant overhead was shown in etcd3 for a paged historical query which is expected since the etcd3 store is an MVCC store and must always filter some values to serve a list. + +For clients that must perform sequential processing of lists (kubectl get, migration commands) this change dramatically improves initial latency - clients got their first chunk of data in milliseconds, rather than seconds for the full set. It also improves user experience for web consoles that may be accessed by administrators with access to large parts of the system. + +It is recommended that most clients attempt to page by default at a large page size (500 or 1000) and gracefully degrade to not chunking. + + +### Other solutions + +Compression from the apiserver and between the apiserver and etcd can reduce total network bandwidth, but cannot reduce the peak CPU and memory used inside the client, apiserver, or etcd processes. + +Various optimizations exist that can and should be applied to minimizing the amount of data that is transferred from etcd to the client or number of allocations made in each location, but do not how response size scales with number of entries. + + +## Plan + +The initial chunking implementation would focus on consistent listing on server and client as well as measuring the impact of chunking on total system load, since chunking will slightly increase the cost to view large data sets because of the additional per page processing. The initial implementation should make the fewest assumptions possible in constraining future backend storage. + +For the initial alpha release, chunking would be behind a feature flag and attempts to provide the `continue` or `limit` flags should be ignored. While disabled, a `continue` token should never be returned by the server as part of a list. + +Future work might offer more options for clients to page in an inconsistent fashion, or allow clients to directly specify the parts of the namespace / name keyspace they wish to range over (paging). diff --git a/contributors/design-proposals/api-group.md b/contributors/design-proposals/api-machinery/api-group.md index 28b98216..442c9ca8 100644 --- a/contributors/design-proposals/api-group.md +++ b/contributors/design-proposals/api-machinery/api-group.md @@ -113,7 +113,3 @@ To expose a list of the supported Openshift groups to clients, OpenShift just ha ## Future work 1. Dependencies between groups: we need an interface to register the dependencies between groups. It is not our priority now as the use cases are not clear yet. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/api-machinery/apiserver-build-in-admission-plugins.md b/contributors/design-proposals/api-machinery/apiserver-build-in-admission-plugins.md new file mode 100644 index 00000000..cefaf8fd --- /dev/null +++ b/contributors/design-proposals/api-machinery/apiserver-build-in-admission-plugins.md @@ -0,0 +1,80 @@ +# Build some Admission Controllers into the Generic API server library + +**Related PR:** + +| Topic | Link | +| ----- | ---- | +| Admission Control | https://git.k8s.io/community/contributors/design-proposals/api-machinery/admission_control.md | + +## Introduction + +An admission controller is a piece of code that intercepts requests to the Kubernetes API - think a middleware. +The API server lets you have a whole chain of them. Each is run in sequence before a request is accepted +into the cluster. If any of the plugins in the sequence rejects the request, the entire request is rejected +immediately and an error is returned to the user. + +Many features in Kubernetes require an admission control plugin to be enabled in order to properly support the feature. +In fact in the [documentation](https://kubernetes.io/docs/admin/admission-controllers/#is-there-a-recommended-set-of-plug-ins-to-use) you will find +a recommended set of them to use. + +At the moment admission controllers are implemented as plugins and they have to be compiled into the +final binary in order to be used at a later time. Some even require an access to cache, an authorizer etc. +This is where an admission plugin initializer kicks in. An admission plugin initializer is used to pass additional +configuration and runtime references to a cache, a client and an authorizer. + +To streamline the process of adding new plugins especially for aggregated API servers we would like to build some plugins +into the generic API server library and provide a plugin initializer. While anyone can author and register one, having a known set of +provided references let's people focus on what they need their admission plugin to do instead of paying attention to wiring. + +## Implementation + +The first step would involve creating a "standard" plugin initializer that would be part of the +generic API server. It would use kubeconfig to populate +[external clients](https://git.k8s.io/kubernetes/pkg/kubeapiserver/admission/initializer.go#L29) +and [external informers](https://git.k8s.io/kubernetes/pkg/kubeapiserver/admission/initializer.go#L35). +By default for servers that would be run on the kubernetes cluster in-cluster config would be used. +The standard initializer would also provide a client config for connecting to the core kube-apiserver. +Some API servers might be started as static pods, which don't have in-cluster configs. +In that case the config could be easily populated form the file. + +The second step would be to move some plugins from [admission pkg](https://git.k8s.io/kubernetes/plugin/pkg/admission) +to the generic API server library. Some admission plugins are used to ensure consistent user expectations. +These plugins should be moved. One example is the Namespace Lifecycle plugin which prevents users +from creating resources in non-existent namespaces. + +*Note*: +For loading in-cluster configuration [visit](https://git.k8s.io/kubernetes/staging/src/k8s.io/client-go/examples/in-cluster-client-configuration/main.go) + For loading the configuration directly from a file [visit](https://git.k8s.io/kubernetes/staging/src/k8s.io/client-go/examples/out-of-cluster-client-configuration/main.go) + +## How to add an admission plugin ? + At this point adding an admission plugin is very simple and boils down to performing the +following series of steps: + 1. Write an admission plugin + 2. Register the plugin + 3. Reference the plugin in the admission chain + +## An example +The sample apiserver provides an example admission plugin that makes meaningful use of the "standard" plugin initializer. +The admission plugin ensures that a resource name is not on the list of banned names. +The source code of the plugin can be found [here](https://github.com/kubernetes/kubernetes/blob/2f00e6d72c9d58fe3edc3488a91948cf4bfcc6d9/staging/src/k8s.io/sample-apiserver/pkg/admission/plugin/banflunder/admission.go). + +Having the plugin, the next step is the registration. [AdmissionOptions](https://github.com/kubernetes/kubernetes/blob/2f00e6d72c9d58fe3edc3488a91948cf4bfcc6d9/staging/src/k8s.io/apiserver/pkg/server/options/admission.go) +provides two important things. Firstly it exposes [a register](https://github.com/kubernetes/kubernetes/blob/2f00e6d72c9d58fe3edc3488a91948cf4bfcc6d9/staging/src/k8s.io/apiserver/pkg/server/options/admission.go#L43) +under which all admission plugins are registered. In fact, that's exactly what the [Register](https://github.com/kubernetes/kubernetes/blob/2f00e6d72c9d58fe3edc3488a91948cf4bfcc6d9/staging/src/k8s.io/sample-apiserver/pkg/admission/plugin/banflunder/admission.go#L33) +method does from our example admission plugin. It accepts a global registry as a parameter and then simply registers itself in that registry. +Secondly, it adds an admission chain to the server configuration via [ApplyTo](https://github.com/kubernetes/kubernetes/blob/2f00e6d72c9d58fe3edc3488a91948cf4bfcc6d9/staging/src/k8s.io/apiserver/pkg/server/options/admission.go#L66) method. +The method accepts optional parameters in the form of `pluginInitalizers`. This is useful when admission plugins need custom configuration that is not provided by the generic initializer. + +The following code has been extracted from the sample server and illustrates how to register and wire an admission plugin: + +```go + // register admission plugins + banflunder.Register(o.Admission.Plugins) + + // create custom plugin initializer + informerFactory := informers.NewSharedInformerFactory(client, serverConfig.LoopbackClientConfig.Timeout) + admissionInitializer, _ := wardleinitializer.New(informerFactory) + + // add admission chain to the server configuration + o.Admission.ApplyTo(serverConfig, admissionInitializer) +``` diff --git a/contributors/design-proposals/api-machinery/apiserver-count-fix.md b/contributors/design-proposals/api-machinery/apiserver-count-fix.md new file mode 100644 index 00000000..6ea3d703 --- /dev/null +++ b/contributors/design-proposals/api-machinery/apiserver-count-fix.md @@ -0,0 +1,86 @@ +# apiserver-count fix proposal + +Authors: @rphillips + +## Table of Contents + +1. [Overview](#overview) +2. [Known Issues](#known-issues) +3. [Proposal](#proposal) +4. [Alternate Proposals](#alternate-proposals) + 1. [Custom Resource Definitions](#custom-resource-definitions) + 2. [Refactor Old Reconciler](#refactor-old-reconciler) + +## Overview + +Proposal to fix Issue [#22609](https://github.com/kubernetes/kubernetes/issues/22609) + +`kube-apiserver` currently has a command-line argument `--apiserver-count` +specifying the number of api servers. This masterCount is used in the +MasterCountEndpointReconciler on a 10 second interval to potentially cleanup +stale API Endpoints. The issue is when the number of kube-apiserver instances +gets below or above the masterCount. If the below case happens, the stale +instances within the Endpoints does not get cleaned up, or in the latter case +the endpoints start to flap. + +## Known Issues + +Each apiserver’s reconciler only cleans up for it's own IP. If a new +server is spun up at a new IP, then the old IP in the Endpoints list is +only reclaimed if the number of apiservers becomes greater-than or equal +to the masterCount. For example: + +* If the masterCount = 3, and there are 3 API servers running (named: A, B, and C) +* ‘B’ API server is terminated for any reason +* The IP for endpoint ‘B’ is not +removed from the Endpoints list + +There is logic within the +[MasterCountEndpointReconciler](https://github.com/kubernetes/kubernetes/blob/68814c0203c4b8abe59812b1093844a1f9bdac05/pkg/master/controller.go#L293) +to attempt to make the Endpoints eventually consistent, but the code relies on +the Endpoints count becoming equal to or greater than masterCount. When the +apiservers become greater than the masterCount the Endpoints tend to flap. + +If the number endpoints were scaled down from automation, then the +Endpoints would never become consistent. + +## Proposal + +### Create New Reconciler + +| Kubernetes Release | Quality | Description | +| ------------- | ------------- | ----------- | +| 1.9 | alpha | <ul><li>Add a new reconciler</li><li>Add a command-line type `--alpha-apiserver-endpoint-reconciler-type`<ul><li>storage</li><li>default</li></ul></li></ul> +| 1.10 | beta | <ul><li>Turn on the `storage` type by default</li></ul> +| 1.11 | stable | <ul><li>Remove code for old reconciler</li><li>Remove --apiserver-count</li></ul> + +The MasterCountEndpointReconciler does not meet the current needs for durability +of API Endpoint creation, deletion, or failure cases. + +Custom Resource Definitions were proposed, but they do not have clean layering. +Additionally, liveness and locking would be a nice to have feature for a long +term solution. + +ConfigMaps were proposed, but since they are watched globally, liveliness +updates could be overly chatty. + +By porting OpenShift's +[LeaseEndpointReconciler](https://github.com/openshift/origin/blob/master/pkg/cmd/server/election/lease_endpoint_reconciler.go) +to Kubernetes we can use use the Storage API directly to store Endpoints +dynamically within the system. + +### Alternate Proposals + +#### Custom Resource Definitions and ConfigMaps + +CRD's and ConfigMaps were considered for this proposal. They were not adopted +for this proposal by the community due to technical issues explained earlier. + +#### Refactor Old Reconciler + +| Release | Quality | Description | +| ------- | ------- | ------------------------------------------------------------ | +| 1.9 | stable | Change the logic in the current reconciler + +We could potentially reuse the old reconciler by changing the reconciler to count +the endpoints and set the `masterCount` (with a RWLock) to the count. diff --git a/contributors/design-proposals/apiserver-watch.md b/contributors/design-proposals/api-machinery/apiserver-watch.md index ef395036..7d509e4d 100644 --- a/contributors/design-proposals/apiserver-watch.md +++ b/contributors/design-proposals/api-machinery/apiserver-watch.md @@ -132,13 +132,8 @@ the same time, we can introduce an additional etcd event type: EtcdResync Thus, we need to create the EtcdResync event, extend watch.Interface and its implementations to support it and handle those events appropriately in places like - [Reflector](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/cache/reflector.go) + [Reflector](https://git.k8s.io/kubernetes/staging/src/k8s.io/client-go/tools/cache/reflector.go) However, this might turn out to be unnecessary optimization if apiserver will always keep up (which is possible in the new design). We will work out all necessary details at that point. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/auditing.md b/contributors/design-proposals/api-machinery/auditing.md index ff1b9092..b4def584 100644 --- a/contributors/design-proposals/auditing.md +++ b/contributors/design-proposals/api-machinery/auditing.md @@ -80,7 +80,7 @@ When implementing audit logging there are basically two options: 1. put a logging proxy in front of the apiserver 2. integrate audit logging into the apiserver itself -Both approaches have advantages and disadvanteges: +Both approaches have advantages and disadvantages: - **pro proxy**: + keeps complexity out of the apiserver + reuses existing solutions @@ -94,7 +94,7 @@ In the following, the second approach is described without a proxy. At which po 1. as one of the REST handlers (as in [#27087](https://github.com/kubernetes/kubernetes/pull/27087)), 2. as an admission controller. -The former approach (currently implemented) was picked over the other one, due to the need to be able to get information about both the user submitting the request and the impersonated user (and group), which is being overridden inside the [impersonation filter](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go). Additionally admission controller does not have access to the response and runs after authorization which will prevent logging failed authorization. All of that resulted in continuing the solution started in [#27087](https://github.com/kubernetes/kubernetes/pull/27087), which implements auditing as one of the REST handlers +The former approach (currently implemented) was picked over the other one, due to the need to be able to get information about both the user submitting the request and the impersonated user (and group), which is being overridden inside the [impersonation filter](https://git.k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go). Additionally admission controller does not have access to the response and runs after authorization which will prevent logging failed authorization. All of that resulted in continuing the solution started in [#27087](https://github.com/kubernetes/kubernetes/pull/27087), which implements auditing as one of the REST handlers after authentication, but before impersonation and authorization. ## Proposed Design @@ -374,7 +374,3 @@ Below are the possible future extensions to the auditing mechanism: * Define how filters work. They should enable dropping sensitive fields from the request/response/storage objects. * Allow setting a unique identifier which allows matching audit events across apiserver and federated servers. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/bulk_watch.md b/contributors/design-proposals/api-machinery/bulk_watch.md index ae3d543f..fc64d589 100644 --- a/contributors/design-proposals/bulk_watch.md +++ b/contributors/design-proposals/api-machinery/bulk_watch.md @@ -11,7 +11,7 @@ discussions see https://github.com/kubernetes/kubernetes/issues/40476. of the system, by significantly reducing amount of api calls coming from kubelets. As of now, to avoid situation that kubelet is watching all secrets/ configmaps/... in the system, it is not using watch for this purpose. Instead of -that, it is retrieving indidual objects, by sending individual GET requests. +that, it is retrieving individual objects, by sending individual GET requests. However, to enable automatic updates of mounted secrets/configmaps/..., Kubelet is sending those GET requests periodically. In large clusters, this is generating huge unnecessary load, as this load in principle should be @@ -64,7 +64,7 @@ API (by lists I mean e.g. `PodList` object) - the API has to be implemented also in aggregator so that bulk operations are supported also if different resource types are served by different apiservers -- clients has to be able to alter their watch subscribtions incrementally (it +- clients has to be able to alter their watch subscriptions incrementally (it may not be implemented in the initial version though, but has to be designed) @@ -76,7 +76,7 @@ call). Spanning multiple resources, resource types or conditions will be more and more important for large number of watches. As an example, federation will be adding watches for every type it federates. With that in mind, bypassing aggregation at the resource type level and going to aggregation over objects -with different resource types will allow us to more aggresively optimize in the +with different resource types will allow us to more aggressively optimize in the future (it doesn't mean you have to watch resources of different types in a single watch, but we would like to make it possible). @@ -124,8 +124,8 @@ websocket /apis/bulk.k8s.io/v1/bulkgetoperations?watch=1 handling LIST requests, where first client sends a filter definition over the channel and then server sends back the response, but we dropped this for now.* -*Note: We aso considered implementing the POST-based watch handler that doesn't -allow for altering subsriptions, which should be very simple once we have list +*Note: We also considered implementing the POST-based watch handler that doesn't +allow for altering subscriptions, which should be very simple once we have list implemented. But since websocket API is needed anyway, we also dropped it.* @@ -173,14 +173,14 @@ will be described together with dynamic watch description below. ### Dynamic watch As mentioned in the Proposal section, we will implement bulk watch that will -allow for dynamic subscribtion/unsubscribtion for (sets of) objects on top of +allow for dynamic subscription/unsubscription for (sets of) objects on top of websockets protocol. Note that we already support websockets in the regular Kubernetes API for watch requests (in addition to regular http requests), so for the purpose of bulk watch we will be extending websocket support. - The the high level, the propocol will look: + The the high level, the protocol will look: 1. client opens a new websocket connection to a bulk watch endpoint to the server via ghttp GET 1. this results in creating a single channel that is used only to handle @@ -232,7 +232,7 @@ type Response struct { With the above structure we can guarantee that we only send and receive objects of a single type over the channel. -We should also introduce some way of correleting responses with requests +We should also introduce some way of correlating responses with requests when a client is sending multiple of them at the same time. To achieve this we will add a `request identified` field to the `Request` that user can set and that will then be returned as part of `Response`. With this mechanism @@ -288,7 +288,7 @@ frameworks like reflector) that rely on two crucial watch invariants: 1. there is at most one watch event delivered for any resource version However, we have no guarantee that resource version series is shared between -diferent resource types (in fact in default GCE setup events are not sharing +different resource types (in fact in default GCE setup events are not sharing the same series as they are stored in a separate etcd instance). That said, to avoid introducing too many assumptions (that already aren't really met) we can't guarantee exactly the same. @@ -344,7 +344,7 @@ aggregator, which is crucial requirement here. NOTE: For watch requests, as an initial step we can consider implementing this API only in aggregator and simply start an individual watch for any subrequest. With http2 we shouldn't get rid of descriptors and it can be -enough as a prrof of concept. However, with such approach there will be +enough as a proof of concept. However, with such approach there will be difference between sending a given request to aggregator and apiserver so we need to implement it properly in apiserver before entering alpha anyway. This would just give us early results faster. @@ -359,7 +359,7 @@ single response to the user. The only non-trivial operation above is sending the request for a single resource type down the stack. In order to implement it, we will need to -slighly modify the interface of "Registry" in apiserver. The modification +slightly modify the interface of "Registry" in apiserver. The modification will have to allow passing both what we are passing now and BulkListOptions (in some format) (this may e.g. changing signature to accept BulkListOptions and translating ListOptions to BulkListOptions in the current code). @@ -433,7 +433,7 @@ do it in deterministic way. The crucial requirements are: 1. Whenever "list" request returns a list of objects and a resource version "rv", starting a watch from the returned "rv" will never drop any events. 2. For a given watch request (with resource version "rv"), the returned stream -of events is always the same (e.g. very slow laggin watch may not cause dropped +of events is always the same (e.g. very slow lagging watch may not cause dropped events). We can't really satisfy these conditions using the existing machinery. To solve @@ -468,7 +468,7 @@ no matter if we implement it or not) there are few selectors selecting the same object, in dynamic approach it will be send multiple times, once over each channel, here it would be send once) - we would have to introduce a dedicate "BulkWatchEvent" type to incorporate -resource type. This would make those two incompatible even at the ouput format. +resource type. This would make those two incompatible even at the output format. With all of those in mind, even though the implementation would be much much simpler (and could potentially be a first step and would probably solve the diff --git a/contributors/design-proposals/client-package-structure.md b/contributors/design-proposals/api-machinery/client-package-structure.md index 9cbd6aa8..1aa47b18 100644 --- a/contributors/design-proposals/client-package-structure.md +++ b/contributors/design-proposals/api-machinery/client-package-structure.md @@ -1,5 +1,3 @@ -<!-- BEGIN MUNGE: GENERATED_TOC --> - - [Client: layering and package structure](#client-layering-and-package-structure) - [Desired layers](#desired-layers) - [Transport](#transport) @@ -12,7 +10,6 @@ - [Package Structure](#package-structure) - [Client Guarantees (and testing)](#client-guarantees-and-testing) -<!-- END MUNGE: GENERATED_TOC --> # Client: layering and package structure @@ -310,7 +307,3 @@ that client will not have to change their code until they are deliberately upgrading their import. We probably will want to generate some sort of stub test with a clientset, to ensure that we don't change the interface. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/controller-ref.md b/contributors/design-proposals/api-machinery/controller-ref.md index c5514345..eee9629a 100644 --- a/contributors/design-proposals/controller-ref.md +++ b/contributors/design-proposals/api-machinery/controller-ref.md @@ -237,7 +237,7 @@ This section lists considerations specific to a given controller. * **ReplicaSet/ReplicationController** - * These controllers currenly only enable ControllerRef behavior when the + * These controllers currently only enable ControllerRef behavior when the Garbage Collector is enabled. When ControllerRef was first added to these controllers, the main purpose was to enable server-side cascading deletion via the Garbage Collector, so it made sense to gate it behind the same flag. @@ -415,7 +415,3 @@ Summary of significant revisions to this document: * Specify ControllerRef-related behavior changes upon upgrade/downgrade. * [Implementation](#implementation) * List all work to be done and mark items already completed as of this edit. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/csi-client-structure-proposal.md b/contributors/design-proposals/api-machinery/csi-client-structure-proposal.md index 4f46d32e..4f46d32e 100644 --- a/contributors/design-proposals/csi-client-structure-proposal.md +++ b/contributors/design-proposals/api-machinery/csi-client-structure-proposal.md diff --git a/contributors/design-proposals/csi-new-client-library-procedure.md b/contributors/design-proposals/api-machinery/csi-new-client-library-procedure.md index ee433856..dba0b0fb 100644 --- a/contributors/design-proposals/csi-new-client-library-procedure.md +++ b/contributors/design-proposals/api-machinery/csi-new-client-library-procedure.md @@ -24,7 +24,7 @@ Development would be based on a generated client using OpenAPI and [swagger-code ### Client Capabilities -* Bronze Requirements [](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/csi-new-client-library-procedure.md#client-capabilities) +* Bronze Requirements [](/contributors/design-proposals/api-machinery/csi-new-client-library-procedure.md#client-capabilities) * Support loading config from kube config file @@ -40,11 +40,11 @@ Development would be based on a generated client using OpenAPI and [swagger-code * Works from within the cluster environment. -* Silver Requirements [](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/csi-new-client-library-procedure.md#client-capabilities) +* Silver Requirements [](/contributors/design-proposals/api-machinery/csi-new-client-library-procedure.md#client-capabilities) * Support watch calls -* Gold Requirements [](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/csi-new-client-library-procedure.md#client-capabilities) +* Gold Requirements [](/contributors/design-proposals/api-machinery/csi-new-client-library-procedure.md#client-capabilities) * Support exec, attach, port-forward calls (these are not normally supported out of the box from [swagger-codegen](https://github.com/swagger-api/swagger-codegen)) @@ -54,11 +54,11 @@ Development would be based on a generated client using OpenAPI and [swagger-code ### Client Support Level -* Alpha [](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/csi-new-client-library-procedure.md#client-support-level) +* Alpha [](/contributors/design-proposals/api-machinery/csi-new-client-library-procedure.md#client-support-level) * Clients don’t even have to meet bronze requirements -* Beta [](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/csi-new-client-library-procedure.md#client-support-level) +* Beta [](/contributors/design-proposals/api-machinery/csi-new-client-library-procedure.md#client-support-level) * Client at least meets bronze standards @@ -68,7 +68,7 @@ Development would be based on a generated client using OpenAPI and [swagger-code * 2+ individual maintainers/owners of the repository -* Stable [](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/csi-new-client-library-procedure.md#client-support-level) +* Stable [](/contributors/design-proposals/api-machinery/csi-new-client-library-procedure.md#client-support-level) * Support level documented per-platform @@ -96,5 +96,5 @@ For each client language, we’ll make a client-[lang]-base and client-[lang] re # Support -These clients will be supported by the Kubernetes [API Machinery special interest group](https://github.com/kubernetes/community/tree/master/sig-api-machinery); however, individual owner(s) will be needed for each client language for them to be considered stable; the SIG won’t be able to handle the support load otherwise. If the generated clients prove as easy to maintain as we hope, then a few individuals may be able to own multiple clients. +These clients will be supported by the Kubernetes [API Machinery special interest group](/sig-api-machinery); however, individual owner(s) will be needed for each client language for them to be considered stable; the SIG won’t be able to handle the support load otherwise. If the generated clients prove as easy to maintain as we hope, then a few individuals may be able to own multiple clients. diff --git a/contributors/design-proposals/api-machinery/customresources-subresources.md b/contributors/design-proposals/api-machinery/customresources-subresources.md new file mode 100644 index 00000000..54c5f5bb --- /dev/null +++ b/contributors/design-proposals/api-machinery/customresources-subresources.md @@ -0,0 +1,203 @@ +# Subresources for CustomResources + +Authors: @nikhita, @sttts + +## Table of Contents + +1. [Abstract](#abstract) +2. [Goals](#goals) +3. [Non-Goals](#non-goals) +4. [Proposed Extension of CustomResourceDefinition](#proposed-extension-of-customresourcedefinition) + 1. [API Types](#api-types) + 2. [Feature Gate](#feature-gate) +5. [Semantics](#semantics) + 1. [Validation Behavior](#validation-behavior) + 1. [Status](#status) + 2. [Scale](#scale) + 2. [Status Behavior](#status-behavior) + 3. [Scale Behavior](#scale-behavior) + 1. [Status Replicas Behavior](#status-replicas-behavior) + 2. [Selector Behavior](#selector-behavior) +4. [Implementation Plan](#implementation-plan) +5. [Alternatives](#alternatives) + 1. [Scope](#scope) + +## Abstract + +[CustomResourceDefinitions](https://github.com/kubernetes/community/pull/524) (CRDs) were introduced in 1.7. The objects defined by CRDs are called CustomResources (CRs). Currently, we do not provide subresources for CRs. + +However, it is one of the [most requested features](https://github.com/kubernetes/kubernetes/issues/38113) and this proposal seeks to add `/status` and `/scale` subresources for CustomResources. + +## Goals + +1. Support status/spec split for CustomResources: + 1. Status changes are ignored on the main resource endpoint. + 2. Support a `/status` subresource HTTP path for status changes. + 3. `metadata.Generation` is increased only on spec changes. +2. Support a `/scale` subresource for CustomResources. +3. Maintain backward compatibility by allowing CRDs to opt-in to enable subresources. +4. If a CustomResource is already structured using spec/status, allow it to easily transition to use the `/status` and `/scale` endpoint. +5. Work seamlessly with [JSON Schema validation](https://github.com/kubernetes/community/pull/708). + +## Non-Goals + +1. Allow defining arbitrary subresources i.e. subresources except `/status` and `/scale`. + +## Proposed Extension of CustomResourceDefinition + +### API Types + +The addition of the following external types in `apiextensions.k8s.io/v1beta1` is proposed: + +```go +type CustomResourceDefinitionSpec struct { + ... + // SubResources describes the subresources for CustomResources + // This field is alpha-level and should only be sent to servers that enable + // subresources via the CurstomResourceSubResources feature gate. + // +optional + SubResources *CustomResourceSubResources `json:"subResources,omitempty"` +} + +// CustomResourceSubResources defines the status and scale subresources for CustomResources. +type CustomResourceSubResources struct { + // Status denotes the status subresource for CustomResources + Status *CustomResourceSubResourceStatus `json:"status,omitempty"` + // Scale denotes the scale subresource for CustomResources + Scale *CustomResourceSubResourceScale `json:"scale,omitempty"` +} + +// CustomResourceSubResourceStatus defines how to serve the HTTP path <CR Name>/status. +type CustomResourceSubResourceStatus struct { +} + +// CustomResourceSubResourceScale defines how to serve the HTTP path <CR name>/scale. +type CustomResourceSubResourceScale struct { + // required, e.g. “.spec.replicas”. Must be under `.spec`. + // Only JSON paths without the array notation are allowed. + SpecReplicasPath string `json:"specReplicasPath"` + // optional, e.g. “.status.replicas”. Must be under `.status`. + // Only JSON paths without the array notation are allowed. + StatusReplicasPath string `json:"statusReplicasPath,omitempty"` + // optional, e.g. “.spec.labelSelector”. Must be under `.spec`. + // Only JSON paths without the array notation are allowed. + LabelSelectorPath string `json:"labelSelectorPath,omitempty"` + // ScaleGroupVersion denotes the GroupVersion of the Scale + // object sent as the payload for /scale. It allows transition + // to future versions easily. + // Today only autoscaling/v1 is allowed. + ScaleGroupVersion schema.GroupVersion `json:"groupVersion"` +} +``` + +### Feature Gate + +The `SubResources` field in `CustomResourceDefinitionSpec` will be gated under the `CustomResourceSubResources` alpha feature gate. +If the gate is not open, the value of the new field within `CustomResourceDefinitionSpec` is dropped on creation and updates of CRDs. + +### Scale type + +The `Scale` object is the payload sent over the wire for `/scale`. The [polymorphic `Scale` type](https://github.com/kubernetes/kubernetes/pull/53743) i.e. `autoscaling/v1.Scale` is used for the `Scale` object. + +Since the GroupVersion of the `Scale` object is specified in `CustomResourceSubResourceScale`, transition to future versions (eg `autoscaling/v2.Scale`) can be done easily. + +Note: If `autoscaling/v1.Scale` is deprecated, then it would be deprecated here as well. + +## Semantics + +### Validation Behavior + +#### Status + +The status endpoint of a CustomResource receives a full CR object. Changes outside of the `.status` subpath are ignored. +For validation, the JSON Schema present in the CRD is validated only against the `.status` subpath. + +To validate only against the schema for the `.status` subpath, `oneOf` and `anyOf` constructs are not allowed within the root of the schema, but only under a properties sub-schema (with this restriction, we can project a schema to a sub-path). The following is forbidden in the CRD spec: + +```yaml +validation: + openAPIV3Schema: + oneOf: + ... +``` + +**Note**: The restriction for `oneOf` and `anyOf` allows us to write a projection function `ProjectJSONSchema(schema *JSONSchemaProps, path []string) (*JSONSchemaProps, error)` that can be used to apply a given schema for the whole object to only the sub-path `.status` or `.spec`. + +#### Scale + +Moreover, if the scale subresource is enabled: + +On update, we copy the values from the `Scale` object into the specified paths in the CustomResource, if the path is set (`StatusReplicasPath` and `LabelSelectorPath` are optional). +If `StatusReplicasPath` or `LabelSelectorPath` is not set, we validate that the value in `Scale` is also not specified and return an error otherwise. + +On `get` and on `update` (after copying the values into the CustomResource as described above), we verify that: + +- The value at the specified JSON Path `SpecReplicasPath` (e.g. `.spec.replicas`) is a non-negative integer value and is not empty. + +- The value at the optional JSON Path `StatusReplicasPath` (e.g. `.status.replicas`) is an integer value if it exists (i.e. this can be empty). + +- The value at the optional JSON Path `LabelSelectorPath` (e.g. `.spec.labelSelector`) is a valid label selector if it exists (i.e. this can be empty). + +**Note**: The values at the JSON Paths specified by `SpecReplicasPath`, `LabelSelectorPath` and `StatusReplicasPath` are also validated with the same rules when the whole object or, in case the `/status` subresource is enabled, the `.status` sub-object is updated. + +### Status Behavior + +If the `/status` subresource is enabled, the following behaviors change: + +- The main resource endpoint will ignore all changes in the status subpath. +(note: it will **not** reject requests which try to change the status, following the existing semantics of other resources). + +- The `.metadata.generation` field is updated if and only if the value at the `.spec` subpath changes. +Additionally, if the spec does not change, `.metadata.generation` is not updated. + +- The `/status` subresource receives a full resource object, but only considers the value at the `.status` subpath for the update. +The value at the `.metadata` subpath is **not** considered for update as decided in https://github.com/kubernetes/kubernetes/issues/45539. + +Both the status and the spec (and everything else if there is anything) of the object share the same key in the storage layer, i.e. the value at `.metadata.resourceVersion` is increased for any kind of change. There is no split of status and spec in the storage layer. + +The `/status` endpoint supports both `get` and `update` verbs. + +### Scale Behavior + +The number of CustomResources can be easily scaled up or down depending on the replicas field present in the `.spec` subpath. + +Only `ScaleSpec.Replicas` can be written. All other values are read-only and changes will be ignored. i.e. upon updating the scale subresource, two fields are modified: + +1. The replicas field is copied back from the `Scale` object to the main resource as specified by `SpecReplicasPath` in the CRD, e.g. `.spec.replicas = scale.Spec.Replicas`. + +2. The resource version is copied back from the `Scale` object to the main resource before writing to the storage: `.metadata.resourceVersion = scale.ResourceVersion`. +In other words, the scale and the CustomResource share the resource version used for optimistic concurrency. +Updates with outdated resource versions are rejected with a conflict error, read requests will return the resource version of the CustomResource. + +The `/scale` endpoint supports both `get` and `update` verbs. + +#### Status Replicas Behavior + +As only the `scale.Spec.Replicas` field is to be written to by the CR user, the user-provided controller (not any generic CRD controller) counts its children and then updates the controlled object by writing to the `/status` subresource, i.e. the `scale.Status.Replicas` field is read-only. + +#### Selector Behavior + +`CustomResourceSubResourceScale.LabelSelectorPath` is the label selector over CustomResources that should match the replicas count. +The value in the `Scale` object is one-to-one the value from the CustomResource if the label selector is non-empty. +Intentionally we do not default it to another value from the CustomResource (e.g. `.spec.template.metadata.labels`) as this turned out to cause trouble (e.g. in `kubectl apply`) and it is generally seen as a wrong approach with existing resources. + +## Implementation Plan + +The `/scale` and `/status` subresources are mostly distinct. It is proposed to do the implementation in two phases (the order does not matter much): + +1. `/status` subresource +2. `/scale` subresource + +## Alternatives + +### Scope + +In this proposal we opted for an opinionated concept of subresources i.e. we restrict the subresource spec to the two very specific subresources: `/status` and `/scale`. +We do not aim for a more generic subresource concept. In Kubernetes there are a number of other subresources like `/log`, `/exec`, `/bind`. But their semantics is much more special than `/status` and `/scale`. +Hence, we decided to leave those other subresources to the domain of User provided API Server (UAS) instead of inventing a more complex subresource concept for CustomResourceDefinitions. + +**Note**: The types do not make the addition of other subresources impossible in the future. + +We also restrict the JSON path for the status and the spec within the CustomResource. +We could make them definable by the user and the proposed types actually allow us to open this up in the future. +For the time being we decided to be opinionated as all status and spec subobjects in existing types live under `.status` and `.spec`. Keeping this pattern imposes consistency on user provided CustomResources as well. diff --git a/contributors/design-proposals/customresources-validation.md b/contributors/design-proposals/api-machinery/customresources-validation.md index ae907107..64448ee3 100644 --- a/contributors/design-proposals/customresources-validation.md +++ b/contributors/design-proposals/api-machinery/customresources-validation.md @@ -70,7 +70,7 @@ The schema is referenced in [`CustomResourceDefinitionSpec`](https://github.com/ The schema types follow those of the OpenAPI library, but we decided to define them independently for the API to have full control over the serialization and versioning. Hence, it is easy to convert our types into those used for validation or to integrate them into an OpenAPI spec later. -Reference http://json-schema.org is also used by OpenAPI. We propose this as there are implementations available in Go and with OpenAPI, we will also be able to serve OpenAPI specs for CustomResourceDefintions. +Reference http://json-schema.org is also used by OpenAPI. We propose this as there are implementations available in Go and with OpenAPI, we will also be able to serve OpenAPI specs for CustomResourceDefinitions. ```go // CustomResourceSpec describes how a user wants their resource to appear diff --git a/contributors/design-proposals/dynamic-admission-control-configuration.md b/contributors/design-proposals/api-machinery/dynamic-admission-control-configuration.md index 40d3f587..b1efb0db 100644 --- a/contributors/design-proposals/dynamic-admission-control-configuration.md +++ b/contributors/design-proposals/api-machinery/dynamic-admission-control-configuration.md @@ -2,13 +2,13 @@ ## Background The extensible admission control -[proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_extension.md) +[proposal](admission_control_extension.md) proposed making admission control extensible. In the proposal, the `initializer admission controller` and the `generic webhook admission controller` are the two controllers that set default initializers and external admission hooks for resources newly created. These two admission controllers are in the same binary as the apiserver. This -[section](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_extension.md#dynamic-configuration) +[section](admission_control_extension.md#dynamic-configuration) gave a preliminary design of the dynamic configuration of the list of the default admission controls. This document hashes out the implementation details. @@ -21,21 +21,21 @@ default admission controls. This document hashes out the implementation details. * As a fallback, admin can always restart an apiserver and guarantee it sees the latest config -* Do not block the entire cluster if the intializers/webhooks are not ready +* Do not block the entire cluster if the initializers/webhooks are not ready after registration. ## Specification We assume initializers could be "fail open". We need to update the extensible admission control -[proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_extension.md) +[proposal](admission_control_extension.md) if this is accepted. The schema is evolved from the prototype in [#132](https://github.com/kubernetes/community/pull/132). ```golang -// InitializerConfiguration describes the configuration of intializers. +// InitializerConfiguration describes the configuration of initializers. type InitializerConfiguration struct { metav1.TypeMeta @@ -43,9 +43,9 @@ type InitializerConfiguration struct { // Initializers is a list of resources and their default initializers // Order-sensitive. - // When merging multiple InitializerConfigurations, we sort the intializers + // When merging multiple InitializerConfigurations, we sort the initializers // from different InitializerConfigurations by the name of the - // InitializerConfigurations; the order of the intializers from the same + // InitializerConfigurations; the order of the initializers from the same // InitializerConfiguration is preserved. // +optional Initializers []Initializer `json:"initializers,omitempty" patchStrategy:"merge" patchMergeKey:"name"` @@ -63,7 +63,7 @@ type Initializer struct { Name string `json:"name"` // Rules describes what resources/subresources the initializer cares about. - // The intializer cares about an operation if it matches _any_ Rule. + // The initializer cares about an operation if it matches _any_ Rule. Rules []Rule `json:"rules,omitempty"` // FailurePolicy defines what happens if the responsible initializer controller @@ -106,7 +106,7 @@ type Rule struct { type FailurePolicyType string const ( - // Ignore means the initilizer is removed from the initializers list of an + // Ignore means the initializer is removed from the initializers list of an // object if the initializer is timed out. Ignore FailurePolicyType = "Ignore" // For 1.7, only "Ignore" is allowed. "Fail" will be allowed when the @@ -114,7 +114,7 @@ const ( Fail FailurePolicyType = "Fail" ) -// ExternalAdmissionHookConfiguration describes the configuration of intializers. +// ExternalAdmissionHookConfiguration describes the configuration of initializers. type ExternalAdmissionHookConfiguration struct { metav1.TypeMeta @@ -211,7 +211,7 @@ Notes: in the beta version. * We excluded `Retry` as a FailurePolicy, because we want to expose the - flakeness of an admission controller; and admission controllers like the quota + flakiness of an admission controller; and admission controllers like the quota controller are not idempotent. * There are multiple ways to compose `Rules []Rule` to achieve the same effect. @@ -248,7 +248,7 @@ See [Considered but REJECTED alternatives](#considered-but-rejected-alternatives ## Handling fail-open initializers -The original [proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_extension.md) assumed initializers always failed closed. It is dangerous since crashed +The original [proposal](admission_control_extension.md) assumed initializers always failed closed. It is dangerous since crashed initializers can block the whole cluster. We propose to allow initializers to fail open, and in 1.7, let all initializers fail open. @@ -263,7 +263,7 @@ the timed out initializer. If the apiserver crashes, then we fall back to a `read repair` mechanism. When handling a GET request, the apiserver checks the objectMeta.CreationTimestamp of -the object, if a global intializer timeout (e.g., 10 mins) has reached, the +the object, if a global initializer timeout (e.g., 10 mins) has reached, the apiserver removes the first initializer in the object. In the HA setup, apiserver needs to take the clock drift into account as well. @@ -282,7 +282,7 @@ See [Considered but REJECTED alternatives](#considered-but-rejected-alternatives 2. #1 will allow parallel initializers as well. 3. implement the fail closed initializers according to - [proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_extension.md#initializers). + [proposal](admission_control_extension.md#initializers). 4. more efficient check of AdmissionControlConfiguration changes. Currently we do periodic consistent read every second. @@ -365,7 +365,7 @@ initializers from objects' initializers list. The controller uses shared informers to track uninitialized objects. Every 30s, the controller * makes a snapshot of the uninitialized objects in the informers. -* indexes the objects by the name of the first initialilzer in the objectMeta.Initializers +* indexes the objects by the name of the first initializer in the objectMeta.Initializers * compares with the snapshot 30s ago, finds objects whose first initializers haven't changed * does a consistent read of AdmissionControllerConfiguration, finds which initializers are fail-open * spawns goroutines to send patches to remove fail-open initializers diff --git a/contributors/design-proposals/event_compression.md b/contributors/design-proposals/api-machinery/event_compression.md index 7a1cbb33..258adbb3 100644 --- a/contributors/design-proposals/event_compression.md +++ b/contributors/design-proposals/api-machinery/event_compression.md @@ -53,7 +53,7 @@ Each binary that generates events: * Maintains a historical record of previously generated events: * Implemented with ["Least Recently Used Cache"](https://github.com/golang/groupcache/blob/master/lru/lru.go) -in [`pkg/client/record/events_cache.go`](../../pkg/client/record/events_cache.go). +in [`pkg/client/record/events_cache.go`](https://git.k8s.io/kubernetes/staging/src/k8s.io/client-go/tools/record/events_cache.go). * Implemented behind an `EventCorrelator` that manages two subcomponents: `EventAggregator` and `EventLogger`. * The `EventCorrelator` observes all incoming events and lets each @@ -98,7 +98,7 @@ of time and generates tons of unique events, the previously generated events cache will not grow unchecked in memory. Instead, after 4096 unique events are generated, the oldest events are evicted from the cache. * When an event is generated, the previously generated events cache is checked -(see [`pkg/client/unversioned/record/event.go`](http://releases.k8s.io/HEAD/pkg/client/record/event.go)). +(see [`pkg/client/unversioned/record/event.go`](https://git.k8s.io/kubernetes/staging/src/k8s.io/client-go/tools/record/event.go)). * If the key for the new event matches the key for a previously generated event (meaning all of the above fields match between the new event and some previously generated event), then the event is considered to be a duplicate and @@ -162,8 +162,3 @@ compressing multiple recurring events in to a single event. single event to optimize etcd storage. * PR [#4444](http://pr.k8s.io/4444): Switch events history to use LRU cache instead of map. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/extending-api.md b/contributors/design-proposals/api-machinery/extending-api.md index c9aeb449..f5e2de6a 100644 --- a/contributors/design-proposals/extending-api.md +++ b/contributors/design-proposals/api-machinery/extending-api.md @@ -196,8 +196,3 @@ Thus, listing a third-party resource can be achieved by listing the directory: ``` ${standard-k8s-prefix}/third-party-resources/${third-party-resource-namespace}/${third-party-resource-name}/${resource-namespace}/ ``` - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/garbage-collection.md b/contributors/design-proposals/api-machinery/garbage-collection.md index 0bf42e5d..4ee1cabc 100644 --- a/contributors/design-proposals/garbage-collection.md +++ b/contributors/design-proposals/api-machinery/garbage-collection.md @@ -349,9 +349,3 @@ In case the garbage collector is mistakenly deleting objects, we should provide * Before an object is deleted from the registry, the API server clears fields like DeletionTimestamp, then creates the object in /archive and sets a TTL. * Add a `kubectl restore` command, which takes a resource/name pair as input, creates the object with the spec stored in the /archive, and deletes the archived object. - - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/metadata-policy.md b/contributors/design-proposals/api-machinery/metadata-policy.md index f924bc65..68f144a1 100644 --- a/contributors/design-proposals/metadata-policy.md +++ b/contributors/design-proposals/api-machinery/metadata-policy.md @@ -130,8 +130,3 @@ single scheduler, as opposed to choosing a scheduler, a desire mentioned in `MetadataPolicy` could be used. Issue #17324 proposes to create a generalized API for matching "claims" to "service classes"; matching a pod to a scheduler would be one use for such an API. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/protobuf.md b/contributors/design-proposals/api-machinery/protobuf.md index 6741bbab..455cc955 100644 --- a/contributors/design-proposals/protobuf.md +++ b/contributors/design-proposals/api-machinery/protobuf.md @@ -301,7 +301,7 @@ that the returned value is not in the known type. We add the `contentEncoding` field here to preserve room for future optimizations like encryption-at-rest or compression of the nested content. Clients should error when receiving an encoding they do not support. -Negotioting encoding is not defined here, but introducing new encodings +Negotiating encoding is not defined here, but introducing new encodings is similar to introducing a schema change or new API version. A client should use the `kind` and `apiVersion` fields to identify the @@ -473,8 +473,3 @@ The generated protobuf will be checked with a verify script before merging. ## Open Questions * Is supporting stored protobuf files on disk in the kubectl client worth it? - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/server-get.md b/contributors/design-proposals/api-machinery/server-get.md index fb508a3c..ef68e125 100644 --- a/contributors/design-proposals/server-get.md +++ b/contributors/design-proposals/api-machinery/server-get.md @@ -18,7 +18,7 @@ schema). `get` supports a `wide` mode that includes additional columns. Users ca flag. Headers corresponding to the columns are optionally displayed. `kubectl describe` shows a textual representation of individual objects that describes individual fields as subsequent -lines and uses indendation and nested tables to convey deeper structure on the resource (such as events for a pod or +lines and uses indentation and nested tables to convey deeper structure on the resource (such as events for a pod or each container). It sometimes retrieves related objects like events, pods for a replication controller, or autoscalers for a deployment. It supports no significant flags. @@ -177,7 +177,3 @@ fall back to client side functions. Server side code would reuse the existing display functions but replace TabWriter with either a structured writer or the tabular form. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/synchronous-garbage-collection.md b/contributors/design-proposals/api-machinery/synchronous-garbage-collection.md index 6f2a9be5..df4595c9 100644 --- a/contributors/design-proposals/synchronous-garbage-collection.md +++ b/contributors/design-proposals/api-machinery/synchronous-garbage-collection.md @@ -1,5 +1,4 @@ **Table of Contents** -<!-- BEGIN MUNGE: GENERATED_TOC --> - [Overview](#overview) - [API Design](#api-design) @@ -14,7 +13,6 @@ - [Unhandled cases](#unhandled-cases) - [Implications to existing clients](#implications-to-existing-clients) -<!-- END MUNGE: GENERATED_TOC --> # Overview @@ -124,7 +122,7 @@ In addition, if an object popped from `dirtyQueue` is marked as "GC in progress" * To avoid racing with another controller, it requeues the object if `observedGeneration < Generation`. This is best-effort, see [unhandled cases](#unhandled-cases). * Checks if the object has dependents * If not, send a PUT request to remove the `GCFinalizer`; - * If so, then add all dependents to the `dirtryQueue`; we need bookkeeping to avoid adding the dependents repeatedly if the owner gets in the `synchronousGC queue` multiple times. + * If so, then add all dependents to the `dirtyQueue`; we need bookkeeping to avoid adding the dependents repeatedly if the owner gets in the `synchronousGC queue` multiple times. ## Controllers @@ -169,7 +167,3 @@ To make the new kubectl compatible with the 1.4 and earlier masters, kubectl nee 1.4 `kubectl delete rc/rs` uses `DeleteOptions.OrphanDependents=true`, which is going to be converted to `DeletePropagationBackground` (see [API Design](#api-changes)) by a 1.5 master, so its behavior keeps the same. Pre 1.4 `kubectl delete` uses `DeleteOptions.OrphanDependents=nil`, so does the 1.4 `kubectl delete` for resources other than rc and rs. The option is going to be converted to `DeletePropagationDefault` (see [API Design](#api-changes)) by a 1.5 master, so these commands behave the same as when working with a 1.4 master. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/thirdpartyresources.md b/contributors/design-proposals/api-machinery/thirdpartyresources.md index 8f651d0d..05dfff76 100644 --- a/contributors/design-proposals/thirdpartyresources.md +++ b/contributors/design-proposals/api-machinery/thirdpartyresources.md @@ -13,7 +13,7 @@ prevent future challenges in upgrading. 1. Ensure ThirdPartyResource APIs operate consistently with first party Kubernetes APIs. 2. Enable ThirdPartyResources to specify how they will appear in API -discovery to be consistent with other resources and avoid naming confilcts +discovery to be consistent with other resources and avoid naming conflicts 3. Move TPR into their own API group to allow the extensions group to be [removed](https://github.com/kubernetes/kubernetes/issues/43214) 4. Support cluster scoped TPR resources diff --git a/contributors/design-proposals/apiserver-build-in-admission-plugins.md b/contributors/design-proposals/apiserver-build-in-admission-plugins.md deleted file mode 100644 index 10708b1f..00000000 --- a/contributors/design-proposals/apiserver-build-in-admission-plugins.md +++ /dev/null @@ -1,58 +0,0 @@ -# Build some Admission Controllers into the Generic API server library - -**Related PR:** - -| Topic | Link | -| ----- | ---- | -| Admission Control | https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control.md | - -## Introduction - -An admission controller is a piece of code that intercepts requests to the Kubernetes API - think a middleware. -The API server lets you have a whole chain of them. Each is run in sequence before a request is accepted -into the cluster. If any of the plugins in the sequence rejects the request, the entire request is rejected -immediately and an error is returned to the user. - -Many features in Kubernetes require an admission control plugin to be enabled in order to properly support the feature. -In fact in the [documentation](https://kubernetes.io/docs/admin/admission-controllers/#is-there-a-recommended-set-of-plug-ins-to-use) you will find -a recommended set of them to use. - -At the moment admission controllers are implemented as plugins and they have to be compiled into the -final binary in order to be used at a later time. Some even require an access to cache, an authorizer etc. -This is where an admission plugin initializer kicks in. An admission plugin initializer is used to pass additional -configuration and runtime references to a cache, a client and an authorizer. - -To streamline the process of adding new plugins especially for aggregated API servers we would like to build some plugins -into the generic API server library and provide a plugin initializer. While anyone can author and register one, having a known set of -provided references let's people focus on what they need their admission plugin to do instead of paying attention to wiring. - -## Implementation - -The first step would involve creating a "standard" plugin initializer that would be part of the -generic API server. It would use kubeconfig to populate -[external clients](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubeapiserver/admission/initializer.go#L29) -and [external informers](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubeapiserver/admission/initializer.go#L35). -By default for servers that would be run on the kubernetes cluster in-cluster config would be used. -The standard initializer would also provide a client config for connecting to the core kube-apiserver. -Some API servers might be started as static pods, which don't have in-cluster configs. -In that case the config could be easily populated form the file. - -The second step would be to move some plugins from [admission pkg](https://github.com/kubernetes/kubernetes/tree/master/plugin/pkg/admission) -to the generic API server library. Some admission plugins are used to ensure consistent user expectations. -These plugins should be moved. One example is the Namespace Lifecycle plugin which prevents users -from creating resources in non-existent namespaces. - -*Note*: -For loading in-cluster configuration [visit](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/examples/in-cluster/main.go#L30) - For loading the configuration directly from a file [visit](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/examples/out-of-cluster/main.go) - -## How to add an admission plugin ? - At this point adding an admission plugin is very simple and boils down to performing the -following series of steps: - 1. Write an admission plugin - 2. Register the plugin - 3. Reference the plugin in the admission chain - -**TODO**(p0lyn0mial): There is also a [sample apiserver](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/sample-apiserver/main.go) to demonstrate the usage of the generic API library. -After implementation sample could would be placed there - copy & paste it here and include a reference. - diff --git a/contributors/design-proposals/templates.md b/contributors/design-proposals/apps/OBSOLETE_templates.md index 50712932..a1213830 100644 --- a/contributors/design-proposals/templates.md +++ b/contributors/design-proposals/apps/OBSOLETE_templates.md @@ -562,8 +562,3 @@ Openshift handles template processing via a server endpoint which consumes a tem produced by processing the template. It is also possible to handle the entire template processing flow via the client, but this was deemed undesirable as it would force each client tool to reimplement template processing (e.g. the standard CLI tool, an eclipse plugin, a plugin for a CI system like Jenkins, etc). The assumption in this proposal is that server side template processing is the preferred implementation approach for this reason. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/apps/OWNERS b/contributors/design-proposals/apps/OWNERS new file mode 100644 index 00000000..12723930 --- /dev/null +++ b/contributors/design-proposals/apps/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-apps-leads +approvers: + - sig-apps-leads +labels: + - sig/apps diff --git a/contributors/design-proposals/configmap.md b/contributors/design-proposals/apps/configmap.md index 658ac73b..55571448 100644 --- a/contributors/design-proposals/configmap.md +++ b/contributors/design-proposals/apps/configmap.md @@ -294,7 +294,3 @@ spec: In the future, we may add the ability to specify an init-container that can watch the volume contents for updates and respond to changes when they occur. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/controller_history.md b/contributors/design-proposals/apps/controller_history.md index fbf89b70..5c650510 100644 --- a/contributors/design-proposals/controller_history.md +++ b/contributors/design-proposals/apps/controller_history.md @@ -427,7 +427,7 @@ its feasibility, we construct such a scheme here. However, this proposal does not mandate its use. Given a hash function with output size `HashSize` defined -as `func H(s srtring) [HashSize] byte`, in order to resolve collisions we +as `func H(s string) [HashSize] byte`, in order to resolve collisions we define a new function `func H'(s string, n int) [HashSize]byte` where `H'` returns the result of invoking `H` on the concatenation of `s` with the string value of `n`. We define a third function diff --git a/contributors/design-proposals/cronjob.md b/contributors/design-proposals/apps/cronjob.md index 7e41f9d0..41ae7a47 100644 --- a/contributors/design-proposals/cronjob.md +++ b/contributors/design-proposals/apps/cronjob.md @@ -207,7 +207,7 @@ but old ones still satisfy the schedule and are not re-run just because the temp If you delete and replace a CronJob with one of the same name, it will: - not use any old Status.Active, and not consider any existing running or terminated jobs from the previous - CronJob (with a different UID) at all when determining coflicts, what needs to be started, etc. + CronJob (with a different UID) at all when determining conflicts, what needs to be started, etc. - If there is an existing Job with the same time-based hash in its name (see below), then new instances of that job will not be able to be created. So, delete it if you want to re-run. with the same name as conflicts. @@ -322,6 +322,10 @@ by two instances (replicated or restarting) of the controller process. We chose to use the hashed-date suffix approach. +## Manually triggering CronJobs + +A user may wish to manually trigger a CronJob for some reason (see [#47538](http://issues.k8s.io/47538)), such as testing it prior to its scheduled time. This could be made possible via an `/instantiate` subresource in the API, which when POSTed to would immediately spawn a Job from the JobSpec contained within the CronJob. + ## Future evolution Below are the possible future extensions to the Job controller: @@ -329,7 +333,3 @@ Below are the possible future extensions to the Job controller: happening in [#18827](https://issues.k8s.io/18827). * Be able to specify more general template in `.spec` field, to create arbitrary types of resources. This relates to the work happening in [#18215](https://issues.k8s.io/18215). - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/daemon.md b/contributors/design-proposals/apps/daemon.md index e6e74f77..bd2f281e 100644 --- a/contributors/design-proposals/daemon.md +++ b/contributors/design-proposals/apps/daemon.md @@ -66,8 +66,7 @@ nodes, preempting other pods if necessary. "kubernetes.io/created-by: \<json API object reference\>" ``` - YAML example: -``` - YAML +```yaml apiVersion: extensions/v1beta1 kind: DaemonSet metadata: @@ -202,7 +201,3 @@ restartPolicy set to Always. - Should work similarly to [Deployment](http://issues.k8s.io/1743). - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/daemonset-update.md b/contributors/design-proposals/apps/daemonset-update.md index b50beb86..aea7e244 100644 --- a/contributors/design-proposals/daemonset-update.md +++ b/contributors/design-proposals/apps/daemonset-update.md @@ -132,7 +132,7 @@ type DaemonSetSpec struct { // DaemonSetStatus represents the current status of a daemon set. type DaemonSetStatus struct { - // Note: Existing fields, including CurrentNumberScheduled, NumberMissscheduled, + // Note: Existing fields, including CurrentNumberScheduled, NumberMisscheduled, // DesiredNumberScheduled, NumberReady, and ObservedGeneration are omitted in // this proposal. @@ -202,7 +202,7 @@ For each pending DaemonSet updates, it will: - The history will be labeled with `DefaultDaemonSetUniqueLabelKey`. - DaemonSet controller will add a ControllerRef in the history `.ownerReferences`. - - Current history should have the largest `.revision` number amonst all + - Current history should have the largest `.revision` number amongst all existing history. Update `.revision` if it's not (e.g. after a rollback.) - If more than one current history is found, remove duplicates and relabel their pods' `DefaultDaemonSetUniqueLabelKey`. @@ -250,7 +250,7 @@ In DaemonSet strategy (pkg/registry/extensions/daemonset/strategy.go#PrepareForU increase DaemonSet's `.spec.templateGeneration` by 1 if any changes is made to DaemonSet's `.spec.template`. -This was originally implmeneted in 1.6, and kept in 1.7 for backward compatibility. +This was originally implemented in 1.6, and kept in 1.7 for backward compatibility. ### kubectl diff --git a/contributors/design-proposals/deploy.md b/contributors/design-proposals/apps/deploy.md index 8185757d..0165dd9d 100644 --- a/contributors/design-proposals/deploy.md +++ b/contributors/design-proposals/apps/deploy.md @@ -1,5 +1,3 @@ -<!-- BEGIN MUNGE: GENERATED_TOC --> - - [Deploy through CLI](#deploy-through-cli) - [Motivation](#motivation) - [Requirements](#requirements) @@ -16,13 +14,12 @@ - [Pause Deployments](#pause-deployments) - [Perm-failed Deployments](#perm-failed-deployments) -<!-- END MUNGE: GENERATED_TOC --> # Deploy through CLI ## Motivation -Users can use [Deployments](../user-guide/deployments.md) or [`kubectl rolling-update`](../user-guide/kubectl/kubectl_rolling-update.md) to deploy in their Kubernetes clusters. A Deployment provides declarative update for Pods and ReplicationControllers, whereas `rolling-update` allows the users to update their earlier deployment without worrying about schemas and configurations. Users need a way that's similar to `rolling-update` to manage their Deployments more easily. +Users can use [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) or [`kubectl rolling-update`](https://kubernetes.io/docs/tasks/run-application/rolling-update-replication-controller/) to deploy in their Kubernetes clusters. A Deployment provides declarative update for Pods and ReplicationControllers, whereas `rolling-update` allows the users to update their earlier deployment without worrying about schemas and configurations. Users need a way that's similar to `rolling-update` to manage their Deployments more easily. `rolling-update` expects ReplicationController as the only resource type it deals with. It's not trivial to support exactly the same behavior with Deployment, which requires: - Print out scaling up/down events. @@ -141,7 +138,3 @@ Users sometimes need to temporarily disable a deployment. See issue [#14516](htt ### Perm-failed Deployments The deployment could be marked as "permanently failed" for a given spec hash so that the system won't continue thrashing on a doomed deployment. The users can retry a failed deployment with `kubectl rollout retry`. See issue [#14519](https://github.com/kubernetes/kubernetes/issues/14519). - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/deployment.md b/contributors/design-proposals/apps/deployment.md index 5256ffeb..16c35dfe 100644 --- a/contributors/design-proposals/deployment.md +++ b/contributors/design-proposals/apps/deployment.md @@ -257,8 +257,3 @@ Apart from the above, we want to add support for the following: - https://github.com/kubernetes/kubernetes/issues/1743 has most of the discussion that resulted in this proposal. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/indexed-job.md b/contributors/design-proposals/apps/indexed-job.md index bc2860b9..bfad092e 100644 --- a/contributors/design-proposals/indexed-job.md +++ b/contributors/design-proposals/apps/indexed-job.md @@ -761,7 +761,7 @@ spec: spec: containers: - name: c - image: gcr.io/google_containers/busybox + image: k8s.gcr.io/busybox command: - 'sh' - '-c' @@ -894,7 +894,3 @@ is verbose. For StatefulSet, this is less of a problem. This differs from StatefulSet in that StatefulSet uses names and not indexes. StatefulSet is intended to support ones to tens of things. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/job.md b/contributors/design-proposals/apps/job.md index 160b38dd..2094b125 100644 --- a/contributors/design-proposals/job.md +++ b/contributors/design-proposals/apps/job.md @@ -18,6 +18,7 @@ Several existing issues and PRs were already created regarding that particular s 1. Be able to get the job status. 1. Be able to specify the number of instances performing a job at any one time. 1. Be able to specify the number of successfully finished instances required to finish a job. +1. Be able to specify a backoff policy, when job is continuously failing. ## Motivation @@ -26,6 +27,41 @@ Jobs are needed for executing multi-pod computation to completion; a good exampl here would be the ability to implement any type of batch oriented tasks. +## Backoff policy and failed pod limit + +By design, Jobs do not have any notion of failure, other than a pod's `restartPolicy` +which is mistakenly taken as Job's restart policy ([#30243](https://github.com/kubernetes/kubernetes/issues/30243), +[#[43964](https://github.com/kubernetes/kubernetes/issues/43964)]). There are +situation where one wants to fail a Job after some amount of retries over a certain +period of time, due to a logical error in configuration etc. To do so we are going +to introduce the following fields, which will control the backoff policy: a number of +retries and an initial time of retry. The two fields will allow fine-grained control +over the backoff policy. Each of the two fields will use a default value if none +is provided, `BackoffLimit` is set by default to 6 and `BackoffSeconds` to 10s. +This will result in the following retry sequence: 10s, 20s, 40s, 1m20s, 2m40s, +5m20s. After which the job will be considered failed. + +Additionally, to help debug the issue with a Job, and limit the impact of having +too many failed pods left around (as mentioned in [#30243](https://github.com/kubernetes/kubernetes/issues/30243)), +we are going to introduce a field which will allow specifying the maximum number +of failed pods to keep around. This number will also take effect if none of the +limits described above are set. By default it will take value of 1, to allow debugging +job issues, but not to flood the cluster with too many failed jobs and their +accompanying pods. + +All of the above fields will be optional and will apply when `restartPolicy` is +set to `Never` on a `PodTemplate`. With restart policy `OnFailure` only `BackoffLimit` +applies. The reason for that is that failed pods are already restarted by the +kubelet with an [exponential backoff](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy). +Additionally, failures are counted differently depending on `restartPolicy` +setting. For `Never` we count actual pod failures (reflected in `.status.failed` +field). With `OnFailure`, we take an approximate value of pod restarts (as reported +in `.status.containerStatuses[*].restartCount`). +When `.spec.parallelism` is set to a value higher than 1, the failures are an +overall number (as coming from `.status.failed`) because the controller does not +hold information about failures coming from separate pods. + + ## Implementation Job controller is similar to replication controller in that they manage pods. @@ -73,14 +109,31 @@ type JobSpec struct { // run at any given time. The actual number of pods running in steady state will // be less than this number when ((.spec.completions - .status.successful) < .spec.parallelism), // i.e. when the work left to do is less than max parallelism. - Parallelism *int + Parallelism *int32 // Completions specifies the desired number of successfully finished pods the // job should be run with. Defaults to 1. - Completions *int + Completions *int32 + + // Optional duration in seconds relative to the startTime that the job may be active + // before the system tries to terminate it; value must be a positive integer. + // It applies to overall job run time, no matter of the value of completions + // or parallelism parameters. + ActiveDeadlineSeconds *int64 + + // Optional number of retries before marking this job failed. + // Defaults to 6. + BackoffLimit *int32 + + // Optional time (in seconds) specifying how long the initial backoff will last. + // Defaults to 10s. + BackoffSeconds *int64 + + // Optional number of failed pods to retain. + FailedPodsLimit *int32 // Selector is a label query over pods running a job. - Selector map[string]string + Selector LabelSelector // Template is the object that describes the pod that will be created when // executing a job. @@ -107,14 +160,14 @@ type JobStatus struct { CompletionTime unversioned.Time // Active is the number of actively running pods. - Active int + Active int32 - // Successful is the number of pods successfully completed their job. - Successful int + // Succeeded is the number of pods successfully completed their job. + Succeeded int32 - // Unsuccessful is the number of pods failures, this applies only to jobs + // Failed is the number of pods failures, this applies only to jobs // created with RestartPolicyNever, otherwise this value will always be 0. - Unsuccessful int + Failed int32 } type JobConditionType string @@ -153,7 +206,3 @@ Below are the possible future extensions to the Job controller: by providing pointers to Pods in the JobStatus ([see comment](https://github.com/kubernetes/kubernetes/pull/11746/files#r37142628)). * help users avoid non-unique label selectors ([see this proposal](../../docs/design/selector-generation.md)) - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/selector-generation.md b/contributors/design-proposals/apps/selector-generation.md index 9b4b51fa..e0b3bf22 100644 --- a/contributors/design-proposals/selector-generation.md +++ b/contributors/design-proposals/apps/selector-generation.md @@ -174,7 +174,3 @@ Docs will be edited to show examples without a `job.spec.selector`. We probably want as much as possible the same behavior for Job and ReplicationController. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/stateful-apps.md b/contributors/design-proposals/apps/stateful-apps.md index ee9f9cd5..286f57da 100644 --- a/contributors/design-proposals/stateful-apps.md +++ b/contributors/design-proposals/apps/stateful-apps.md @@ -181,7 +181,7 @@ status, back-off (like a scheduler or replication controller), and try again lat by a StatefulSet controller must have a set of labels that match the selector, support orphaning, and have a controller back reference annotation identifying the owning StatefulSet by name and UID. -When a StatefulSet is scaled down, the pod for the removed indentity should be deleted. It is less clear what the +When a StatefulSet is scaled down, the pod for the removed identity should be deleted. It is less clear what the controller should do to supporting resources. If every pod requires a PV, and a user accidentally scales up to N=200 and then back down to N=3, leaving 197 PVs lying around may be undesirable (potential for abuse). On the other hand, a cluster of 5 that is accidentally scaled down to 3 might irreparably destroy @@ -346,7 +346,7 @@ Requested features: * Jobs can be used to perform a run-once initialization of the cluster * Init containers can be used to prime PVs and config with the identity of the pod. -* Templates and how fields are overriden in the resulting object should have broad alignment +* Templates and how fields are overridden in the resulting object should have broad alignment * DaemonSet defines the core model for how new controllers sit alongside replication controller and how upgrades can be implemented outside of Deployment objects. @@ -355,9 +355,3 @@ Requested features: StatefulSets were formerly known as PetSets and were renamed to be less "cutesy" and more descriptive as a prerequisite to moving to beta. No animals were harmed in the making of this proposal. - - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/statefulset-update.md b/contributors/design-proposals/apps/statefulset-update.md index c8801861..27d3000f 100644 --- a/contributors/design-proposals/statefulset-update.md +++ b/contributors/design-proposals/apps/statefulset-update.md @@ -435,7 +435,7 @@ constraints. `PartitionStatefulSetStrategyType`, the API Server should fail validation if any of the following conditions are true. 1. `.Spec.UpdateStrategy.Partition` is nil. - 1. `.Spec.UpdateStrategy.Parition` is not nil, and + 1. `.Spec.UpdateStrategy.Partition` is not nil, and `.Spec.UpdateStrategy.Partition.Ordinal` not in the sequence `(0,.Spec.Replicas)`. 1. The API Server will fail validation on any update to a StatefulSetStatus @@ -443,7 +443,7 @@ object if any of the following conditions are true. 1. `.Status.Replicas` is negative. 1. `.Status.ReadyReplicas` is negative or greater than `.Status.Replicas`. 1. `.Status.CurrentReplicas` is negative or greater than `.Status.Replicas`. - 1. `.Stauts.UpdateReplicas` is negative or greater than `.Status.Replicas`. + 1. `.Status.UpdateReplicas` is negative or greater than `.Status.Replicas`. ## Kubectl Kubectl will use the `rollout` command to control and provide the status of @@ -479,7 +479,7 @@ spec: spec: containers: - name: nginx - image: gcr.io/google_containers/nginx-slim:0.8 + image: k8s.gcr.io/nginx-slim:0.8 ports: - containerPort: 80 name: web @@ -530,7 +530,7 @@ spec: type: RollingUpdate containers: - name: nginx - image: gcr.io/google_containers/nginx-slim:0.9 + image: k8s.gcr.io/nginx-slim:0.9 ports: - containerPort: 80 name: web @@ -582,7 +582,7 @@ spec: ordinal: 2 containers: - name: nginx - image: gcr.io/google_containers/nginx-slim:0.9 + image: k8s.gcr.io/nginx-slim:0.9 ports: - containerPort: 80 name: web @@ -626,7 +626,7 @@ spec: ordinal: 3 containers: - name: nginx - image: gcr.io/google_containers/nginx-slim:0.9 + image: k8s.gcr.io/nginx-slim:0.9 ports: - containerPort: 80 name: web @@ -670,7 +670,7 @@ spec: ordinal: 2 containers: - name: nginx - image: gcr.io/google_containers/nginx-slim:0.9 + image: k8s.gcr.io/nginx-slim:0.9 ports: - containerPort: 80 name: web @@ -714,7 +714,7 @@ spec: ordinal: 1 containers: - name: nginx - image: gcr.io/google_containers/nginx-slim:0.9 + image: k8s.gcr.io/nginx-slim:0.9 ports: - containerPort: 80 name: web diff --git a/contributors/design-proposals/architecture.dia b/contributors/design-proposals/architecture.dia Binary files differdeleted file mode 100644 index 5c87409f..00000000 --- a/contributors/design-proposals/architecture.dia +++ /dev/null diff --git a/contributors/design-proposals/architecture.png b/contributors/design-proposals/architecture.png Binary files differdeleted file mode 100644 index 0ee8bceb..00000000 --- a/contributors/design-proposals/architecture.png +++ /dev/null diff --git a/contributors/design-proposals/architecture.svg b/contributors/design-proposals/architecture.svg deleted file mode 100644 index d6b6aab0..00000000 --- a/contributors/design-proposals/architecture.svg +++ /dev/null @@ -1,1943 +0,0 @@ -<?xml version="1.0" encoding="UTF-8" standalone="no"?> -<svg - xmlns:dc="http://purl.org/dc/elements/1.1/" - xmlns:cc="http://creativecommons.org/ns#" - xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" - xmlns:svg="http://www.w3.org/2000/svg" - xmlns="http://www.w3.org/2000/svg" - xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" - xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" - width="68cm" - height="56cm" - viewBox="-55 -75 1348 1117" - id="svg2" - version="1.1" - inkscape:version="0.91 r13725" - sodipodi:docname="architecture.svg" - inkscape:export-filename="D:\Work\PaaS\V1R2\Kubernetes\Src\kubernetes\docs\design\architecture.png" - inkscape:export-xdpi="90" - inkscape:export-ydpi="90"> - <metadata - id="metadata738"> - <rdf:RDF> - <cc:Work - rdf:about=""> - <dc:format>image/svg+xml</dc:format> - <dc:type - rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> - </cc:Work> - </rdf:RDF> - </metadata> - <defs - id="defs736" /> - <sodipodi:namedview - pagecolor="#ffffff" - bordercolor="#666666" - borderopacity="1" - objecttolerance="10" - gridtolerance="10" - guidetolerance="10" - inkscape:pageopacity="0" - inkscape:pageshadow="2" - inkscape:window-width="1680" - inkscape:window-height="988" - id="namedview734" - showgrid="false" - inkscape:zoom="0.33640324" - inkscape:cx="1204.7244" - inkscape:cy="992.12598" - inkscape:window-x="-8" - inkscape:window-y="-8" - inkscape:window-maximized="1" - inkscape:current-layer="svg2" /> - <g - id="g4"> - <rect - style="fill: #ffffff" - x="662" - y="192" - width="630" - height="381" - id="rect6" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="662" - y="192" - width="630" - height="381" - id="rect8" /> - </g> - <g - id="g10"> - <rect - style="fill: #ffffff" - x="688" - y="321" - width="580" - height="227" - id="rect12" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="688" - y="321" - width="580" - height="227" - id="rect14" /> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="687" - y="224" - id="text16"> - <tspan - x="687" - y="224" - id="tspan18">Node</tspan> - </text> - <g - id="g20"> - <rect - style="fill: #ffffff" - x="723.2" - y="235" - width="69.6" - height="38" - id="rect22" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="723.2" - y="235" - width="69.6" - height="38" - id="rect24" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="758" - y="258.8" - id="text26"> - <tspan - x="758" - y="258.8" - id="tspan28">kubelet</tspan> - </text> - </g> - <g - id="g30"> - <rect - style="fill: #ffffff" - x="720.2" - y="368.1" - width="148" - height="133" - id="rect32" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="720.2" - y="368.1" - width="148" - height="133" - id="rect34" /> - </g> - <g - id="g36"> - <rect - style="fill: #ffffff" - x="760.55" - y="438.1" - width="89.3" - height="38" - id="rect38" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="760.55" - y="438.1" - width="89.3" - height="38" - id="rect40" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="805.2" - y="461.9" - id="text42"> - <tspan - x="805.2" - y="461.9" - id="tspan44">container</tspan> - </text> - </g> - <g - id="g46"> - <rect - style="fill: #ffffff" - x="749.8" - y="428.2" - width="89.3" - height="38" - id="rect48" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="749.8" - y="428.2" - width="89.3" - height="38" - id="rect50" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="794.45" - y="452" - id="text52"> - <tspan - x="794.45" - y="452" - id="tspan54">container</tspan> - </text> - </g> - <g - id="g56"> - <rect - style="fill: #ffffff" - x="739.4" - y="418.3" - width="89.3" - height="38" - id="rect58" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="739.4" - y="418.3" - width="89.3" - height="38" - id="rect60" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="784.05" - y="442.1" - id="text62"> - <tspan - x="784.05" - y="442.1" - id="tspan64">cAdvisor</tspan> - </text> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="794.2" - y="434.6" - id="text66"> - <tspan - x="794.2" - y="434.6" - id="tspan68" /> - </text> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="742.2" - y="394.6" - id="text70"> - <tspan - x="742.2" - y="394.6" - id="tspan72">Pod</tspan> - </text> - <g - id="g74"> - <g - id="g76"> - <rect - style="fill: #ffffff" - x="1108.6" - y="368.1" - width="148" - height="133" - id="rect78" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="1108.6" - y="368.1" - width="148" - height="133" - id="rect80" /> - </g> - <g - id="g82"> - <rect - style="fill: #ffffff" - x="1148.95" - y="438.1" - width="89.3" - height="38" - id="rect84" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="1148.95" - y="438.1" - width="89.3" - height="38" - id="rect86" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="1193.6" - y="461.9" - id="text88"> - <tspan - x="1193.6" - y="461.9" - id="tspan90">container</tspan> - </text> - </g> - <g - id="g92"> - <rect - style="fill: #ffffff" - x="1138.2" - y="428.2" - width="89.3" - height="38" - id="rect94" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="1138.2" - y="428.2" - width="89.3" - height="38" - id="rect96" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="1182.85" - y="452" - id="text98"> - <tspan - x="1182.85" - y="452" - id="tspan100">container</tspan> - </text> - </g> - <g - id="g102"> - <rect - style="fill: #ffffff" - x="1127.8" - y="418.3" - width="89.3" - height="38" - id="rect104" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="1127.8" - y="418.3" - width="89.3" - height="38" - id="rect106" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="1172.45" - y="442.1" - id="text108"> - <tspan - x="1172.45" - y="442.1" - id="tspan110">container</tspan> - </text> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="1182.6" - y="434.6" - id="text112"> - <tspan - x="1182.6" - y="434.6" - id="tspan114" /> - </text> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="1130.6" - y="394.6" - id="text116"> - <tspan - x="1130.6" - y="394.6" - id="tspan118">Pod</tspan> - </text> - </g> - <g - id="g120"> - <g - id="g122"> - <rect - style="fill: #ffffff" - x="902.9" - y="368.1" - width="148" - height="133" - id="rect124" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="902.9" - y="368.1" - width="148" - height="133" - id="rect126" /> - </g> - <g - id="g128"> - <rect - style="fill: #ffffff" - x="943.25" - y="438.1" - width="89.3" - height="38" - id="rect130" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="943.25" - y="438.1" - width="89.3" - height="38" - id="rect132" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="987.9" - y="461.9" - id="text134"> - <tspan - x="987.9" - y="461.9" - id="tspan136">container</tspan> - </text> - </g> - <g - id="g138"> - <rect - style="fill: #ffffff" - x="932.5" - y="428.2" - width="89.3" - height="38" - id="rect140" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="932.5" - y="428.2" - width="89.3" - height="38" - id="rect142" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="977.15" - y="452" - id="text144"> - <tspan - x="977.15" - y="452" - id="tspan146">container</tspan> - </text> - </g> - <g - id="g148"> - <rect - style="fill: #ffffff" - x="922.1" - y="418.3" - width="89.3" - height="38" - id="rect150" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="922.1" - y="418.3" - width="89.3" - height="38" - id="rect152" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="966.75" - y="442.1" - id="text154"> - <tspan - x="966.75" - y="442.1" - id="tspan156">container</tspan> - </text> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="976.9" - y="434.6" - id="text158"> - <tspan - x="976.9" - y="434.6" - id="tspan160" /> - </text> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="924.9" - y="394.6" - id="text162"> - <tspan - x="924.9" - y="394.6" - id="tspan164">Pod</tspan> - </text> - </g> - <g - id="g166"> - <rect - style="fill: #ffffff" - x="949.748" - y="228" - width="57.1" - height="38" - id="rect168" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="949.748" - y="228" - width="57.1" - height="38" - id="rect170" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="978.298" - y="251.8" - id="text172"> - <tspan - x="978.298" - y="251.8" - id="tspan174">Proxy</tspan> - </text> - </g> - <g - id="g176"> - <rect - style="fill: #ffffff" - x="126.911" - y="92.49" - width="189.4" - height="38" - id="rect178" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="126.911" - y="92.49" - width="189.4" - height="38" - id="rect180" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="221.611" - y="116.29" - id="text182"> - <tspan - x="221.611" - y="116.29" - id="tspan184">kubectl (user commands)</tspan> - </text> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="142.476" - y="866.282" - id="text186"> - <tspan - x="142.476" - y="866.282" - id="tspan188" /> - </text> - <g - id="g190"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="758" - y1="273" - x2="782.332" - y2="408.717" - id="line192" /> - <polygon - style="fill: #000000" - points="783.655,416.099 776.969,407.138 782.332,408.717 786.812,405.374 " - id="polygon194" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="783.655,416.099 776.969,407.138 782.332,408.717 786.812,405.374 " - id="polygon196" /> - </g> - <g - id="g198"> - <rect - style="fill: #ffffff" - x="942.576" - y="75.6768" - width="70.2" - height="38" - id="rect200" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="942.576" - y="75.6768" - width="70.2" - height="38" - id="rect202" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="977.676" - y="99.4768" - id="text204"> - <tspan - x="977.676" - y="99.4768" - id="tspan206">Firewall</tspan> - </text> - </g> - <g - id="g208"> - <path - style="fill: #ffffff" - d="M 949.242 -47.953 C 939.87,-48.2618 921.694,-41.7773 924.25,-27.8819 C 926.806,-13.9865 939.018,-10.8988 944.13,-14.9129 C 949.242,-18.9271 936.178,4.54051 961.17,10.7162 C 986.161,16.8919 998.941,7.01079 995.249,-0.0912821 C 991.557,-7.19336 1017.12,16.5832 1029.04,2.99658 C 1040.97,-10.59 1016.83,-23.5589 1021.94,-21.7062 C 1027.06,-19.8535 1042.68,-22.3237 1037.56,-45.4827 C 1032.45,-68.6416 986.445,-50.7321 991.557,-54.1287 C 996.669,-57.5253 983.889,-74.5086 967.986,-71.112 C 952.082,-67.7153 950.954,-61.5516 949.25,-47.965 L 949.242,-47.953z" - id="path210" /> - <path - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - d="M 949.242 -47.953 C 939.87,-48.2618 921.694,-41.7773 924.25,-27.8819 C 926.806,-13.9865 939.018,-10.8988 944.13,-14.9129 C 949.242,-18.9271 936.178,4.54051 961.17,10.7162 C 986.161,16.8919 998.941,7.01079 995.249,-0.0912821 C 991.557,-7.19336 1017.12,16.5832 1029.04,2.99658 C 1040.97,-10.59 1016.83,-23.5589 1021.94,-21.7062 C 1027.06,-19.8535 1042.68,-22.3237 1037.56,-45.4827 C 1032.45,-68.6416 986.445,-50.7321 991.557,-54.1287 C 996.669,-57.5253 983.889,-74.5086 967.986,-71.112 C 952.082,-67.7153 950.954,-61.5516 949.25,-47.965 L 949.242,-47.953" - id="path212" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="985.428" - y="-22.3971" - id="text214"> - <tspan - x="985.428" - y="-22.3971" - id="tspan216">Internet</tspan> - </text> - </g> - <g - id="g218"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="975.985" - y1="12.703" - x2="977.415" - y2="65.9442" - id="line220" /> - <polygon - style="fill: #000000" - points="977.616,73.4415 972.349,63.5793 977.415,65.9442 982.346,63.3109 " - id="polygon222" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="977.616,73.4415 972.349,63.5793 977.415,65.9442 982.346,63.3109 " - id="polygon224" /> - </g> - <g - id="g226"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="977.676" - y1="113.677" - x2="978.245" - y2="218.264" - id="line228" /> - <polygon - style="fill: #000000" - points="978.286,225.764 973.232,215.791 978.245,218.264 983.231,215.737 " - id="polygon230" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="978.286,225.764 973.232,215.791 978.245,218.264 983.231,215.737 " - id="polygon232" /> - </g> - <g - id="g234"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="978.298" - y1="266" - x2="977.033" - y2="358.365" - id="line236" /> - <polygon - style="fill: #000000" - points="976.931,365.864 972.068,355.797 977.033,358.365 982.067,355.934 " - id="polygon238" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="976.931,365.864 972.068,355.797 977.033,358.365 982.067,355.934 " - id="polygon240" /> - </g> - <g - id="g242"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="992.572" - y1="266" - x2="1174.02" - y2="363.492" - id="line244" /> - <polygon - style="fill: #000000" - points="1180.63,367.042 1169.45,366.713 1174.02,363.492 1174.19,357.904 " - id="polygon246" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="1180.63,367.042 1169.45,366.713 1174.02,363.492 1174.19,357.904 " - id="polygon248" /> - </g> - <g - id="g250"> - <rect - style="fill: #ffffff" - x="-54" - y="370.5" - width="562" - height="383.25" - id="rect252" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="-54" - y="370.5" - width="562" - height="383.25" - id="rect254" /> - </g> - <g - id="g256"> - <rect - style="fill: #ffffff" - x="-30" - y="416.75" - width="364" - height="146" - id="rect258" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="-30" - y="416.75" - width="364" - height="146" - id="rect260" /> - </g> - <g - id="g262"> - <rect - style="fill: #ffffff" - x="128" - y="598.318" - width="189" - height="54" - id="rect264" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="128" - y="598.318" - width="189" - height="54" - id="rect266" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="222.5" - y="622.118" - id="text268"> - <tspan - x="222.5" - y="622.118" - id="tspan270">controller manager</tspan> - <tspan - x="222.5" - y="638.118" - id="tspan272">(replication controller etc.)</tspan> - </text> - </g> - <g - id="g274"> - <rect - style="fill: #ffffff" - x="15.8884" - y="622.914" - width="86.15" - height="38" - id="rect276" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="15.8884" - y="622.914" - width="86.15" - height="38" - id="rect278" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="58.9634" - y="646.714" - id="text280"> - <tspan - x="58.9634" - y="646.714" - id="tspan282">Scheduler</tspan> - </text> - </g> - <g - id="g284"> - <rect - style="fill: #ffffff" - x="1.162" - y="599.318" - width="86.15" - height="38" - id="rect286" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="1.162" - y="599.318" - width="86.15" - height="38" - id="rect288" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="44.237" - y="623.118" - id="text290"> - <tspan - x="44.237" - y="623.118" - id="tspan292">Scheduler</tspan> - </text> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="-34.876" - y="699.256" - id="text294"> - <tspan - x="-34.876" - y="699.256" - id="tspan296">Master components</tspan> - <tspan - x="-34.876" - y="715.256" - id="tspan298">Colocated, or spread across machines,</tspan> - <tspan - x="-34.876" - y="731.256" - id="tspan300">as dictated by cluster size.</tspan> - </text> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="370.886" - y="731.5" - id="text302"> - <tspan - x="370.886" - y="731.5" - id="tspan304" /> - </text> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="370.886" - y="731.5" - id="text306"> - <tspan - x="370.886" - y="731.5" - id="tspan308" /> - </text> - <g - id="g310"> - <rect - style="fill: #ffffff" - x="136.717" - y="468.5" - width="172.175" - height="70" - id="rect312" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="136.717" - y="468.5" - width="172.175" - height="70" - id="rect314" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="222.804" - y="492.3" - id="text316"> - <tspan - x="222.804" - y="492.3" - id="tspan318">REST</tspan> - <tspan - x="222.804" - y="508.3" - id="tspan320">(pods, services,</tspan> - <tspan - x="222.804" - y="524.3" - id="tspan322">rep. controllers)</tspan> - </text> - </g> - <g - id="g324"> - <rect - style="fill: #ffffff" - x="165.958" - y="389.5" - width="115" - height="54" - id="rect326" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="165.958" - y="389.5" - width="115" - height="54" - id="rect328" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="223.458" - y="413.3" - id="text330"> - <tspan - x="223.458" - y="413.3" - id="tspan332">authentication</tspan> - <tspan - x="223.458" - y="429.3" - id="tspan334">authorization</tspan> - </text> - </g> - <g - id="g336"> - <rect - style="fill: #ffffff" - x="-0.65" - y="476.5" - width="91.3" - height="54" - id="rect338" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="-0.65" - y="476.5" - width="91.3" - height="54" - id="rect340" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="45" - y="500.3" - id="text342"> - <tspan - x="45" - y="500.3" - id="tspan344">scheduling</tspan> - <tspan - x="45" - y="516.3" - id="tspan346">actuator</tspan> - </text> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="-13" - y="436.75" - id="text348"> - <tspan - x="-13" - y="436.75" - id="tspan350">APIs</tspan> - </text> - <g - id="g352"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="44.237" - y1="599.318" - x2="44.8921" - y2="540.235" - id="line354" /> - <polygon - style="fill: #000000" - points="44.9752,532.736 49.864,542.791 44.8921,540.235 39.8647,542.68 " - id="polygon356" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="44.9752,532.736 49.864,542.791 44.8921,540.235 39.8647,542.68 " - id="polygon358" /> - </g> - <g - id="g360"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="44.237" - y1="599.318" - x2="170.878" - y2="542.486" - id="line362" /> - <polygon - style="fill: #000000" - points="177.72,539.416 170.644,548.071 170.878,542.486 166.55,538.948 " - id="polygon364" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="177.72,539.416 170.644,548.071 170.878,542.486 166.55,538.948 " - id="polygon366" /> - </g> - <g - id="g368"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="222.5" - y1="598.318" - x2="222.755" - y2="548.236" - id="line370" /> - <polygon - style="fill: #000000" - points="222.793,540.736 227.742,550.761 222.755,548.236 217.742,550.71 " - id="polygon372" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="222.793,540.736 227.742,550.761 222.755,548.236 217.742,550.71 " - id="polygon374" /> - </g> - <g - id="g376"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="223.458" - y1="443.5" - x2="223.059" - y2="458.767" - id="line378" /> - <polygon - style="fill: #000000" - points="222.862,466.265 218.126,456.137 223.059,458.767 228.122,456.399 " - id="polygon380" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="222.862,466.265 218.126,456.137 223.059,458.767 228.122,456.399 " - id="polygon382" /> - </g> - <g - id="g384"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="313.554" - y1="548.463" - x2="366.76" - y2="662.181" - id="line386" /> - <polygon - style="fill: #000000" - points="318.082,546.344 309.316,539.406 309.025,550.582 " - id="polygon388" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="318.082,546.344 309.316,539.406 309.025,550.582 " - id="polygon390" /> - <polygon - style="fill: #000000" - points="369.938,668.975 361.172,662.036 366.76,662.181 370.229,657.798 " - id="polygon392" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="369.938,668.975 361.172,662.036 366.76,662.181 370.229,657.798 " - id="polygon394" /> - </g> - <g - id="g396"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="221.612" - y1="130.49" - x2="223.389" - y2="379.764" - id="line398" /> - <polygon - style="fill: #000000" - points="223.442,387.264 218.371,377.3 223.389,379.764 228.371,377.229 " - id="polygon400" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="223.442,387.264 218.371,377.3 223.389,379.764 228.371,377.229 " - id="polygon402" /> - </g> - <g - id="g404"> - <path - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - d="M 319.892 503.5 C 392.964,503.5 639.13,254 713.464,254" - id="path406" /> - <polygon - style="fill: #000000" - points="319.892,498.5 309.892,503.5 319.892,508.5 " - id="polygon408" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="319.892,498.5 309.892,503.5 319.892,508.5 " - id="polygon410" /> - <polygon - style="fill: #000000" - points="720.964,254 710.964,259 713.464,254 710.964,249 " - id="polygon412" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="720.964,254 710.964,259 713.464,254 710.964,249 " - id="polygon414" /> - </g> - <g - id="g416"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="90.65" - y1="503.5" - x2="126.981" - y2="503.5" - id="line418" /> - <polygon - style="fill: #000000" - points="134.481,503.5 124.481,508.5 126.981,503.5 124.481,498.5 " - id="polygon420" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="134.481,503.5 124.481,508.5 126.981,503.5 124.481,498.5 " - id="polygon422" /> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="221.612" - y="111.49" - id="text424"> - <tspan - x="221.612" - y="111.49" - id="tspan426" /> - </text> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="1209" - y="339.5" - id="text428"> - <tspan - x="1209" - y="339.5" - id="tspan430">docker</tspan> - </text> - <g - id="g432"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="793.753" - y1="272.636" - x2="968.266" - y2="363.6" - id="line434" /> - <polygon - style="fill: #000000" - points="974.917,367.066 963.738,366.878 968.266,363.6 968.361,358.01 " - id="polygon436" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="974.917,367.066 963.738,366.878 968.266,363.6 968.361,358.01 " - id="polygon438" /> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="978" - y="434.5" - id="text440"> - <tspan - x="978" - y="434.5" - id="tspan442">..</tspan> - </text> - <text - font-size="27.0933" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="1067" - y="437" - id="text444"> - <tspan - x="1067" - y="437" - id="tspan446">...</tspan> - </text> - <g - id="g448"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="792.8" - y1="273" - x2="1173.14" - y2="365.792" - id="line450" /> - <polygon - style="fill: #000000" - points="1180.43,367.57 1169.53,370.057 1173.14,365.792 1171.9,360.342 " - id="polygon452" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="1180.43,367.57 1169.53,370.057 1173.14,365.792 1171.9,360.342 " - id="polygon454" /> - </g> - <g - id="g456"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="792.8" - y1="273" - x2="794.057" - y2="358.365" - id="line458" /> - <polygon - style="fill: #000000" - points="794.167,365.864 789.02,355.939 794.057,358.365 799.019,355.792 " - id="polygon460" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="794.167,365.864 789.02,355.939 794.057,358.365 799.019,355.792 " - id="polygon462" /> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="720" - y="220" - id="text464"> - <tspan - x="720" - y="220" - id="tspan466" /> - </text> - <g - id="g468"> - <rect - style="fill: #ffffff" - x="660" - y="660" - width="630" - height="381" - id="rect470" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="660" - y="660" - width="630" - height="381" - id="rect472" /> - </g> - <g - id="g474"> - <rect - style="fill: #ffffff" - x="686" - y="789" - width="580" - height="227" - id="rect476" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="686" - y="789" - width="580" - height="227" - id="rect478" /> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="685" - y="692" - id="text480"> - <tspan - x="685" - y="692" - id="tspan482">Node</tspan> - </text> - <g - id="g484"> - <rect - style="fill: #ffffff" - x="721.2" - y="703" - width="69.6" - height="38" - id="rect486" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="721.2" - y="703" - width="69.6" - height="38" - id="rect488" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="756" - y="726.8" - id="text490"> - <tspan - x="756" - y="726.8" - id="tspan492">kubelet</tspan> - </text> - </g> - <g - id="g494"> - <rect - style="fill: #ffffff" - x="718.2" - y="836.1" - width="148" - height="133" - id="rect496" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="718.2" - y="836.1" - width="148" - height="133" - id="rect498" /> - </g> - <g - id="g500"> - <rect - style="fill: #ffffff" - x="758.55" - y="906.1" - width="89.3" - height="38" - id="rect502" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="758.55" - y="906.1" - width="89.3" - height="38" - id="rect504" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="803.2" - y="929.9" - id="text506"> - <tspan - x="803.2" - y="929.9" - id="tspan508">container</tspan> - </text> - </g> - <g - id="g510"> - <rect - style="fill: #ffffff" - x="747.8" - y="896.2" - width="89.3" - height="38" - id="rect512" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="747.8" - y="896.2" - width="89.3" - height="38" - id="rect514" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="792.45" - y="920" - id="text516"> - <tspan - x="792.45" - y="920" - id="tspan518">container</tspan> - </text> - </g> - <g - id="g520"> - <rect - style="fill: #ffffff" - x="737.4" - y="886.3" - width="89.3" - height="38" - id="rect522" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="737.4" - y="886.3" - width="89.3" - height="38" - id="rect524" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="782.05" - y="910.1" - id="text526"> - <tspan - x="782.05" - y="910.1" - id="tspan528">cAdvisor</tspan> - </text> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="792.2" - y="902.6" - id="text530"> - <tspan - x="792.2" - y="902.6" - id="tspan532" /> - </text> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="740.2" - y="862.6" - id="text534"> - <tspan - x="740.2" - y="862.6" - id="tspan536">Pod</tspan> - </text> - <g - id="g538"> - <g - id="g540"> - <rect - style="fill: #ffffff" - x="1106.6" - y="836.1" - width="148" - height="133" - id="rect542" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="1106.6" - y="836.1" - width="148" - height="133" - id="rect544" /> - </g> - <g - id="g546"> - <rect - style="fill: #ffffff" - x="1146.95" - y="906.1" - width="89.3" - height="38" - id="rect548" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="1146.95" - y="906.1" - width="89.3" - height="38" - id="rect550" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="1191.6" - y="929.9" - id="text552"> - <tspan - x="1191.6" - y="929.9" - id="tspan554">container</tspan> - </text> - </g> - <g - id="g556"> - <rect - style="fill: #ffffff" - x="1136.2" - y="896.2" - width="89.3" - height="38" - id="rect558" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="1136.2" - y="896.2" - width="89.3" - height="38" - id="rect560" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="1180.85" - y="920" - id="text562"> - <tspan - x="1180.85" - y="920" - id="tspan564">container</tspan> - </text> - </g> - <g - id="g566"> - <rect - style="fill: #ffffff" - x="1125.8" - y="886.3" - width="89.3" - height="38" - id="rect568" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="1125.8" - y="886.3" - width="89.3" - height="38" - id="rect570" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="1170.45" - y="910.1" - id="text572"> - <tspan - x="1170.45" - y="910.1" - id="tspan574">container</tspan> - </text> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="1180.6" - y="902.6" - id="text576"> - <tspan - x="1180.6" - y="902.6" - id="tspan578" /> - </text> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="1128.6" - y="862.6" - id="text580"> - <tspan - x="1128.6" - y="862.6" - id="tspan582">Pod</tspan> - </text> - </g> - <g - id="g584"> - <g - id="g586"> - <rect - style="fill: #ffffff" - x="900.9" - y="836.1" - width="148" - height="133" - id="rect588" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="900.9" - y="836.1" - width="148" - height="133" - id="rect590" /> - </g> - <g - id="g592"> - <rect - style="fill: #ffffff" - x="941.25" - y="906.1" - width="89.3" - height="38" - id="rect594" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="941.25" - y="906.1" - width="89.3" - height="38" - id="rect596" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="985.9" - y="929.9" - id="text598"> - <tspan - x="985.9" - y="929.9" - id="tspan600">container</tspan> - </text> - </g> - <g - id="g602"> - <rect - style="fill: #ffffff" - x="930.5" - y="896.2" - width="89.3" - height="38" - id="rect604" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="930.5" - y="896.2" - width="89.3" - height="38" - id="rect606" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="975.15" - y="920" - id="text608"> - <tspan - x="975.15" - y="920" - id="tspan610">container</tspan> - </text> - </g> - <g - id="g612"> - <rect - style="fill: #ffffff" - x="920.1" - y="886.3" - width="89.3" - height="38" - id="rect614" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="920.1" - y="886.3" - width="89.3" - height="38" - id="rect616" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="964.75" - y="910.1" - id="text618"> - <tspan - x="964.75" - y="910.1" - id="tspan620">container</tspan> - </text> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="974.9" - y="902.6" - id="text622"> - <tspan - x="974.9" - y="902.6" - id="tspan624" /> - </text> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="922.9" - y="862.6" - id="text626"> - <tspan - x="922.9" - y="862.6" - id="tspan628">Pod</tspan> - </text> - </g> - <g - id="g630"> - <rect - style="fill: #ffffff" - x="947.748" - y="696" - width="57.1" - height="38" - id="rect632" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="947.748" - y="696" - width="57.1" - height="38" - id="rect634" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="976.298" - y="719.8" - id="text636"> - <tspan - x="976.298" - y="719.8" - id="tspan638">Proxy</tspan> - </text> - </g> - <g - id="g640"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="756" - y1="741" - x2="780.332" - y2="876.717" - id="line642" /> - <polygon - style="fill: #000000" - points="781.655,884.099 774.969,875.138 780.332,876.717 784.812,873.374 " - id="polygon644" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="781.655,884.099 774.969,875.138 780.332,876.717 784.812,873.374 " - id="polygon646" /> - </g> - <g - id="g648"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="976.298" - y1="734" - x2="975.033" - y2="826.365" - id="line650" /> - <polygon - style="fill: #000000" - points="974.931,833.864 970.068,823.797 975.033,826.365 980.067,823.934 " - id="polygon652" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="974.931,833.864 970.068,823.797 975.033,826.365 980.067,823.934 " - id="polygon654" /> - </g> - <g - id="g656"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="990.572" - y1="734" - x2="1172.02" - y2="831.492" - id="line658" /> - <polygon - style="fill: #000000" - points="1178.63,835.042 1167.45,834.713 1172.02,831.492 1172.19,825.904 " - id="polygon660" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="1178.63,835.042 1167.45,834.713 1172.02,831.492 1172.19,825.904 " - id="polygon662" /> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="1207" - y="807.5" - id="text664"> - <tspan - x="1207" - y="807.5" - id="tspan666">docker</tspan> - </text> - <g - id="g668"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="791.753" - y1="740.636" - x2="966.266" - y2="831.6" - id="line670" /> - <polygon - style="fill: #000000" - points="972.917,835.066 961.738,834.878 966.266,831.6 966.361,826.01 " - id="polygon672" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="972.917,835.066 961.738,834.878 966.266,831.6 966.361,826.01 " - id="polygon674" /> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="976" - y="902.5" - id="text676"> - <tspan - x="976" - y="902.5" - id="tspan678">..</tspan> - </text> - <text - font-size="27.0933" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="1065" - y="905" - id="text680"> - <tspan - x="1065" - y="905" - id="tspan682">...</tspan> - </text> - <g - id="g684"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="790.8" - y1="741" - x2="1171.14" - y2="833.792" - id="line686" /> - <polygon - style="fill: #000000" - points="1178.43,835.57 1167.53,838.057 1171.14,833.792 1169.9,828.342 " - id="polygon688" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="1178.43,835.57 1167.53,838.057 1171.14,833.792 1169.9,828.342 " - id="polygon690" /> - </g> - <g - id="g692"> - <line - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x1="790.8" - y1="741" - x2="792.057" - y2="826.365" - id="line694" /> - <polygon - style="fill: #000000" - points="792.167,833.864 787.02,823.939 792.057,826.365 797.019,823.792 " - id="polygon696" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="792.167,833.864 787.02,823.939 792.057,826.365 797.019,823.792 " - id="polygon698" /> - </g> - <text - font-size="12.8" - style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" - x="718" - y="688" - id="text700"> - <tspan - x="718" - y="688" - id="tspan702" /> - </text> - <g - id="g704"> - <path - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - d="M 319.892 521 C 392.964,521 637.13,722 711.464,722" - id="path706" /> - <polygon - style="fill: #000000" - points="319.892,516 309.892,521 319.892,526 " - id="polygon708" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="319.892,516 309.892,521 319.892,526 " - id="polygon710" /> - <polygon - style="fill: #000000" - points="718.964,722 708.964,727 711.464,722 708.964,717 " - id="polygon712" /> - <polygon - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - points="718.964,722 708.964,727 711.464,722 708.964,717 " - id="polygon714" /> - </g> - <g - id="g716"> - <rect - style="fill: #ffffff" - x="282.774" - y="671" - width="176.225" - height="121" - id="rect718" /> - <rect - style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" - x="282.774" - y="671" - width="176.225" - height="121" - id="rect720" /> - <text - font-size="12.8" - style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" - x="370.886" - y="704.3" - id="text722"> - <tspan - x="370.886" - y="704.3" - id="tspan724">Distributed</tspan> - <tspan - x="370.886" - y="720.3" - id="tspan726">Watchable</tspan> - <tspan - x="370.886" - y="736.3" - id="tspan728">Storage</tspan> - <tspan - x="370.886" - y="752.3" - id="tspan730" /> - <tspan - x="370.886" - y="768.3" - id="tspan732">(implemented via etcd)</tspan> - </text> - </g> -</svg> diff --git a/contributors/design-proposals/architecture/OWNERS b/contributors/design-proposals/architecture/OWNERS new file mode 100644 index 00000000..87364abb --- /dev/null +++ b/contributors/design-proposals/architecture/OWNERS @@ -0,0 +1,8 @@ +reviewers: + - sig-architecture-leads + - jbeda +approvers: + - sig-architecture-leads + - jbeda +labels: + - sig/architecture diff --git a/contributors/design-proposals/architecture.md b/contributors/design-proposals/architecture/architecture.md index 2c0fc316..8387c886 100644 --- a/contributors/design-proposals/architecture.md +++ b/contributors/design-proposals/architecture/architecture.md @@ -1,5 +1,8 @@ # Kubernetes Design and Architecture +A much more detailed and updated [Architectural +Roadmap](../../devel/architectural-roadmap.md) is also available. + ## Overview Kubernetes is a production-grade, open-source infrastructure for the deployment, scaling, @@ -82,7 +85,7 @@ A running Kubernetes cluster contains node agents (kubelet) and a cluster contro The Kubernetes [control plane](https://en.wikipedia.org/wiki/Control_plane) is split into a set of components, which can all run on a single *master* node, or can be replicated in order to support high-availability clusters, or can even be run on Kubernetes itself (AKA -[self-hosted](self-hosted-kubernetes.md#what-is-self-hosted)). +[self-hosted](../cluster-lifecycle/self-hosted-kubernetes.md#what-is-self-hosted)). Kubernetes provides a REST API supporting primarily CRUD operations on (mostly) persistent resources, which serve as the hub of its control plane. Kubernetes’s API provides IaaS-like @@ -170,7 +173,7 @@ Kubernetes supports user-provided schedulers and multiple concurrent cluster sch using the shared-state approach pioneered by [Omega](https://research.google.com/pubs/pub41684.html). In addition to the disadvantages of pessimistic concurrency described by the Omega paper, -[two-level scheduling models](http://mesos.berkeley.edu/mesos_tech_report.pdf) that hide information from the upper-level +[two-level scheduling models](https://amplab.cs.berkeley.edu/wp-content/uploads/2011/06/Mesos-A-Platform-for-Fine-Grained-Resource-Sharing-in-the-Data-Center.pdf) that hide information from the upper-level schedulers need to implement all of the same features in the lower-level scheduler as required by all upper-layer schedulers in order to ensure that their scheduling requests can be satisfied by available desired resources. @@ -214,7 +217,7 @@ agent. Each node runs a container runtime, which is responsible for downloading images and running containers. Kubelet does not link in the base container runtime. Instead, we're defining a -[Container Runtime Interface](container-runtime-interface-v1.md) to control the +[Container Runtime Interface](/contributors/devel/container-runtime-interface.md) to control the underlying runtime and facilitate pluggability of that layer. This decoupling is needed in order to maintain clear component boundaries, facilitate testing, and facilitate pluggability. Runtimes supported today, either upstream or by forks, include at least docker (for Linux and Windows), @@ -225,7 +228,7 @@ Runtimes supported today, either upstream or by forks, include at least docker ( The [service](https://kubernetes.io/docs/concepts/services-networking/service/) abstraction provides a way to group pods under a common access policy (e.g., load-balanced). The implementation of this creates -A virtual IP which clients can access and which is transparently proxied to the pods in a Service. +a virtual IP which clients can access and which is transparently proxied to the pods in a Service. Each node runs a [kube-proxy](https://kubernetes.io/docs/admin/kube-proxy/) process which programs `iptables` rules to trap access to service IPs and redirect them to the correct backends. This provides a highly-available load-balancing solution with low performance overhead by balancing client traffic from a node on that same node. @@ -234,10 +237,10 @@ Service endpoints are found primarily via [DNS](https://kubernetes.io/docs/conce ### Add-ons and other dependencies -A number of components, called [*add-ons*](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons) typically run on Kubernetes +A number of components, called [*add-ons*](https://git.k8s.io/kubernetes/cluster/addons) typically run on Kubernetes itself: -* [DNS](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns) -* [Ingress controller](https://github.com/kubernetes/ingress/tree/master/controllers) +* [DNS](https://git.k8s.io/kubernetes/cluster/addons/dns) +* [Ingress controller](https://github.com/kubernetes/ingress-gce) * [Heapster](https://github.com/kubernetes/heapster/) (resource monitoring) * [Dashboard](https://github.com/kubernetes/dashboard/) (GUI) @@ -245,8 +248,4 @@ itself: A single Kubernetes cluster may span multiple availability zones. -However, for the highest availability, we recommend using [cluster federation](federation.md). - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> +However, for the highest availability, we recommend using [cluster federation](../multicluster/federation.md). diff --git a/contributors/design-proposals/identifiers.md b/contributors/design-proposals/architecture/identifiers.md index a37411f9..977c0c99 100644 --- a/contributors/design-proposals/identifiers.md +++ b/contributors/design-proposals/architecture/identifiers.md @@ -107,7 +107,3 @@ from whence it came. unique across time. 1. This may correspond to Docker's container ID. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/namespaces.md b/contributors/design-proposals/architecture/namespaces.md index 703c1cb3..a5536cba 100644 --- a/contributors/design-proposals/namespaces.md +++ b/contributors/design-proposals/architecture/namespaces.md @@ -83,7 +83,7 @@ and reference particular entities across operations. A *Namespace* provides an authorization scope for accessing content associated with the *Namespace*. -See [Authorization plugins](../admin/authorization.md) +See [Authorization plugins](https://kubernetes.io/docs/admin/authorization/) ### Limit Resource Consumption @@ -92,13 +92,13 @@ A *Namespace* provides a scope to limit resource consumption. A *LimitRange* defines min/max constraints on the amount of resources a single entity can consume in a *Namespace*. -See [Admission control: Limit Range](admission_control_limit_range.md) +See [Admission control: Limit Range](../resource-management/admission_control_limit_range.md) A *ResourceQuota* tracks aggregate usage of resources in the *Namespace* and allows cluster operators to define *Hard* resource usage limits that a *Namespace* may consume. -See [Admission control: Resource Quota](admission_control_resource_quota.md) +See [Admission control: Resource Quota](../resource-management/admission_control_resource_quota.md) ### Finalizers @@ -363,8 +363,3 @@ storage. At this point, all content associated with that Namespace, and the Namespace itself are gone. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/principles.md b/contributors/design-proposals/architecture/principles.md index fe66d519..ae98d660 100644 --- a/contributors/design-proposals/principles.md +++ b/contributors/design-proposals/architecture/principles.md @@ -4,7 +4,7 @@ Principles to follow when extending Kubernetes. ## API -See also the [API conventions](../devel/api-conventions.md). +See also the [API conventions](../../devel/api-conventions.md). * All APIs should be declarative. * API objects should be complementary and composable, not opaque wrappers. @@ -96,7 +96,3 @@ TODO * [Eric Raymond's 17 UNIX rules](https://en.wikipedia.org/wiki/Unix_philosophy#Eric_Raymond.E2.80.99s_17_Unix_Rules) - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/auth/OWNERS b/contributors/design-proposals/auth/OWNERS new file mode 100644 index 00000000..3100c753 --- /dev/null +++ b/contributors/design-proposals/auth/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-auth-leads +approvers: + - sig-auth-leads +labels: + - sig/auth diff --git a/contributors/design-proposals/access.md b/contributors/design-proposals/auth/access.md index b23e463b..927bb033 100644 --- a/contributors/design-proposals/access.md +++ b/contributors/design-proposals/auth/access.md @@ -212,7 +212,7 @@ Engine `project`. It provides a namespace for objects created by a group of people co-operating together, preventing name collisions with non-cooperating groups. It also serves as a reference point for authorization policies. -Namespaces are described in [namespaces.md](namespaces.md). +Namespaces are described in [namespaces](../architecture/namespaces.md). In the Enterprise Profile: - a `userAccount` may have permission to access several `namespace`s. @@ -223,7 +223,7 @@ In the Simple Profile: Namespaces versus userAccount vs. Labels: - `userAccount`s are intended for audit logging (both name and UID should be logged), and to define who has access to `namespace`s. -- `labels` (see [docs/user-guide/labels.md](../../docs/user-guide/labels.md)) +- `labels` (see [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)) should be used to distinguish pods, users, and other objects that cooperate towards a common goal but are different in some way, such as version, or responsibilities. @@ -326,7 +326,7 @@ namespaces, or to make a K8s User into a K8s Project Admin.) The API should have a `quota` concept (see http://issue.k8s.io/442). A quota object relates a namespace (and optionally a label selector) to a maximum -quantity of resources that may be used (see [resources design doc](resources.md)). +quantity of resources that may be used (see [resources design doc](../scheduling/resources.md)). Initially: - A `quota` object is immutable. @@ -370,7 +370,3 @@ Improvements: - Policies to drop logging for high rate trusted API calls, or by users performing audit or other sensitive functions. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/apparmor.md b/contributors/design-proposals/auth/apparmor.md index 0a4f620a..a88154bb 100644 --- a/contributors/design-proposals/apparmor.md +++ b/contributors/design-proposals/auth/apparmor.md @@ -1,5 +1,3 @@ -<!-- BEGIN MUNGE: GENERATED_TOC --> - - [Overview](#overview) - [Motivation](#motivation) - [Related work](#related-work) @@ -22,8 +20,6 @@ - [Profile authoring](#profile-authoring) - [Appendix](#appendix) -<!-- END MUNGE: GENERATED_TOC --> - # Overview AppArmor is a [mandatory access control](https://en.wikipedia.org/wiki/Mandatory_access_control) @@ -85,7 +81,7 @@ annotation. If a profile is specified, the Kubelet will verify that the node mee the container, and will not run the container if the profile cannot be applied. If the requirements are met, the container runtime will configure the appropriate options to apply the profile. Profile requirements and defaults can be specified on the -[PodSecurityPolicy](security-context-constraints.md). +[PodSecurityPolicy](pod-security-policy.md). ## Prerequisites @@ -136,7 +132,7 @@ The profiles can be specified in the following formats (following the convention ### Pod Security Policy -The [PodSecurityPolicy](security-context-constraints.md) allows cluster administrators to control +The [PodSecurityPolicy](pod-security-policy.md) allows cluster administrators to control the security context for a pod and its containers. An annotation can be specified on the PodSecurityPolicy to restrict which AppArmor profiles can be used, and specify a default if no profile is specified. @@ -272,7 +268,7 @@ already underway for Docker, called ## Container Runtime Interface Other container runtimes will likely add AppArmor support eventually, so the -[Container Runtime Interface](container-runtime-interface-v1.md) (CRI) needs to be made compatible +[Container Runtime Interface](/contributors/devel/container-runtime-interface.md) (CRI) needs to be made compatible with this design. The two important pieces are a way to report whether AppArmor is supported by the runtime, and a way to specify the profile to load (likely through the `LinuxContainerConfig`). @@ -304,7 +300,3 @@ documentation for following this process in a Kubernetes environment. ``` $ apparmor_parser --remove /path/to/profile ``` - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/auth/cluster-role-aggregation.md b/contributors/design-proposals/auth/cluster-role-aggregation.md new file mode 100644 index 00000000..12739589 --- /dev/null +++ b/contributors/design-proposals/auth/cluster-role-aggregation.md @@ -0,0 +1,94 @@ +# Cluster Role Aggregation +In order to support easy RBAC integration for CustomResources and Extension +APIServers, we need to have a way for API extenders to add permissions to the +"normal" roles for admin, edit, and view. + +These roles express an intent for the namespaced power of administrators of the +namespace (manage ownership), editors of the namespace (manage content like +pods), and viewers of the namespace (see what is present). As new APIs are +made available, these roles should reflect that intent to prevent migration +concerns every time a new API is added. + +To do this, we will allow one ClusterRole to be built out of a selected set of +ClusterRoles. + +## API Changes +```yaml +aggregationRule: + selectors: + - matchLabels: + rbac.authorization.k8s.io/aggregate-to-admin: true +``` + +```go +// ClusterRole is a cluster level, logical grouping of PolicyRules that can be referenced as a unit by a RoleBinding or ClusterRoleBinding. +type ClusterRole struct { + metav1.TypeMeta + // Standard object's metadata. + metav1.ObjectMeta + + // Rules holds all the PolicyRules for this ClusterRole + Rules []PolicyRule + + // AggregationRule is an optional field that describes how to build the Rules for this ClusterRole. + // If AggregationRule is set, then the Rules are controller managed and direct changes to Rules will be + // stomped by the controller. + AggregationRule *AggregationRule +} + +// AggregationRule describes how to locate ClusterRoles to aggregate into the ClusterRole +type AggregationRule struct { + // Selector holds a list of selectors which will be used to find ClusterRoles and create the rules. + // If any of the selectors match, then the ClusterRole's permissions will be added + Selectors []metav1.LabelSelector +} +``` + +The `aggregationRule` stanza contains a list of LabelSelectors which are used +to select the set of ClusterRoles which should be combined. When +`aggregationRule` is set, the list of `rules` becomes controller managed and is +subject to overwriting at any point. + +`aggregationRule` needs to be protected from escalation. The simplest way to +do this is to restrict it to users with verb=`*`, apiGroups=`*`, resources=`*`. We +could later loosen it by using a covers check against all aggregated rules +without changing backward compatibility. + +## Controller +There is a controller which watches for changes to ClusterRoles and then +updates all aggregated ClusterRoles if their list of Rules has changed. Since +there are relatively few ClusterRoles, it checks them all and most +short-circuit. + +## The Payoff +If you want to create a CustomResource for your operator and you want namespace +admin's to be able to create one, instead of trying to: + 1. Create a new ClusterRole + 2. Update every namespace with a matching RoleBinding + 3. Teach everyone to add the RoleBinding to all their admin users + 4. When you remove it, clean up dangling RoleBindings + + Or + + 1. Make a non-declarative patch against the admin ClusterRole + 2. When you remove it, try to safely create a new non-declarative patch to + remove it. + +You can simply create a new ClusterRole like +```yaml +apiVersion: rbac.authorization.k8s.io/v1beta1 +kind: ClusterRole +metadata: + name: etcd-operator-admin + label: + rbac.authorization.k8s.io/aggregate-to-admin: true +rules: +- apiGroups: + - etcd.database.coreos.com + resources: + - etcdclusters + verbs: + - "*" +``` +alongside your CustomResourceDefinition. The admin role is updated correctly and +removal is a `kubectl delete -f` away.
\ No newline at end of file diff --git a/contributors/design-proposals/enhance-pluggable-policy.md b/contributors/design-proposals/auth/enhance-pluggable-policy.md index 14fff3dd..29eff236 100644 --- a/contributors/design-proposals/enhance-pluggable-policy.md +++ b/contributors/design-proposals/auth/enhance-pluggable-policy.md @@ -422,8 +422,3 @@ type LocalResourceAccessReviewResponse struct { Groups []string } ``` - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/auth/flex-volumes-drivers-psp.md b/contributors/design-proposals/auth/flex-volumes-drivers-psp.md new file mode 100644 index 00000000..453d736f --- /dev/null +++ b/contributors/design-proposals/auth/flex-volumes-drivers-psp.md @@ -0,0 +1,94 @@ +# Allow Pod Security Policy to manage access to the Flexvolumes + +## Current state + +Cluster admins can control the usage of specific volume types by using Pod +Security Policy (PSP). Admins can allow the use of Flexvolumes by listing the +`flexVolume` type in the `volumes` field. The only thing that can be managed is +allowance or disallowance of Flexvolumes. + +Technically, Flexvolumes are implemented as vendor drivers. They are executable +files that must be placed on every node at +`/usr/libexec/kubernetes/kubelet-plugins/volume/exec/<vendor~driver>/<driver>`. +In most cases they are scripts. Limiting driver access means not only limiting +an access to the volumes that this driver can provide, but also managing access +to executing a driver’s code (that is arbitrary, in fact). + +It is possible to have many flex drivers for the different storage types. In +essence, Flexvolumes represent not a single volume type, but the different +types that allow usage of various vendor volumes. + +## Desired state + +In order to further improve security and to provide more granular control for +the usage of the different Flexvolumes, we need to enhance PSP. When such a +change takes place, cluster admins will be able to grant access to any +Flexvolumes of a particular driver (in contrast to any volume of all drivers). + +For example, if we have two drivers for Flexvolumes (`cifs` and +`digitalocean`), it will become possible to grant access for one group to use +only volumes from DigitalOcean and grant access for another group to use +volumes from all Flexvolumes. + +## Proposed changes + +It has been suggested to add a whitelist of allowed Flexvolume drivers to the +PSP. It should behave similar to [the existing +`allowedHostPaths`](https://github.com/kubernetes/kubernetes/pull/50212) except +that: + +1) comparison of equality will be used instead of comparison of prefixes. +2) Flexvolume’s driver field will be inspected rather than `hostPath`’s path field. + +### PodSecurityPolicy modifications + +```go +// PodSecurityPolicySpec defines the policy enforced. +type PodSecurityPolicySpec struct { + ... + // AllowedFlexVolumes is a whitelist of allowed Flexvolumes. Empty or nil indicates that all + // Flexvolumes may be used. This parameter is effective only when the usage of the Flexvolumes + // is allowed in the "Volumes" field. + // +optional + AllowedFlexVolumes []AllowedFlexVolume +} + +// AllowedFlexVolume represents a single Flexvolume that is allowed to be used. +type AllowedFlexVolume struct { + // Driver is the name of the Flexvolume driver. + Driver string +} +``` + +Empty `AllowedFlexVolumes` allows usage of Flexvolumes with any driver. It must +behave as before and provide backward compatibility. + +Non-empty `AllowedFlexVolumes` changes the behavior from "all allowed" to "all +disallowed except those that are explicitly listed here". + +### Admission controller modifications + +Admission controller should be updated accordingly to inspect a Pod's volumes. +If it finds a `flexVolume`, it should ensure that its driver is allowed to be +used. + +### Validation rules + +Flexvolume driver names must be non-empty. + +If a PSP disallows to pods to request volumes of type `flexVolume` then +`AllowedFlexVolumes` must be empty. In case it is not empty, API server must +report an error. + +API server should allow granting an access to Flexvolumes that do not exist at +time of PSP creation. + +## Notes +It is possible to have even more flexible control over the Flexvolumes and take +into account options that have been passed to a driver. We decided that this is +a desirable feature but outside the scope of this proposal. + +The current change could be enough for many cases. Also, when cluster admins +are able to manage access to particular Flexvolume drivers, it becomes possible +to "emulate" control over the driver’s options by using many drivers with +hard-coded options. diff --git a/contributors/design-proposals/image-provenance.md b/contributors/design-proposals/auth/image-provenance.md index 7a5580d9..942ea416 100644 --- a/contributors/design-proposals/image-provenance.md +++ b/contributors/design-proposals/auth/image-provenance.md @@ -322,10 +322,5 @@ It will not be a generic webhook. A generic webhook would need a lot more discus Additionally, just sending all the fields of just the Pod kind also has problems: - it exposes our whole API to a webhook backend without giving us (the project) any chance to review or understand how it is being used. - because we do not know which fields of an object are inspected by the backend, caching of decisions is not effective. Sending fewer fields allows caching. -- sending fewer fields makes it possible to rev the version of the webhook request slower than the version of our internal obejcts (e.g. pod v2 could still use imageReview v1.) +- sending fewer fields makes it possible to rev the version of the webhook request slower than the version of our internal objects (e.g. pod v2 could still use imageReview v1.) probably lots more reasons. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/auth/kms-grpc-class-diagram.png b/contributors/design-proposals/auth/kms-grpc-class-diagram.png Binary files differnew file mode 100644 index 00000000..fe63d8d0 --- /dev/null +++ b/contributors/design-proposals/auth/kms-grpc-class-diagram.png diff --git a/contributors/design-proposals/auth/kms-grpc-deployment-diagram.png b/contributors/design-proposals/auth/kms-grpc-deployment-diagram.png Binary files differnew file mode 100644 index 00000000..c5ff1df5 --- /dev/null +++ b/contributors/design-proposals/auth/kms-grpc-deployment-diagram.png diff --git a/contributors/design-proposals/auth/kms-plugin-grpc-api.md b/contributors/design-proposals/auth/kms-plugin-grpc-api.md new file mode 100644 index 00000000..f31dd6c9 --- /dev/null +++ b/contributors/design-proposals/auth/kms-plugin-grpc-api.md @@ -0,0 +1,114 @@ +# KMS Plugin API for secrets encryption + +## Background + +Since v1.7, Kubernetes allows encryption of resources. It supports 3 kinds of encryptions: ``aescbc``, ``aesgcm`` and ``secretbox``. They are implemented as value transformer. This feature currently only supports encryption using keys in the configuration file (plain text, encoded with base64). + +Using an external trusted service to manage the keys separates the responsibility of key management from operating and managing a Kubernetes cluster. So a new transformer, “Envelope Transformer”, was introduced in 1.8 ([49350](https://github.com/kubernetes/kubernetes/pull/49350)). “Envelope Transformer” defines an extension point, the interface ``envelope.Service``. The intent was to make it easy to add new KMS provider by implementing the interface. For example the provider for Google Cloud KMS, Hashicorp Vault and Microsoft azure KeyVault. + +But as more KMS providers are added, more vendor dependencies are also introduced. So now we wish to pull all KMS providers out of API server, while retaining the ability of the API server to delegate encrypting secrets to an external trusted KMS service . + +## High Level Design + +At a high-level, (see [51965](https://github.com/kubernetes/kubernetes/issues/51965)), use gRPC to decouple the API server from the out-of-tree KMS providers. There is only one envelope service implementation (implement the interface ``envelope.Service``), and the envelope service communicates with the out-of-tree KMS provider through gRPC. The deployment diagram is like below: + + + +Here we assume the remote KMS provider is accessible from the API server. Not care how to launch the KMS provider process and how to manage it in this document. + +API server side (gRPC client) should know nothing about the external KMS. We only need to configure the KMS provider (gRPC server) endpoint for it. + +KMS provider (gRPC server) must handle all details related to external KMS. It need know how to connect to KMS, how to pass the authentication, and which key or keys will be used, etc.. A qualified KMS provider implementation should hide all details from API server. + +To add new KMS provider, we just implement the gRPC server. No new code or dependencies are added into API server. We configure the API server to make the gRPC client communicate with the new KMS provider. + +Following is the class diagram that illustrates a possible implementation: + + + +The class ``envelope.envelopeTransformer`` and the interface ``envelope.Service`` exists in current code base. What we need to do is to implement the class ``envelope.grpcService``. + +## Proto File Definition + +```protobuf +// envelope/service.proto +syntax = "proto3"; + +package envelope; + +service KMSService { + // Version returns the runtime name and runtime version. + rpc Version(VersionRequest) returns (VersionResponse) {} + rpc Decrypt(DecryptRequest) returns (DecryptResponse) {} + rpc Encrypt(EncryptRequest) returns (EncryptResponse) {} +} + +message VersionRequest { + // Version of the KMS plugin API. + string version = 1; +} + +message VersionResponse { + // Version of the KMS plugin API. + string version = 1; + // Name of the KMS provider. + string runtime_name = 2; + // Version of the KMS provider. The string must be semver-compatible. + string runtime_version = 3; +} + +message DecryptRequest { + // Version of the KMS plugin API, now use “v1beta1” + string version = 1; + bytes cipher = 2; +} + +message DecryptResponse { + bytes plain = 1; +} + +message EncryptRequest { + // Version of the KMS plugin API, now use “v1beta1” + string version = 1; + bytes plain = 2; +} + +message EncryptResponse { + bytes cipher = 1; +} +``` + +## Deployment + +To avoid the need to implement authentication and authorization, the KMS provider should run on the master and be called via a local unix domain socket. + +Cluster administrators have various options to ensure the KMS provider runs on the master: taints and tolerations is one. On GKE we target configuration at the kubelet that runs on the master directly (it isn't registered as a regular kubelet) and will have it start the KMS provider. + +## Performance + +The KMS provider will be called on every secret write and make a remote RPC to the KMS provider to do the actual encrypt/decrypt. To keep the overhead of the gRPC call to the KMS provider low, the KMS provider should run on the master. This should mean the extra overhead is small compared to the remote call. + +Unencrypted DEKs are cached on the API server side, so gRPC calls to the KMS provider are only required to fill the cache on startup. + +## Configuration + +The out-of-tree provider will be specified in existing configuration file used to configure any of the encryption providers. The location of this configuration file is identified by the existing startup parameter ``--experimental-encryption-provider-config``. + +To specify the gRPC server endpoint, we add a new configuration parameter ``endpoint`` for the KMS configuration in a current deployment. The endpoint is a unix domain socket connection, for example ``unix:///tmp/kms-provider.sock``. + +Now we expect the API server and KMS provider run in the same Pod. Not support TCP socket connection. So it’s not necessary to add TLS support. + +Here is a sample configuration file with vault out-of-tree provider configured: + +```yaml +kind: EncryptionConfig +apiVersion: v1 +resources: + - resources: + - secrets + providers: + - kms: + name: grpc-kms-provider + cachesize: 1000 + endpoint: unix:///tmp/kms-provider.sock +``` diff --git a/contributors/design-proposals/no-new-privs.md b/contributors/design-proposals/auth/no-new-privs.md index f764e399..b467c35d 100644 --- a/contributors/design-proposals/no-new-privs.md +++ b/contributors/design-proposals/auth/no-new-privs.md @@ -24,7 +24,7 @@ is inherited across `fork`, `clone` and `execve` and can not be unset. With that could not have been done without the `execve` call. For more details about `no_new_privs`, please check the -[Linux kernel documention](https://www.kernel.org/doc/Documentation/prctl/no_new_privs.txt). +[Linux kernel documentation](https://www.kernel.org/doc/Documentation/prctl/no_new_privs.txt). This is different from `NOSUID` in that `no_new_privs`can give permission to the container process to further restrict child processes with seccomp. This diff --git a/contributors/design-proposals/pod-security-context.md b/contributors/design-proposals/auth/pod-security-context.md index bfaffa59..29ff7ff0 100644 --- a/contributors/design-proposals/pod-security-context.md +++ b/contributors/design-proposals/auth/pod-security-context.md @@ -368,7 +368,3 @@ E2E test cases will be added to test the correct determination of the security c 1. The Kubelet will use the new fields on the `PodSecurityContext` for host namespace control 2. The Kubelet will be modified to correctly implement the backward compatibility and effective security context determination defined here - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/security-context-constraints.md b/contributors/design-proposals/auth/pod-security-policy.md index a61d2f3b..39a4e4bc 100644 --- a/contributors/design-proposals/security-context-constraints.md +++ b/contributors/design-proposals/auth/pod-security-policy.md @@ -18,7 +18,7 @@ granting the user themselves an elevated set of permissions. ## Goals -1. Associate [service accounts](../design-proposals/service_accounts.md), groups, and users with +1. Associate [service accounts](service_accounts.md), groups, and users with a set of constraints that dictate how a security context is established for a pod and the pod's containers. 1. Provide the ability for users and infrastructure components to run pods with elevated privileges on behalf of another user or within a namespace where privileges are more restrictive. @@ -343,9 +343,3 @@ for a specific UID and fail early if possible. However, if the `RunAsUser` is n it should still admit the pod and allow the Kubelet to ensure that the image does not run as `root` with the existing non-root checks. - - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/auth/runas-groupid.md b/contributors/design-proposals/auth/runas-groupid.md new file mode 100644 index 00000000..b05b2950 --- /dev/null +++ b/contributors/design-proposals/auth/runas-groupid.md @@ -0,0 +1,238 @@ +# RunAsGroup Proposal + +**Author**: krmayankk@ + +**Status**: Proposal + +## Abstract + + +As a Kubernetes User, we should be able to specify both user id and group id for the containers running +inside a pod on a per Container basis, similar to how docker allows that using docker run options `-u, +--user="" Username or UID (format: <name|uid>[:<group|gid>]) format`. + +PodSecurityContext allows Kubernetes users to specify RunAsUser which can be overridden by RunAsUser +in SecurityContext on a per Container basis. There is no equivalent field for specifying the primary +Group of the running container. + +## Motivation + +Enterprise Kubernetes users want to run containers as non root. This means running containers with a +non zero user id and non zero primary group id. This gives Enterprises, confidence that their customer code +is running with least privilege and if it escapes the container boundary, will still cause least harm +by decreasing the attack surface. + +### What is the significance of Primary Group Id ? +Primary Group Id is the group id used when creating files and directories. It is also the default group +associated with a user, when he/she logins. All groups are defined in `/etc/group` file and are created +with the `groupadd` command. A Process/Container runs with uid/primary gid of the calling user. If no +primary group is specified for a user, 0(root) group is assumed. This means , any files/directories created +by a process running as user with no primary group associated with it, will be owned by group id 0(root). + +## Goals + +1. Provide the ability to specify the Primary Group id for a container inside a Pod +2. Bring launching of containers using Kubernetes at par with Dockers by supporting the same features. + + +## Use Cases + +### Use case 1: +As a Kubernetes User, I should be able to control both user id and primary group id of containers +launched using Kubernetes at runtime, so that i can run the container as non root with least possible +privilege. + +### Use case 2: +As a Kubernetes User, I should be able to control both user id and primary group id of containers +launched using Kubernetes at runtime, so that i can override the user id and primary group id specified +in the Dockerfile of the container image, without having to create a new Docker image. + +## Design + +### Model + +Introduce a new API field in SecurityContext and PodSecurityContext called `RunAsGroup`. + +#### SecurityContext + +``` +// SecurityContext holds security configuration that will be applied to a container. +// Some fields are present in both SecurityContext and PodSecurityContext. When both +// are set, the values in SecurityContext take precedence. +type SecurityContext struct { + //Other fields not shown for brevity + ..... + + // The UID to run the entrypoint of the container process. + // Defaults to user specified in image metadata if unspecified. + // May also be set in PodSecurityContext. If set in both SecurityContext and + // PodSecurityContext, the value specified in SecurityContext takes precedence. + // +optional + RunAsUser *int64 + // The GID to run the entrypoint of the container process. + // Defaults to group specified in image metadata if unspecified. + // May also be set in PodSecurityContext. If set in both SecurityContext and + // PodSecurityContext, the value specified in SecurityContext takes precedence. + // +optional + RunAsGroup *int64 + // Indicates that the container must run as a non-root user. + // If true, the Kubelet will validate the image at runtime to ensure that it + // does not run as UID 0 (root) and fail to start the container if it does. + // If unset or false, no such validation will be performed. + // May also be set in SecurityContext. If set in both SecurityContext and + // PodSecurityContext, the value specified in SecurityContext takes precedence. + // +optional + RunAsNonRoot *bool + // Indicates that the container must run as a non-root group. + // If true, the Kubelet will validate the image at runtime to ensure that it + // does not run as GID 0 (root) and fail to start the container if it does. + // If unset or false, no such validation will be performed. + // May also be set in SecurityContext. If set in both SecurityContext and + // PodSecurityContext, the value specified in SecurityContext takes precedence. + // +optional + RunAsNonRootGroup *bool + + ..... + } +``` + +#### PodSecurityContext + +``` +type PodSecurityContext struct { + //Other fields not shown for brevity + ..... + + // The UID to run the entrypoint of the container process. + // Defaults to user specified in image metadata if unspecified. + // May also be set in SecurityContext. If set in both SecurityContext and + // PodSecurityContext, the value specified in SecurityContext takes precedence + // for that container. + // +optional + RunAsUser *int64 + // The GID to run the entrypoint of the container process. + // Defaults to group specified in image metadata if unspecified. + // May also be set in PodSecurityContext. If set in both SecurityContext and + // PodSecurityContext, the value specified in SecurityContext takes precedence. + // +optional + RunAsGroup *int64 + // Indicates that the container must run as a non-root user. + // If true, the Kubelet will validate the image at runtime to ensure that it + // does not run as UID 0 (root) and fail to start the container if it does. + // If unset or false, no such validation will be performed. + // May also be set in SecurityContext. If set in both SecurityContext and + // PodSecurityContext, the value specified in SecurityContext takes precedence. + // +optional + RunAsNonRoot *bool + // Indicates that the container must run as a non-root group. + // If true, the Kubelet will validate the image at runtime to ensure that it + // does not run as GID 0 (root) and fail to start the container if it does. + // If unset or false, no such validation will be performed. + // May also be set in SecurityContext. If set in both SecurityContext and + // PodSecurityContext, the value specified in SecurityContext takes precedence. + // +optional + RunAsNonRootGroup *bool + + + ..... + } +``` + +#### PodSecurityPolicy + +PodSecurityPolicy defines strategies or conditions that a pod must run with in order to be accepted +into the system. Two of the relevant strategies are RunAsUser and SupplementalGroups. We introduce +a new strategy called RunAsGroup which will support the following options: +- MustRunAs +- MustRunAsNonRoot +- RunAsAny + +``` +// PodSecurityPolicySpec defines the policy enforced. + type PodSecurityPolicySpec struct { + //Other fields not shown for brevity + ..... + // RunAsUser is the strategy that will dictate the allowable RunAsUser values that may be set. + RunAsUser RunAsUserStrategyOptions + // SupplementalGroups is the strategy that will dictate what supplemental groups are used by the SecurityContext. + SupplementalGroups SupplementalGroupsStrategyOptions + + + // RunAsGroup is the strategy that will dictate the allowable RunAsGroup values that may be set. + RunAsGroup RunAsGroupStrategyOptions + ..... +} + +// RunAsGroupStrategyOptions defines the strategy type and any options used to create the strategy. + type RunAsUserStrategyOptions struct { + // Rule is the strategy that will dictate the allowable RunAsGroup values that may be set. + Rule RunAsGroupStrategy + // Ranges are the allowed ranges of gids that may be used. + // +optional + Ranges []GroupIDRange + } + +// RunAsGroupStrategy denotes strategy types for generating RunAsGroup values for a + // SecurityContext. + type RunAsGroupStrategy string + + const ( + // container must run as a particular gid. + RunAsGroupStrategyMustRunAs RunAsGroupStrategy = "MustRunAs" + // container must run as a non-root gid + RunAsGroupStrategyMustRunAsNonRoot RunAsGroupStrategy = "MustRunAsNonRoot" + // container may make requests for any gid. + RunAsGroupStrategyRunAsAny RunAsGroupStrategy = "RunAsAny" + ) +``` + +## Behavior + +Following points should be noted: + +- `FSGroup` and `SupplementalGroups` will continue to have their old meanings and would be untouched. +- The `RunAsGroup` In the SecurityContext will override the `RunAsGroup` in the PodSecurityContext. +- If both `RunAsUser` and `RunAsGroup` are NOT provided, the USER field in Dockerfile is used +- If both `RunAsUser` and `RunAsGroup` are specified, that is passed directly as User. +- If only one of `RunAsUser` or `RunAsGroup` is specified, the remaining value is decided by the Runtime, + where the Runtime behavior is to make it run with uid or gid as 0. +- If a non numeric Group is specified in the Dockerfile and `RunAsNonRootGroup` is set, this will be + treated as error, similar to the behavior of `RunAsNonRoot` for non numeric User in Dockerfile. + +Basically, we guarantee to set the values provided by user, and the runtime dictates the rest. + +Here is an example of what gets passed to docker User +- runAsUser set to 9999, runAsGroup set to 9999 -> Config.User set to 9999:9999 +- runAsUser set to 9999, runAsGroup unset -> Config.User set to 9999 -> docker runs you with 9999:0 +- runAsUser unset, runAsGroup set to 9999 -> Config.User set to :9999 -> docker runs you with 0:9999 +- runAsUser unset, runAsGroup unset -> Config.User set to whatever is present in Dockerfile +This is to keep the behavior backward compatible and as expected. + +## Summary of Changes needed + +At a high level, the changes classify into: +1. API +2. Validation +3. CRI +4. Runtime for Docker and rkt +5. Swagger +6. DockerShim +7. Admission +8. Registry + +- plugin/pkg/admission/security/podsecuritypolicy +- plugin/pkg/admission/securitycontext +- pkg/securitycontext/util.go +- pkg/security/podsecuritypolicy/selinux +- pkg/security/podsecuritypolicy/user +- pkg/security/podsecuritypolicy/group +- pkg/registry/extensions/podsecuritypolicy/storage +- pkg/kubelet/rkt +- pkg/kubelet/kuberuntime +- pkg/kubelet/dockershim/ +- pkg/kubelet/apis/cri/v1alpha1/runtime +- pkg/apis/extensions/validation/ +- pkg/api/validation/ +- api/swagger-spec/ +- api/openapi-spec/swagger.json + diff --git a/contributors/design-proposals/secrets.md b/contributors/design-proposals/auth/secrets.md index 29d18411..a5352444 100644 --- a/contributors/design-proposals/secrets.md +++ b/contributors/design-proposals/auth/secrets.md @@ -1,9 +1,9 @@ ## Abstract -A proposal for the distribution of [secrets](../user-guide/secrets.md) +A proposal for the distribution of [secrets](https://kubernetes.io/docs/concepts/configuration/secret/) (passwords, keys, etc) to the Kubelet and to containers inside Kubernetes using -a custom [volume](../user-guide/volumes.md#secrets) type. See the -[secrets example](../user-guide/secrets/) for more information. +a custom [volume](https://kubernetes.io/docs/concepts/storage/volumes/#secret) type. See the +[secrets example](https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets) for more information. ## Motivation @@ -622,7 +622,3 @@ on their filesystems: /etc/secret-volume/username /etc/secret-volume/password - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/security.md b/contributors/design-proposals/auth/security.md index b1aeacbd..846b2c39 100644 --- a/contributors/design-proposals/security.md +++ b/contributors/design-proposals/auth/security.md @@ -212,7 +212,3 @@ a separate component that can delete bindings but not create them). The scheduler may need read access to user or project-container information to determine preferential location (underspecified at this time). - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/security_context.md b/contributors/design-proposals/auth/security_context.md index 76bc8ee8..d7a3e458 100644 --- a/contributors/design-proposals/security_context.md +++ b/contributors/design-proposals/auth/security_context.md @@ -186,7 +186,3 @@ privileged. Contexts that attempt to define a UID or SELinux options will be denied by default. In the future the admission plugin will base this decision upon configurable policies that reside within the [service account](http://pr.k8s.io/2297). - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/service_accounts.md b/contributors/design-proposals/auth/service_accounts.md index 79902140..af72e467 100644 --- a/contributors/design-proposals/service_accounts.md +++ b/contributors/design-proposals/auth/service_accounts.md @@ -204,7 +204,3 @@ Finally, it may provide an interface to automate creation of new serviceAccounts. In that case, the user may want to GET serviceAccounts to see what has been created. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/autoscaling/OWNERS b/contributors/design-proposals/autoscaling/OWNERS new file mode 100644 index 00000000..17089492 --- /dev/null +++ b/contributors/design-proposals/autoscaling/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-autoscaling-leads +approvers: + - sig-autoscaling-leads +labels: + - sig/autoscaling diff --git a/contributors/design-proposals/horizontal-pod-autoscaler.md b/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md index 1ac9c24b..98ad92ef 100644 --- a/contributors/design-proposals/horizontal-pod-autoscaler.md +++ b/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md @@ -254,10 +254,3 @@ autoscaler to create a new pod. Discussed in issue [#3247](https://github.com/k * *[future]* **When scaling down, make more educated decision which pods to kill.** E.g.: if two or more pods from the same replication controller are on the same node, kill one of them. Discussed in issue [#4301](https://github.com/kubernetes/kubernetes/issues/4301). - - - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/hpa-status-conditions.md b/contributors/design-proposals/autoscaling/hpa-status-conditions.md index efb5ad0b..d3354582 100644 --- a/contributors/design-proposals/hpa-status-conditions.md +++ b/contributors/design-proposals/autoscaling/hpa-status-conditions.md @@ -2,7 +2,7 @@ Horizontal Pod Autoscaler Status Conditions =========================================== Currently, the HPA status conveys the last scale time, current and desired -replacas, and the last-retrieved values of the metrics used to autoscale. +replicas, and the last-retrieved values of the metrics used to autoscale. However, the status field conveys no information about whether or not the HPA controller encountered difficulties while attempting to fetch metrics, @@ -77,7 +77,7 @@ entirely. - *FailedRescale*: a scale update was needed and the HPA controller was unable to actually update the scale subresource of the target scalable. -- *SuccesfulRescale*: a scale update was needed and everything went +- *SuccessfulRescale*: a scale update was needed and everything went properly. - *FailedUpdateStatus*: the HPA controller failed to update the status of diff --git a/contributors/design-proposals/hpa-v2.md b/contributors/design-proposals/autoscaling/hpa-v2.md index 9f814ef5..fba4364c 100644 --- a/contributors/design-proposals/hpa-v2.md +++ b/contributors/design-proposals/autoscaling/hpa-v2.md @@ -280,12 +280,8 @@ Mechanical Concerns The HPA will derive metrics from two sources: resource metrics (i.e. CPU request percentage) will come from the -[master metrics API](resource-metrics-api.md), while other metrics will -come from the [custom metrics API](custom-metrics-api.md), which is +[master metrics API](../instrumentation/resource-metrics-api.md), while other metrics will +come from the [custom metrics API](../instrumentation/custom-metrics-api.md), which is an adapter API which sources metrics directly from the monitoring pipeline. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/autoscaling/images/vpa-architecture.png b/contributors/design-proposals/autoscaling/images/vpa-architecture.png Binary files differnew file mode 100644 index 00000000..c8af3073 --- /dev/null +++ b/contributors/design-proposals/autoscaling/images/vpa-architecture.png diff --git a/contributors/design-proposals/initial-resources.md b/contributors/design-proposals/autoscaling/initial-resources.md index 432f6021..7ce09770 100644 --- a/contributors/design-proposals/initial-resources.md +++ b/contributors/design-proposals/autoscaling/initial-resources.md @@ -5,7 +5,7 @@ and set them before the container is run. This document describes design of the ## Motivation -Since we want to make Kubernetes as simple as possible for its users we don't want to require setting [Resources](../design/resource-qos.md) for container by its owner. +Since we want to make Kubernetes as simple as possible for its users we don't want to require setting [Resources](../node/resource-qos.md) for container by its owner. On the other hand having Resources filled is critical for scheduling decisions. Current solution to set up Resources to hardcoded value has obvious drawbacks. We need to implement a component which will set initial Resources to a reasonable value. @@ -18,7 +18,7 @@ For every container without Resources specified it will try to predict amount of So that a pod without specified resources will be treated as . -InitialResources will set only [request](../design/resource-qos.md#requests-and-limits) (independently for each resource type: cpu, memory) field in the first version to avoid killing containers due to OOM (however the container still may be killed if exceeds requested resources). +InitialResources will set only [request](../node/resource-qos.md#requests-and-limits) (independently for each resource type: cpu, memory) field in the first version to avoid killing containers due to OOM (however the container still may be killed if exceeds requested resources). To make the component work with LimitRanger the estimated value will be capped by min and max possible values if defined. It will prevent from situation when the pod is rejected due to too low or too high estimation. @@ -70,6 +70,3 @@ and should be introduced shortly after the first version is done: * add estimation as annotations for those containers that already has resources set * support for other data sources like [Hawkular](http://www.hawkular.org/) -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/autoscaling/vertical-pod-autoscaler.md b/contributors/design-proposals/autoscaling/vertical-pod-autoscaler.md new file mode 100644 index 00000000..b4b08704 --- /dev/null +++ b/contributors/design-proposals/autoscaling/vertical-pod-autoscaler.md @@ -0,0 +1,730 @@ +Vertical Pod Autoscaler +======================= +**Authors:** kgrygiel, mwielgus +**Contributors:** DirectXMan12, fgrzadkowski, jszczepkowski, smarterclayton + +Vertical Pod Autoscaler +([#10782](https://github.com/kubernetes/kubernetes/issues/10782)), +later referred to as VPA (aka. "rightsizing" or "autopilot") is an +infrastructure service that automatically sets resource requirements of Pods +and dynamically adjusts them in runtime, based on analysis of historical +resource utilization, amount of resources available in the cluster and real-time +events, such as OOMs. + +- [Introduction](#introduction) + - [Background](#background) + - [Purpose](#purpose) + - [Related features](#related-features) +- [Requirements](#requirements) + - [Functional](#functional) + - [Availability](#availability) + - [Extensibility](#extensibility) +- [Design](#design) + - [Overview](#overview) + - [Architecture overview](#architecture-overview) + - [API](#api) + - [Admission Controller](#admission-controller) + - [Recommender](#recommender) + - [Updater](#updater) + - [Recommendation model](#recommendation-model) + - [History Storage](#history-storage) + - [Open questions](#open-questions) +- [Future work](#future-work) + - [Pods that require VPA to start](#pods-that-require-vpa-to-start) + - [Combining vertical and horizontal scaling](#combining-vertical-and-horizontal-scaling) + - [Batch workloads](#batch-workloads) +- [Alternatives considered](#alternatives-considered) + - [Pods point at VPA](#pods-point-at-vpa) + - [VPA points at Deployment](#vpa-points-at-deployment) + - [Actuation using the Deployment update mechanism](#actuation-using-the-deployment-update-mechanism) + +------------ +Introduction +------------ + +### Background ### +* [Compute resources](https://kubernetes.io/docs/user-guide/compute-resources/) +* [Resource QoS](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-qos.md) +* [Admission Controllers](https://kubernetes.io/docs/admin/admission-controllers/) +* [External Admission Webhooks](https://kubernetes.io/docs/admin/extensible-admission-controllers/#external-admission-webhooks) + +### Purpose ### +Vertical scaling has two objectives: + +1. Reducing the maintenance cost, by automating configuration of resource +requirements. + +2. Improving utilization of cluster resources, while minimizing the risk of containers running out of memory or getting CPU starved. + +### Related features ### +#### Horizontal Pod Autoscaler #### +["Horizontal Pod Autoscaler"](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) +(often abbreviated to HPA) is an infrastructure service that dynamically adjusts +the number of Pods in a replication controller based on realtime analysis of CPU +utilization or other, user specified signals. +Usually the user will choose horizontal scaling for stateless workloads and +vertical scaling for stateful. In some cases both solutions could be combined +([see more](#combining-vertical-and-horizontal-scaling)). + +#### Cluster Autoscaler #### +["Cluster Autoscaler"](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) +is a tool that automatically adjusts the size of the Kubernetes cluster based on +the overall cluster utilization. +Cluster Autoscaler and Pod Autoscalers (vertical or horizontal) are +complementary features. Combined together they provide a fully automatic scaling +solution. + +#### Initial resources #### +["Initial Resources"](https://github.com/kgrygiel/community/blob/master/contributors/design-proposals/initial-resources.md) +is a very preliminary, proof-of-concept feature providing initial request based +on historical utilization. It is designed to only kick in on Pod creation. +VPA is intended to supersede this feature. + +#### In-place updates #### +In-place Pod updates ([#5774] +(https://github.com/kubernetes/kubernetes/issues/5774)) is a planned feature to +allow changing resources (request/limit) of existing containers without killing them, assuming sufficient free resources available on the node. +Vertical Pod Autoscaler will greatly benefit from this ability, however it is +not considered a blocker for the MVP. + +#### Resource estimation #### +Resource estimation is another planned feature, meant to improve node resource +utilization by temporarily reclaiming unused resources of running containers. +It is different from Vertical Autoscaling in that it operates on a shorter +timeframe (using only local, short-term history), re-offers resources at a +lower quality, and does not provide initial resource predictions. +VPA and resource estimation are complementary. Details will follow once +Resource Estimation is designed. + +------------ +Requirements +------------ + +### Functional ### + +1. VPA is capable of setting container resources (CPU & memory request/limit) at + Pod submission time. + +2. VPA is capable of adjusting container resources of existing Pods, in + particular reacting to CPU starvation and container OOM events. + +3. When VPA restarts Pods, it respects the disruption budget. + +4. It is possible for the user to configure VPA with fixed constraints on + resources, specifically: min & max request. + +5. VPA is compatible with Pod controllers, at least with Deployments. + In particular: + * Updates of resources do not interfere/conflict with spec updates. + * It is possible to do a rolling update of the VPA policy (e.g. min resources) + on an existing Deployment. + +6. It is possible to create Pod(s) that start following the VPA policy + immediately. In particular such Pods must not be scheduled until VPA policy + is applied. + +7. Disabling VPA is easy and fast ("panic button"), without disrupting existing + Pods. + +### Availability ### +1. Downtime of heavy-weight components (database/recommender) must not block + recreating existing Pods. Components on critical path for Pod creation + (admission controller) are designed to be highly available. + +### Extensibility ### +1. VPA is capable of performing in-place updates once they become available. + +------ +Design +------ + +### Overview ### +(see further sections for details and justification) + +1. We introduce a new type of **API resource**: + `VerticalPodAutoscaler`. It consists of a **label selector** to match Pods, + the **resources policy** (controls how VPA computes the resources), the + **update policy** (controls how changes are applied to Pods) and the + recommended Pod resources (an output field). + +2. **VPA Recommender** is a new component which **consumes utilization signals + and OOM events** for all Pods in the cluster from the + [Metrics Server](https://github.com/kubernetes-incubator/metrics-server). + +3. VPA Recommender **watches all Pods**, keeps calculating fresh recommended + resources for them and **stores the recommendations in the VPA objects**. + +4. Additionally the Recommender **exposes a synchronous API** that takes a Pod + description and returns recommended resources. + +5. All Pod creation requests go through the VPA **Admission Controller**. + If the Pod is matched by any VerticalPodAutoscaler object, the admission + controller **overrides resources** of containers in the Pod with the + recommendation provided by the VPA Recommender. If the Recommender is not + available, it falls back to the recommendation cached in the VPA object. + +6. **VPA Updater** is a component responsible for **real-time updates** of Pods. + If a Pod uses VPA in `"Auto"` mode, the Updater can decide to update it with + recommender resources. + In MVP this is realized by just evicting the Pod in order to have it + recreated with new resources. This approach requires the Pod to belong to a + Replica Set (or some other owner capable of recreating it). + In future the Updater will take advantage of in-place updates, which would + most likely lift this constraint. + Because restarting/rescheduling Pods is disruptive to the service, it must be + rare. + +7. VPA only controls the resource **request** of containers. It sets the limit + to infinity. The request is calculated based on analysis of the current and + previous runs (see [Recommendation model](#recommendation-model) below). + +8. **History Storage** is a component that consumes utilization signals and OOMs + (same data as the Recommender) from the API Server and stores it persistently. + It is used by the Recommender to **initialize its state on startup**. + It can be backed by an arbitrary database. The first implementation will use + [Prometheus](https://github.com/kubernetes/charts/tree/master/stable/prometheus), + at least for the resource utilization part. + +### Architecture overview ### + + +### API ### +We introduce a new type of API object `VerticalPodAutoscaler`, which +consists of the Target, that is a [label selector](https://kubernetes.io/docs/api-reference/v1.5/#labelselector-unversioned) +for matching Pods and two policy sections: the update policy and the resources +policy. +Additionally it holds the most recent recommendation computed by VPA. + +#### VPA API object overview #### +```go +// VerticalPodAutoscalerSpec is the specification of the behavior of the autoscaler. +type VerticalPodAutoscalerSpec { + // A label query that determines the set of pods controlled by the Autoscaler. + // More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors + Selector *metav1.LabelSelector + + // Describes the rules on how changes are applied to the pods. + // +optional + UpdatePolicy PodUpdatePolicy + + // Controls how the autoscaler computes recommended resources. + // +optional + ResourcePolicy PodResourcePolicy +} + +// VerticalPodAutoscalerStatus describes the runtime state of the autoscaler. +type VerticalPodAutoscalerStatus { + // The time when the status was last refreshed. + LastUpdateTime metav1.Time + // The most recently computed amount of resources recommended by the + // autoscaler for the controlled pods. + // +optional + Recommendation RecommendedPodResources + // A free-form human readable message describing the status of the autoscaler. + StatusMessage string +} +``` + +The complete API definition is included [below](#complete_vpa_api_object_definition). + +#### Label Selector #### +The label selector determines which Pods will be scaled according to the given +VPA policy. The Recommender will aggregate signals for all Pods matched by a +given VPA, so it is important that the user set labels to group similarly +behaving Pods under one VPA. + +It is yet to be determined how to resolve conflicts, i.e. when the Pod is +matched by more than one VPA (this is not a VPA-specific problem though). + +#### Update Policy #### +The update policy controls how VPA applies changes. In MVP it consists of a +single field `mode` that enables the feature. + +```json +"updatePolicy" { + "mode": "", +} +``` + +Mode can be set to one of the following: + +1. `"Initial"`: VPA only assigns resources on Pod creation and does not + change them during lifetime of the Pod. +2. `"Auto"` (default): VPA assigns resources on Pod creation and + additionally can update them during lifetime of the Pod, including evicting / + rescheduling the Pod. +3. `"Off"`: VPA never changes Pod resources. The recommender still sets the + recommended resources in the VPA object. This can be used for a “dry run”. + +To disable VPA updates the user can do any of the following: (1) change the +updatePolicy to `"Off"` or (2) delete the VPA or (3) change the Pod labels to no +longer match the VPA selector. + +Note: disabling VPA prevents it from doing further changes, but does not revert +resources of the running Pods, until they are updated. +For example, when running a Deployment, the user would need to perform an update +to revert Pod to originally specified resources. + +#### Resource Policy #### +The resources policy controls how VPA computes the recommended resources. +In MVP it consists of (optional) lower and upper bound on the request of each +container. +The resources policy could later be extended with additional knobs to let the +user tune the recommendation algorithm to their specific use-case. + +#### Recommendation #### +The VPA resource has an output-only field keeping a recent recommendation, +filled by the Recommender. This field can be used to obtain a recent +recommendation even during a temporary unavailability of the Recommender. +The recommendation consists of the recommended target amount of resources as +well as an range (min..max), which can be used by the Updater to make decisions +on when to update the pod. +In the case of a resource crunch the Updater may decide to squeeze pod resources +towards the recommended minimum. +The width of the (min..max) range also reflects the confidence of a +recommendation. For example, for a workload with a very spiky usage it is much +harder to determine the optimal balance between performance and resource +utilization, compared to a workload with stable usage. + +#### Complete VPA API object definition #### + +```go +// VerticalPodAutoscaler is the configuration for a vertical pod +// autoscaler, which automatically manages pod resources based on historical and +// real time resource utilization. +type VerticalPodAutoscaler struct { + metav1.TypeMeta + // Standard object metadata. + // More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata + // +optional + metav1.ObjectMeta + + // Specification of the behavior of the autoscaler. + // More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status. + // +optional + Spec VerticalPodAutoscalerSpec + + // Current information about the autoscaler. + // +optional + Status VerticalPodAutoscalerStatus +} + +// VerticalPodAutoscalerSpec is the specification of the behavior of the autoscaler. +type VerticalPodAutoscalerSpec { + // A label query that determines the set of pods controlled by the Autoscaler. + // More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors + Selector *metav1.LabelSelector + + // Describes the rules on how changes are applied to the pods. + // +optional + UpdatePolicy PodUpdatePolicy + + // Controls how the autoscaler computes recommended resources. + // +optional + ResourcePolicy PodResourcePolicy +} + +// VerticalPodAutoscalerStatus describes the runtime state of the autoscaler. +type VerticalPodAutoscalerStatus { + // The time when the status was last refreshed. + LastUpdateTime metav1.Time + // The most recently computed amount of resources recommended by the + // autoscaler for the controlled pods. + // +optional + Recommendation RecommendedPodResources + // A free-form human readable message describing the status of the autoscaler. + StatusMessage string +} + +// UpdateMode controls when autoscaler applies changes to the pod resoures. +type UpdateMode string +const ( + // UpdateModeOff means that autoscaler never changes Pod resources. + // The recommender still sets the recommended resources in the + // VerticalPodAutoscaler object. This can be used for a "dry run". + UpdateModeOff UpdateMode = "Off" + // UpdateModeInitial means that autoscaler only assigns resources on pod + // creation and does not change them during the lifetime of the pod. + UpdateModeInitial UpdateMode = "Initial" + // UpdateModeAuto means that autoscaler assigns resources on pod creation + // and additionally can update them during the lifetime of the pod, + // including evicting / rescheduling the pod. + UpdateModeAuto UpdateMode = "Auto" +) + +// PodUpdatePolicy describes the rules on how changes are applied to the pods. +type PodUpdatePolicy struct { + // Controls when autoscaler applies changes to the pod resoures. + // +optional + UpdateMode UpdateMode +} + +const ( + // DefaultContainerResourcePolicy can be passed as + // ContainerResourcePolicy.Name to specify the default policy. + DefaultContainerResourcePolicy = "*" +) +// ContainerResourcePolicy controls how autoscaler computes the recommended +// resources for a specific container. +type ContainerResourcePolicy struct { + // Name of the container or DefaultContainerResourcePolicy, in which + // case the policy is used by the containers that don't have their own + // policy specified. + Name string + // Whether autoscaler is enabled for the container. Defaults to "On". + // +optional + Mode ContainerScalingMode + // Specifies the minimal amount of resources that will be recommended + // for the container. + // +optional + MinAllowed api.ResourceRequirements + // Specifies the maximum amount of resources that will be recommended + // for the container. + // +optional + MaxAllowed api.ResourceRequirements +} + +// PodResourcePolicy controls how autoscaler computes the recommended resources +// for containers belonging to the pod. +type PodResourcePolicy struct { + // Per-container resource policies. + ContainerPolicies []ContainerResourcePolicy +} + +// ContainerScalingMode controls whether autoscaler is enabled for a speciifc +// container. +type ContainerScalingMode string +const ( + // ContainerScalingModeOn means autoscaling is enabled for a container. + ContainerScalingModeOn ContainerScalingMode = "On" + // ContainerScalingModeOff means autoscaling is disabled for a container. + ContainerScalingModeOff ContainerScalingMode = "Off" +) + +// RecommendedPodResources is the recommendation of resources computed by +// autoscaler. +type RecommendedPodResources struct { + // Resources recommended by the autoscaler for each container. + ContainerRecommendations []RecommendedContainerResources +} + +// RecommendedContainerResources is the recommendation of resources computed by +// autoscaler for a specific container. Respects the container resource policy +// if present in the spec. +type RecommendedContainerResources struct { + // Name of the container. + Name string + // Recommended amount of resources. + Target api.ResourceRequirements + // Minimum recommended amount of resources. + // Running the application with less resources is likely to have + // significant impact on performance/availability. + // +optional + MinRecommended api.ResourceRequirements + // Maximum recommended amount of resources. + // Any resources allocated beyond this value are likely wasted. + // +optional + MaxRecommended api.ResourceRequirements +} +``` + +### Admission Controller ### + +VPA Admission Controller intercepts Pod creation requests. If the Pod is matched +by a VPA config with mode not set to “off”, the controller rewrites the request +by applying recommended resources to the Pod spec. Otherwise it leaves the Pod +spec unchanged. + +The controller gets the recommended resources by fetching +/recommendedPodResources from the Recommender. If the call times out or fails, +the controller falls back to the recommendation cached in the VPA object. +If this is also not available the controller lets the request pass-through +with originally specified resources. + +Note: in future it will be possible to (optionally) enforce using VPA by marking +the Pod as "requiring VPA". This will disallow scheduling the Pod before a +corresponding VPA config is created. The Admission Controller will reject such +Pods if it finds no matching VPA config. This ability will be convenient for the +user who wants to create the VPA config together with submitting the Pod. + +The VPA Admission Controller will be implemented as an +[External Admission Hook](https://kubernetes.io/docs/admin/extensible-admission-controllers/#external-admission-webhooks). +Note however that this depends on the proposed feature to allow +[mutating webhook admission controllers](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/admission_control_extension.md#future-work). + +### Recommender ### +Recommender is the main component of the VPA. It is responsible for +computing recommended resources. On startup the recommender fetches +historical resource utilization of all Pods (regardless of whether +they use VPA) together with the history of Pod OOM events from the +History Storage. It aggregates this data and keeps it in memory. + +During normal operation the recommender consumes real time updates of +resource utilization and new events via the Metrics API from +the [Metrics Server](https://github.com/kubernetes-incubator/metrics-server). +Additionally it watches all Pods and all VPA objects in the +cluster. For every Pod that is matched by some VPA selector the +Recommender computes the recommended resources and sets the +recommendation on the VPA object. + +It is important to realize that one VPA object has one recommendation. +The user is expected to use one VPA to control Pods with similar +resource usage patterns, typically a group of replicas or shards of +a single workload. + +The Recommender acts as an +[extension-apiserver](https://kubernetes.io/docs/concepts/api-extension/apiserver-aggregation/), +exposing a synchronous method that takes a Pod Spec and the Pod metadata +and returns recommended resources. + +#### Recommender API #### + +```POST /recommendationQuery``` + +Request body: +```go +// RecommendationQuery obtains resource recommendation for a pod. +type RecommendationQuery struct { + metav1.TypeMeta + // +optional + metav1.ObjectMeta + + // Spec is filled in by the caller to request a recommendation. + Spec RecommendationQuerySpec + + // Status is filled in by the server with the recommended pod resources. + // +optional + Status RecommendationQueryStatus +} + +// RecommendationQuerySpec is a request of recommendation for a pod. +type RecommendationQuerySpec struct { + // Pod for which to compute the recommendation. Does not need to exist. + Pod core.Pod +} + +// RecommendationQueryStatus is a response to the recommendation request. +type RecommendationQueryStatus { + // Recommendation holds recommended resources for the pod. + // +optional + Recommendation autoscaler.RecommendedPodResources + // Error indicates that the recommendation was not available. Either + // Recommendation or Error must be present. + // +optional + Error string +} +``` + +Notice that this API method may be called for an existing Pod, as well as for a +yet-to-be-created Pod. + +### Updater ### +VPA Updater is a component responsible for applying recommended resources to +existing Pods. +It monitors all VPA objects and Pods in the cluster, periodically fetching +recommendations for the Pods that are controlled by VPA by calling the +Recommender API. +When recommended resources significantly diverge from actually configured +resources, the Updater may decide to update a Pod. +In MVP (until in-place updates of Pod resources are available) +this means evicting Pods in order to have them recreated with the recommended +resources. + +The Updater relies on other mechanisms (such as Replica Set) to recreate a +deleted Pod. However it does not verify whether such mechanism is actually +configured for the Pod. Such checks could be implemented in the CLI and warn +the user when the VPA would match Pods, that are not automatically restarted. + +While terminating Pods is disruptive and generally undesired, it is sometimes +justified in order to (1) avoid CPU starvation (2) reduce the risk of correlated +OOMs across multiple Pods at random time or (3) save resources over long periods +of time. + +Apart from its own policy on how often a Pod can be evicted, the Updater also +respects the Pod disruption budget, by using Eviction API to evict Pods. + +The Updater only touches pods that point to a VPA with updatePolicy.mode set +to `"Auto"`. + +The Updater will also need to understand how to adjust the recommendation before +applying it to a Pod, based on the current state of the cluster (e.g. quota, +space available on nodes or other scheduling constraints). +Otherwise it may deschedule a Pod permanently. This mechanism is not yet +designed. + +### Recommendation model ### + +VPA controls the request (memory and CPU) of containers. In MVP it always sets +the limit to infinity. It is not yet clear whether there is a use-case for VPA +setting the limit. + +The request is calculated based on analysis of the current and revious runs of +the container and other containers with similar properties (name, image, +command, args). +The recommendation model (MVP) assumes that the memory and CPU consumption are +independent random variables with distribution equal to the one observed in the +last N days (recommended value is N=8 to capture weekly peaks). +A more advanced model in future could attempt to detect trends, periodicity and +other time-related patterns. + +For CPU the objective is to **keep the fraction of time when the container usage +exceeds a high percentage (e.g. 95%) of request below a certain threshold** +(e.g. 1% of time). +In this model the "CPU usage" is defined as mean usage measured over a short +interval. The shorter the measurement interval, the better the quality of +recommendations for spiky, latency sensitive workloads. Minimum reasonable +resolution is 1/min, recommended is 1/sec. + +For memory the objective is to **keep the probability of the container usage +exceeding the request in a specific time window below a certain threshold** +(e.g. below 1% in 24h). The window must be long (≥ 24h) to ensure that evictions +caused by OOM do not visibly affect (a) availability of serving applications +(b) progress of batch computations (a more advanced model could allow user to +specify SLO to control this). + +#### Handling OOMs #### +When a container is evicted due to exceeding available memory, its actual memory +requirements are not known (the amount consumed obviously gives the lower +bound). This is modelled by translating OOM events to artificial memory usage +samples by applying a "safety margin" multiplier to the last observed usage. + +### History Storage ### +VPA defines data access API for providers of historical events and resource +utilization. Initially we will use Prometheus as the reference implementation of +this API, at least for the resource utilization part. The historical events +could be backed by another solution, e.g. +[Infrastore](https://github.com/kubernetes/kubernetes/issues/44095). +Users will be able to plug their own implementations. + +History Storage is populated with real time updates of resources utilization and +events, similarly to the Recommender. The storage keeps at least 8 days of data. +This data is only used to initialize the Recommender on startup. + +### Open questions ### +1. How to resolve conflicts if multiple VPA objects match a Pod. + +2. How to adjust the recommendation before applying it to a specific pod, + based on the current state of the cluster (e.g. quota, space available on + nodes or other scheduling constraints). + +----------- +Future work +----------- + +### Pods that require VPA to start ### +In the current proposal the Pod will be scheduled with originally configured +resources if no matching VPA config is present at the Pod admission time. +This may be undesired behavior. In particular the user may want to create the +VPA config together with submitting the Pod, which leads to a race condition: +the outcome depends on which resource (VPA or the Pod) is processed first. + +In order to address this problem we propose to allow marking Pods with a special +annotation ("requires VPA") that prevents the Admission Controller from allowing +the Pod if a corresponding VPA is not available. + +An alternative would be to introduce a VPA Initializer serving the same purpose. + +### Combining vertical and horizontal scaling ### +In principle it may be possible to use both vertical and horizontal scaling for +a single workload (group of Pods), as long as the two mechanisms operate on +different resources. +The right approach is to let the +[Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) +scale the group based on the _bottleneck_ resource. The Vertical Pod Autoscaler +could then control other resources. Examples: + +1. A CPU-bound workload can be scaled horizontally based on the CPU utilization +while using vertical scaling to adjust memory. + +2. An IO-bound workload can be scaled horizontally based on the IO throughput +while using vertical scaling to adjust both memory and CPU. + +However this is a more advanced form of autoscaling and it is not well supported +by the MVP version of Vertical Pod Autoscaler. The difficulty comes from the +fact that changing the number of instances affects not only the utilization of +the bottleneck resource (which is the principle of horizontal scaling) but +potentially also non-bottleneck resources that are controlled by VPA. +The VPA model will have to be extended to take the size of the group into account +when aggregating the historical resource utilization and when producing a +recommendation, in order to allow combining it with HPA. + +### Batch workloads ### +Batch workloads have different CPU requirements than latency sensitive +workloads. Instead of latency they care about throughput, which means VPA should +base the CPU requirements on average CPU consumption rather than high +percentiles of CPU distribution. + +TODO: describe the recommendation model for the batch workloads and how VPA will +distinguish between batch and serving. A possible approach is to look at +`PodSpec.restartPolicy`. +An alternative would be to let the user specify the latency requirements of the +workload in the `PodResourcePolicy`. + +----------------------- +Alternatives considered +----------------------- + +### Pods point at VPA ### +*REJECTED BECAUSE IT REQUIRES MODIFYING THE POD SPEC* + +#### proposal: #### +Instead of VPA using label selectors, Pod Spec is extended with an optional +field `verticalPodAutoscalerPolicy`, +a [reference](https://kubernetes.io/docs/api-reference/v1/definitions/#_v1_localobjectreference) +to the VPA config. + +#### pros: #### +* Consistency is enforced at the API level: + * At most one VPA can point to a given Pod. + * It is always clear at admission stage whether the Pod should use + VPA or not. No race conditions. +* It is cheap to find the VPA for a given Pod. + +#### cons: #### +* Requires changing the core part of the API (Pod Spec). + +### VPA points at Deployment ### + +#### proposal: #### +VPA has a reference to Deployment object. Doesn’t use label selector to match +Pods. + +#### pros: #### +* More consistent with HPA. + +#### cons: #### +* Extending VPA support from Deployment to other abstractions that manage Pods + requires additional work. VPA must be aware of all such abstractions. +* It is not possible to do a rolling update of the VPA config. + For example setting `max_memory` in the VPA config will apply to the whole + Deployment immediately. +* VPA can’t be shared between deployments. + +### Actuation using the Deployment update mechanism ### + +In this solution the Deployment itself is responsible for actuating VPA +decisions. + +#### Actuation by update of spec #### +In this variant changes of resources are applied similarly to normal changes of +the spec, i.e. using the Deployment rolling update mechanism. + +**pros:** existing clean API (and implementation), one common update policy +(e.g. max surge, max unavailable). + +**cons:** conflicting with user (config) update - update of resources and spec +are tied together (they are executed at the same rate), problem with rollbacks, +problem with pause. Not clear how to handle in-place updates? (this problem has +to be solved regardless of VPA though). + +#### Dedicated method for resource update #### +In this variant Deployment still uses the rolling update mechanism for updating +resources, but update of resources is treated in a special way, so that it can +be performed in parallel with config update. + +**pros:** handles concurrent resources and spec updates, solves resource updates +without VPA, more consistent with HPA, all update logic lives in one place (less +error-prone). + +**cons:** specific to Deployment, high complexity (multiple replica set created +underneath - exposed to the user, can be confusing and error-prone). diff --git a/contributors/design-proposals/aws/OWNERS b/contributors/design-proposals/aws/OWNERS new file mode 100644 index 00000000..83317bbe --- /dev/null +++ b/contributors/design-proposals/aws/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-aws-leads +approvers: + - sig-aws-leads +labels: + - sig/aws diff --git a/contributors/design-proposals/aws_under_the_hood.md b/contributors/design-proposals/aws/aws_under_the_hood.md index 13ad435e..ec8b0740 100644 --- a/contributors/design-proposals/aws_under_the_hood.md +++ b/contributors/design-proposals/aws/aws_under_the_hood.md @@ -47,13 +47,13 @@ kube-up. ### Storage -AWS supports persistent volumes by using [Elastic Block Store (EBS)](../user-guide/volumes.md#awselasticblockstore). +AWS supports persistent volumes by using [Elastic Block Store (EBS)](https://kubernetes.io/docs/concepts/storage/volumes/#awselasticblockstore). These can then be attached to pods that should store persistent data (e.g. if you're running a database). By default, nodes in AWS use [instance storage](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html) unless you create pods with persistent volumes -[(EBS)](../user-guide/volumes.md#awselasticblockstore). In general, Kubernetes +[(EBS)](https://kubernetes.io/docs/concepts/storage/volumes/#awselasticblockstore). In general, Kubernetes containers do not have persistent storage unless you attach a persistent volume, and so nodes on AWS use instance storage. Instance storage is cheaper, often faster, and historically more reliable. Unless you can make do with @@ -112,8 +112,8 @@ ELB has some restrictions: on ELB annotations for pods speaking HTTP). To work with these restrictions, in Kubernetes, [LoadBalancer -services](../user-guide/services.md#type-loadbalancer) are exposed as -[NodePort services](../user-guide/services.md#type-nodeport). Then +services](https://kubernetes.io/docs/concepts/services-networking/service/#type-loadbalancer) are exposed as +[NodePort services](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport). Then kube-proxy listens externally on the cluster-wide port that's assigned to NodePort services and forwards traffic to the corresponding pods. @@ -303,8 +303,3 @@ These scripts are responsible for mounting and formatting volumes, downloading Salt and Kubernetes from the S3 bucket, and then triggering Salt to actually install Kubernetes. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/cli/OWNERS b/contributors/design-proposals/cli/OWNERS new file mode 100644 index 00000000..248d3e7c --- /dev/null +++ b/contributors/design-proposals/cli/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-cli-leads +approvers: + - sig-cli-leads +labels: + - sig/cli diff --git a/contributors/design-proposals/cli/apply_refactor.md b/contributors/design-proposals/cli/apply_refactor.md new file mode 100644 index 00000000..8f9ca6da --- /dev/null +++ b/contributors/design-proposals/cli/apply_refactor.md @@ -0,0 +1,184 @@ +# Apply v2 + +## Background + +`kubectl apply` reads a file or set of files, and updates the cluster state based off the file contents. +It does a couple things: + +1. Create / Update / (Delete) the live resources based on the file contents +2. Update currently and previously configured fields, without clobbering fields set by other means, + such as imperative kubectl commands, other deployment and management tools, admission controllers, + initializers, horizontal and vertical autoscalers, operators, and other controllers. + +Essential complexity in the apply code comes from supporting custom strategies for +merging fields built into the object schema - +such as merging lists together based on a field `key` and deleting individual +items from a list by `key`. + +Accidental complexity in the apply code comes from the structure growing organically in ways that have +broken encapsulation and separation of concerns. This has lead to maintenance challenges as +keeping ordering for items in a list, and correctly merging lists of primitives (`key`less). + +Round tripping changes through PATCHes introduces additional accidental complexity, +as they require imperative directives that are not part of the object schema. + +## Objective + + +Reduce maintenance burden by minimizing accidental complexity in the apply codebase. + +This should help: + +- Simplify introducing new merge semantics +- Simplify enabling / disabling new logic with flags + +## Changes + +Implementation of proposed changes under review in PR [52349](https://github.com/kubernetes/kubernetes/pull/52349) + +### Use read-update instead of patch + +#### Why + +Building a PATCH from diff creates additional code complexity vs directly updating the object. + +- Need to generate imperative delete directives instead of simply deleting an item from a list. +- Using PATCH semantics and directives is less well known and understood by most users + than using the object schema itself. This makes it harder for non-experts to maintain the codebase. +- Using PATCH semantics is more work to implement a diff of the changes as + PATCH must be separately merged on the remote object for to display the diff. + +#### New approach + +1. Read the live object +2. Compare the live object to last-applied and local files +3. Update the fields on the live object that was read +4. Send a PUT to update the modified object +5. If encountering optimistic lock failure, retry back to 1. + +### Restructure code into modular components + +In the current implementation of apply - parsing and traversing the object trees, diffing the +contents and generating the patch are entangled. This creates maintenance and +testing challenges. We should instead encapsulate discrete responsibilities in separate packages - +such as collating the object values and updating the target object. + +#### Phase 1: Parse last-applied, local, live objects and collate + +Provide a structure that contains the last, local and live value for each field. This +will make it easy to walk the a single tree when making decisions about how to update the object. +Decisions about ordering of lists or parsing metadata for fields are made here. + +#### Phase 2: Diff and update objects + +Use the visitor pattern to encapsulate how to update each field type for each merge strategy. +Unit test each visit function. Decisions about how to replace, merge, or delete a field or +list item are made here. + +## Notable items + +- Merge will use openapi to get the schema from the server +- Merge can be run either on the server side or the client side +- Merge can handle 2-way or 3-way merges of objects (initially will not support PATCH directives) + +## Out of scope of this doc + +In order to make apply sufficiently maintainable and extensible to new API types, as well as to make its +behavior more intuitive for users, the merge behavior, including how it is specified in the API schema, +must be systematically redesigned and more thoroughly tested. + +Examples of issues that need to be resolved + +- schema metadata `patchStrategy` and `mergeKey` are implicit, unversioned and incorrect in some cases. + to fix the incorrect metadata, the metadata must be versioned so PATCHes generated will old metadata continue + to be merged by the server in the manner they were intended + - need to version all schema metadata for each objects and provide this are part of the request + - e.g. container port [39188](https://github.com/kubernetes/kubernetes/issues/39188) +- no semantic way to represent union fields [35345](https://github.com/kubernetes/kubernetes/issues/35345) + + +## Detailed analysis of structure and impact today + +The following PRs constitute the focus of ~6 months of engineering work. Each of the PRs is very complex +for the work what it is solving. + +### Patterns observed + +- PRs frequently closed or deferred because maintainers / reviewers cannot reason about the impact or + correctness of the changes +- Relatively simple changes + - are 200+ lines of code + - modify dozens of existing locations in the code + - are spread across 1000+ lines of existing code +- Changes that add new directives require updates in multiple locations - create patch + apply patch + +### PRs + +[38665](https://github.com/kubernetes/kubernetes/pull/38665/files) +- Support deletion of primitives from lists +- Lines (non-test): ~200 +- ~6 weeks +[44597](https://github.com/kubernetes/kubernetes/pull/44597/files) +- Support deleting fields not listed in the patch +- Lines (non-test): ~250 +- ~6 weeks +[45980](https://github.com/kubernetes/kubernetes/pull/45980/files#diff-101008d96c4444a5813f7cb6b54aaff6) +- Keep ordering of items when merging lists +- Lines (non-test): ~650 +[46161](https://github.com/kubernetes/kubernetes/pull/46161/files#diff-101008d96c4444a5813f7cb6b54aaff6) +- Support using multiple fields for a merge key +- Status: Deferred indefinitely - too hard for maintainers to understand impact and correctness of changes +[46560](https://github.com/kubernetes/kubernetes/pull/46560/files) +- Support diff apply (1st attempt) +- Status: Closed - too hard for maintainers to understand impact and correctness of changes +[49174](https://github.com/kubernetes/kubernetes/pull/49174/files) +- Support diff apply (2nd attempt) +- Status: Deferred indefinitely - too hard for maintainers to understand impact and correctness of changes +- Maintainer reviews: 3 + + +### Analysis - causes of complexity + +Apply is implemented by diffing the 3 sources (last-applied, local, remote) as 2 2-way diffs and then +merging the results of those 2 diffs into a 3rd result. The diffs can each produce patch request where +a single logic update (e.g. remove 'foo' and add 'bar' to a field that is a list) may require spreading the +patch result across multiple pieces of the patch (a 'delete' directive, an 'order' directive +and the list itself). + +Because of the way diff is implemented with 2-way diffs, a simple bit of logic +"compare local to remote" and do X - is non-trivial to define. The code that compares local to remote +is also executed to compare last-applied to local, but with the local argument differing in location. +To compare local to remote means understanding what will happen when the same code is executed +comparing last-applied to local, and then putting in the appropriate guards to short-circuit the +logic in one context or the other as needed. last-applied and remote are not compared directly, and instead +are only compared indirectly when the 2 diff results are merged. Information that is redundant or +should be checked for consistency across all 3 sources (e.g. checking for conflicts) is spread across +3 logic locations - the first 2-way diff, the second 2-way diff and the merge of the 2 diffs. + +That the diffs each may produce multiple patch directives + results that constitute an update to a single +field compounds the complexity of that comparing a single field occurs across 3 locations. + +The diff / patch logic itself does not follow any sort of structure to encapsulate complexity +into components so that logic doesn't bleed cross concerns. The logic to collate the last-applied, local and +remote field values, the logic to diff the field values and the logic to create the patch is +all combined in the same group of package-scoped functions, instead of encapsulating +each of these responsibilities in its own interface. + +Sprinkling the implementation across dozens of locations makes it very challenging to +flag guard the new behavior. If issues are discovered during the stabilization period we cannot +easily revert to the previous behavior by changing a default flag value. The inability to build +in these sorts of break-glass options further degrades confidence in safely accepting PRs. + +This is a text-book example of what the [Visitor pattern](https://en.wikipedia.org/wiki/Visitor_pattern) +was designed to address. + +- Encapsulate logic in *Element*s and *Visitor*s +- Introduce logic for new a field type by adding a new *Element* type +- Introduce logic for new a merge strategy by defining a new *Visitor* implementation +- Introduce logic on structuring of a field by updating the parsing function for that field type + +If the apply diff logic was redesigned, most of the preceding PRs could be implemented by +only touching a few existing code locations to introduce the new type / method, and +then encapsulating the logic in a single type. This would make it simple to flag guard +new behaviors before defaulting them to on. + diff --git a/contributors/design-proposals/sig-cli/get-describe-apiserver-extensions.md b/contributors/design-proposals/cli/get-describe-apiserver-extensions.md index 42884990..aa129f4a 100644 --- a/contributors/design-proposals/sig-cli/get-describe-apiserver-extensions.md +++ b/contributors/design-proposals/cli/get-describe-apiserver-extensions.md @@ -142,7 +142,7 @@ compiled in printer exist for a type? Client doesn't find the open-api extensions. Fallback on 1.5 behavior. -In the future, this will provide stronger backwards / forwards compability +In the future, this will provide stronger backwards / forwards compatibility as it will allow clients to print objects #### Newer server diff --git a/contributors/design-proposals/kubectl-create-from-env-file.md b/contributors/design-proposals/cli/kubectl-create-from-env-file.md index adc235db..71d6d853 100644 --- a/contributors/design-proposals/kubectl-create-from-env-file.md +++ b/contributors/design-proposals/cli/kubectl-create-from-env-file.md @@ -27,7 +27,7 @@ environment. Any ill-formed line will be flagged as an error and will prevent the `ConfigMap` or `Secret` from being created. -[Docker's environment file processing](https://github.com/docker/docker/blob/master/runconfig/opts/envfile.go) +[Docker's environment file processing](https://github.com/moby/moby/blob/master/opts/env.go) ## Examples diff --git a/contributors/design-proposals/kubectl-extension.md b/contributors/design-proposals/cli/kubectl-extension.md index 8764e74e..1589f4c3 100644 --- a/contributors/design-proposals/kubectl-extension.md +++ b/contributors/design-proposals/cli/kubectl-extension.md @@ -1,36 +1,3 @@ -<!-- BEGIN MUNGE: UNVERSIONED_WARNING --> - -<!-- BEGIN STRIP_FOR_RELEASE --> - -<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING" - width="25" height="25"> -<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING" - width="25" height="25"> -<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING" - width="25" height="25"> -<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING" - width="25" height="25"> -<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING" - width="25" height="25"> - -<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2> - -If you are using a released version of Kubernetes, you should -refer to the docs that go with that version. - -<!-- TAG RELEASE_LINK, added by the munger automatically --> -<strong> -The latest release of this document can be found -[here](http://releases.k8s.io/release-1.3/docs/proposals/kubectl-extension.md). - -Documentation for other releases can be found at -[releases.k8s.io](http://releases.k8s.io). -</strong> --- - -<!-- END STRIP_FOR_RELEASE --> - -<!-- END MUNGE: UNVERSIONED_WARNING --> # Kubectl Extension diff --git a/contributors/design-proposals/kubectl-login.md b/contributors/design-proposals/cli/kubectl-login.md index a333e9dc..01ab19bd 100644 --- a/contributors/design-proposals/kubectl-login.md +++ b/contributors/design-proposals/cli/kubectl-login.md @@ -214,7 +214,3 @@ Phase 2: Further improvements will require adding more authentication providers, and adapting existing plugins to take advantage of challenge based authentication. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/sig-cli/kubectl_apply_getsetdiff_last_applied_config.md b/contributors/design-proposals/cli/kubectl_apply_getsetdiff_last_applied_config.md index 67bcc860..af3c0ff8 100644 --- a/contributors/design-proposals/sig-cli/kubectl_apply_getsetdiff_last_applied_config.md +++ b/contributors/design-proposals/cli/kubectl_apply_getsetdiff_last_applied_config.md @@ -190,7 +190,3 @@ instead of by the configuration. 4. Verify the last-applied-config has been updated - `kubectl apply view-last-applied -f deployment_nginx.yaml` - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/multi-fields-merge-key.md b/contributors/design-proposals/cli/multi-fields-merge-key.md index 13dcf9ce..9db3d549 100644 --- a/contributors/design-proposals/multi-fields-merge-key.md +++ b/contributors/design-proposals/cli/multi-fields-merge-key.md @@ -6,13 +6,13 @@ Support multi-fields merge key in Strategic Merge Patch. ## Background -Strategic Merge Patch is covered in this [doc](https://github.com/kubernetes/community/blob/master/contributors/devel/strategic-merge-patch.md). +Strategic Merge Patch is covered in this [doc](/contributors/devel/strategic-merge-patch.md). In Strategic Merge Patch, we use Merge Key to identify the entries in the list of non-primitive types. It must always be present and unique to perform the merge on the list of non-primitive types, and will be preserved. The merge key exists in the struct tag (e.g. in [types.go](https://github.com/kubernetes/kubernetes/blob/5a9759b0b41d5e9bbd90d5a8f3a4e0a6c0b23b47/pkg/api/v1/types.go#L2831)) -and the [OpenAPI spec](https://github.com/kubernetes/kubernetes/blob/master/api/openapi-spec/swagger.json). +and the [OpenAPI spec](https://git.k8s.io/kubernetes/api/openapi-spec/swagger.json). ## Motivation @@ -35,7 +35,7 @@ That part will be addressed by [#476](https://github.com/kubernetes/community/pu ### API Change -If a merge key has multiple fields, it will be a string of merge key fields seperated by ",", i.e. `patchMergeKey:"<key1>,<key2>,<key3>"`. +If a merge key has multiple fields, it will be a string of merge key fields separated by ",", i.e. `patchMergeKey:"<key1>,<key2>,<key3>"`. If a merge key only has one field, it will be the same as before, i.e. `patchMergeKey:"<key1>"`. diff --git a/contributors/design-proposals/preserve-order-in-strategic-merge-patch.md b/contributors/design-proposals/cli/preserve-order-in-strategic-merge-patch.md index 9fec6f0f..5e9644a6 100644 --- a/contributors/design-proposals/preserve-order-in-strategic-merge-patch.md +++ b/contributors/design-proposals/cli/preserve-order-in-strategic-merge-patch.md @@ -4,7 +4,7 @@ Author: @mengqiy ## Motivation -Background of the Strategic Merge Patch is convered [here](../devel/strategic-merge-patch.md). +Background of the Strategic Merge Patch is covered [here](../devel/strategic-merge-patch.md). The Kubernetes API may apply semantic meaning to the ordering of items within a list, however the strategic merge patch does not keeping the ordering of elements. diff --git a/contributors/design-proposals/simple-rolling-update.md b/contributors/design-proposals/cli/simple-rolling-update.md index c4a5f671..32d75820 100644 --- a/contributors/design-proposals/simple-rolling-update.md +++ b/contributors/design-proposals/cli/simple-rolling-update.md @@ -1,10 +1,10 @@ ## Simple rolling update This is a lightweight design document for simple -[rolling update](../user-guide/kubectl/kubectl_rolling-update.md) in `kubectl`. +[rolling update](https://kubernetes.io/docs/user-guide/kubectl/kubectl_rolling-update.md#rolling-update) in `kubectl`. Complete execution flow can be found [here](#execution-details). See the -[example of rolling update](../user-guide/update-demo/) for more information. +[example of rolling update](https://kubernetes.io/docs/tutorials/kubernetes-basics/update-intro/) for more information. ### Lightweight rollout @@ -124,8 +124,3 @@ rollout with the old version * Set `desired-replicas` annotation on `foo` to match the annotation on `foo-next` * Goto Rollout with `foo` and `foo-next` trading places. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/cloud-provider-refactoring.md b/contributors/design-proposals/cloud-provider/cloud-provider-refactoring.md index 7a554b4c..0d25ccab 100644 --- a/contributors/design-proposals/cloud-provider-refactoring.md +++ b/contributors/design-proposals/cloud-provider/cloud-provider-refactoring.md @@ -133,7 +133,7 @@ This is the only step that is different in the upgrade process. In order to comp kubectl apply -f cloud-controller-manager.yml ``` -This will start the cloud specific controller manager in your kuberentes setup. +This will start the cloud specific controller manager in your kubernetes setup. The downgrade steps are also the same as before for all the components except the cloud-controller-manager. In case of the cloud-controller-manager, the deployment should be deleted using @@ -161,8 +161,3 @@ Release 1.9: All of the legacy cloud providers will be completely removed in thi * Cloud specific operations will be moved out of kube-apiserver using the external admission controller pattern mentioned above. * All cloud specific volume controller loops (attach, detach, provision operation controllers) will be switched to using flex volumes. Flex volumes do not need in-tree cloud specific calls. * As the final step, all of the cloud provider specific code will be moved out of tree. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/cloudprovider-storage-metrics.md b/contributors/design-proposals/cloud-provider/cloudprovider-storage-metrics.md index fbe5a394..838c7e43 100644 --- a/contributors/design-proposals/cloudprovider-storage-metrics.md +++ b/contributors/design-proposals/cloud-provider/cloudprovider-storage-metrics.md @@ -13,7 +13,7 @@ we will implement metrics for: * AWS We will also implement metrics only for storage API calls for now. This feature -does introduces hooks into kubernetes code which can be used to add additonal metrics +does introduces hooks into kubernetes code which can be used to add additional metrics but we only focus on storage API calls here. ## Motivation @@ -29,7 +29,7 @@ but we only focus on storage API calls here. ### Metric format and collection Metrics emitted from cloud provider will fall under category of service metrics -as defined in [Kubernetes Monitoring Architecture](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/monitoring_architecture.md). +as defined in [Kubernetes Monitoring Architecture](/contributors/design-proposals/instrumentation/monitoring_architecture.md). The metrics will be emitted using [Prometheus format](https://prometheus.io/docs/instrumenting/exposition_formats/) and available for collection @@ -40,7 +40,7 @@ metrics on `/metrics` HTTP endpoint. This proposal merely extends available metr Any collector which can parse Prometheus metric format should be able to collect metrics from these endpoints. -A more detailed description of monitoring pipeline can be found in [Monitoring architecture] (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/monitoring_architecture.md#monitoring-pipeline) document. +A more detailed description of monitoring pipeline can be found in [Monitoring architecture] (/contributors/design-proposals/instrumentation/monitoring_architecture.md#monitoring-pipeline) document. #### Metric Types @@ -50,7 +50,7 @@ the external Cloud Provider - we will use [Histogram](https://prometheus.io/docs emitting these metrics. We will be using `HistogramVec` type so as we can attach dimensions at runtime. All metrics will contain API action -being taken as a dimension. The cloudprovider maintainer may choose to add additonal dimensions as needed. If a +being taken as a dimension. The cloudprovider maintainer may choose to add additional dimensions as needed. If a dimension is not available at point of emission sentinel value `<n/a>` should be emitted as a placeholder. We are also interested in counter of cloudprovider API errors. `NewCounterVec` type will be used for keeping diff --git a/contributors/design-proposals/cluster-lifecycle/OWNERS b/contributors/design-proposals/cluster-lifecycle/OWNERS new file mode 100644 index 00000000..d69f24ee --- /dev/null +++ b/contributors/design-proposals/cluster-lifecycle/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-cluster-lifecycle-leads +approvers: + - sig-cluster-lifecycle-leads +labels: + - sig/cluster-lifecycle diff --git a/contributors/design-proposals/bootstrap-discovery.md b/contributors/design-proposals/cluster-lifecycle/bootstrap-discovery.md index 5eef94e3..f481e02d 100644 --- a/contributors/design-proposals/bootstrap-discovery.md +++ b/contributors/design-proposals/cluster-lifecycle/bootstrap-discovery.md @@ -179,7 +179,7 @@ A new controller (`bootstrapsigner`) is introduced that will watch for both new/ Another controller (`tokencleaner`) is introduced that deletes tokens that are past their expiration time. -Logically these controllers could run as a separate component in the control plane. But, for the sake of efficiency, they are bundeled as part of the Kubernetes controller-manager. +Logically these controllers could run as a separate component in the control plane. But, for the sake of efficiency, they are bundled as part of the Kubernetes controller-manager. ## `kubeadm` UX @@ -221,7 +221,7 @@ Only one of `--discovery-file` or `--discovery-token` can be set. If more than Our documentations (and output from `kubeadm`) should stress to users that when the token is configured for authentication and used for TLS bootstrap is a pretty powerful credential due to that any person with access to it can claim to be a node. The highest risk regarding being able to claim a credential in the `system:nodes` group is that it can read all Secrets in the cluster, which may compromise the cluster. -The [Node Authorizer](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-authorizer.md) locks this down a bit, but an untrusted person could still try to +The [Node Authorizer](/contributors/design-proposals/node/kubelet-authorizer.md) locks this down a bit, but an untrusted person could still try to guess a node's name, get such a credential, guess the name of the Secret and be able to get that. Users should set a TTL on the token to limit the above mentioned risk. `kubeadm` sets a 24h TTL on the node bootstrap token by default in v1.8. @@ -242,7 +242,3 @@ The binding of the `system:bootstrappers` (or similar) group to the ability to s - Initial proposal ([@jbeda](https://github.com/jbeda)): [link](https://github.com/kubernetes/community/blob/cb9f198a0763e0a7540cdcc9db912a403ab1acab/contributors/design-proposals/bootstrap-discovery.md) - v1.6 updates ([@jbeda](https://github.com/jbeda)): [link](https://github.com/kubernetes/community/blob/d8ce9e91b0099795318bb06c13f00d9dad41ac26/contributors/design-proposals/bootstrap-discovery.md) - v1.8 updates ([@luxas](https://github.com/luxas)) - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/cluster-deployment.md b/contributors/design-proposals/cluster-lifecycle/cluster-deployment.md index e9ca8d54..a0402ebc 100644 --- a/contributors/design-proposals/cluster-deployment.md +++ b/contributors/design-proposals/cluster-lifecycle/cluster-deployment.md @@ -163,9 +163,5 @@ For simpler UX we will provide simple bash scripts that will wrap all basic comm One disadvantage of using Ansible is that it adds a dependency on a machine which runs deployment scripts. We will workaround this by distributing deployment scripts via a docker image so that user will run the following command to create a cluster: -```docker run gcr.io/google_containers/deploy_kubernetes:v1.2 up --num-nodes=3 --provider=aws``` +```docker run k8s.gcr.io/deploy_kubernetes:v1.2 up --num-nodes=3 --provider=aws``` - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/clustering.md b/contributors/design-proposals/cluster-lifecycle/clustering.md index ca42035b..e681d8e9 100644 --- a/contributors/design-proposals/clustering.md +++ b/contributors/design-proposals/cluster-lifecycle/clustering.md @@ -121,8 +121,3 @@ the `queue` policy defined above. This manual intervention could be replaced by code that can verify the signing requests via other means.  - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/clustering/.gitignore b/contributors/design-proposals/cluster-lifecycle/clustering/.gitignore index 67bcd6cb..67bcd6cb 100644 --- a/contributors/design-proposals/clustering/.gitignore +++ b/contributors/design-proposals/cluster-lifecycle/clustering/.gitignore diff --git a/contributors/design-proposals/clustering/Dockerfile b/contributors/design-proposals/cluster-lifecycle/clustering/Dockerfile index e7abc753..e7abc753 100644 --- a/contributors/design-proposals/clustering/Dockerfile +++ b/contributors/design-proposals/cluster-lifecycle/clustering/Dockerfile diff --git a/contributors/design-proposals/clustering/Makefile b/contributors/design-proposals/cluster-lifecycle/clustering/Makefile index e72d441e..e72d441e 100644 --- a/contributors/design-proposals/clustering/Makefile +++ b/contributors/design-proposals/cluster-lifecycle/clustering/Makefile diff --git a/contributors/design-proposals/clustering/OWNERS b/contributors/design-proposals/cluster-lifecycle/clustering/OWNERS index b3d71823..b3d71823 100644 --- a/contributors/design-proposals/clustering/OWNERS +++ b/contributors/design-proposals/cluster-lifecycle/clustering/OWNERS diff --git a/contributors/design-proposals/clustering/README.md b/contributors/design-proposals/cluster-lifecycle/clustering/README.md index d7e2e2e0..9fe1f027 100644 --- a/contributors/design-proposals/clustering/README.md +++ b/contributors/design-proposals/cluster-lifecycle/clustering/README.md @@ -29,7 +29,3 @@ If you have the fswatch utility installed, you can have it monitor the file system and automatically rebuild when files have changed. Just do a `make watch`. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/clustering/dynamic.png b/contributors/design-proposals/cluster-lifecycle/clustering/dynamic.png Binary files differindex 92b40fee..92b40fee 100644 --- a/contributors/design-proposals/clustering/dynamic.png +++ b/contributors/design-proposals/cluster-lifecycle/clustering/dynamic.png diff --git a/contributors/design-proposals/clustering/dynamic.seqdiag b/contributors/design-proposals/cluster-lifecycle/clustering/dynamic.seqdiag index 567d5bf9..567d5bf9 100644 --- a/contributors/design-proposals/clustering/dynamic.seqdiag +++ b/contributors/design-proposals/cluster-lifecycle/clustering/dynamic.seqdiag diff --git a/contributors/design-proposals/clustering/static.png b/contributors/design-proposals/cluster-lifecycle/clustering/static.png Binary files differindex bcdeca7e..bcdeca7e 100644 --- a/contributors/design-proposals/clustering/static.png +++ b/contributors/design-proposals/cluster-lifecycle/clustering/static.png diff --git a/contributors/design-proposals/clustering/static.seqdiag b/contributors/design-proposals/cluster-lifecycle/clustering/static.seqdiag index bdc54b76..bdc54b76 100644 --- a/contributors/design-proposals/clustering/static.seqdiag +++ b/contributors/design-proposals/cluster-lifecycle/clustering/static.seqdiag diff --git a/contributors/design-proposals/cluster-lifecycle/draft-20171020-bootstrap-checkpointing.md b/contributors/design-proposals/cluster-lifecycle/draft-20171020-bootstrap-checkpointing.md new file mode 100644 index 00000000..5c133a0a --- /dev/null +++ b/contributors/design-proposals/cluster-lifecycle/draft-20171020-bootstrap-checkpointing.md @@ -0,0 +1,147 @@ +# Kubernetes Bootstrap Checkpointing Proposal + +## Metadata + +```yaml +kep-number: draft-20171029-boostrap-checkpointing.md +title: Kubernetes Bootstrap Checkpointing Proposal +authors: + - name: Timothy St. Clair + github: @timothysc + email: timothysc@gmail.com +owning-sig: sig-cluster-lifecycle +participating-sigs: + - sig-node +reviewers: + - @yujuhong + - @luxas + - @roberthbailey +approvers: + - @yujuhong + - @roberthbailey +editor: + name: @timothysc +creation-date: 2017-10-20 +last-updated: 2017-10-20 +status: final-review +``` + +## Table of Contents + +* [Summary](#summary) +* [Objectives](#objectives) + * [Goals](#goals) + * [Non-Goals](#non-goals) +* [Proposal](#proposal) + * [User Stories](#user-stories) +* [Graduation Criteria](#graduation-criteria) +* [Implementation History](#implementation-history) +* [Unresolved Questions](#unresolved-questions) + +## Summary + +There are several methods to deploy a kubernetes cluster, one method that +offers some unique advantages is self hosting. The purpose of this proposal +is to outline a method to checkpoint specific annotated pods, namely the +control plane components, for the purpose of enabling self hosting. + +The details of self hosting are beyond the scope of this proposal, and are +outlined in the references listed below: + + - [Self Hosted Kubernetes][0] + - [Kubeadm Upgrades][1] + +Extra details on this proposal, and its history, can be found in the links +below: + + - [Bootstrap Checkpointing Draft 1][2] + - [Bootstrap Checkpointing Draft 2][3] + - [WIP Implementation][4] + +## Objectives + +The scope of this proposal is **bounded**, but has the potential for broader +reuse in the future. The reader should be mindful of the explicitly stated +[Non-Goals](#non-goals) that are listed below. + +### Goals + + - Provide a basic framework for recording annotated *Pods* to the filesystem. + - Ensure that a restart of the kubelet checks for existence of these files + and loads them on startup. + +### Non-Goals + +- This is not a generic checkpointing mechanism for arbitrary resources. +(e.g. Secrets) Such changes require wider discussions. +- This will not checkpoint internal kubelet state. +- This proposal does not cover self hosted kubelet(s). It is beyond the +scope of this proposal, and comes with it's own unique set of challenges. + +## Proposal +The enablement of this feature is gated by a single command line flag that +is passed to the kubelet on startup, ```--bootstrap-checkpoint-path``` , +and will be denoted that it is ```[Alpha]```. + +### User Stories + +#### Pod Submission to Running +- On submission of a Pod, via kubeadm or an operator, an annotation +```node.kubernetes.io/bootstrap-checkpoint=true``` is added to that Pod, which +indicates that it should be checkpointed by the kubelet. When the kubelet +receives a notification from the apiserver that a new pod is to run, it will +inspect the ```--bootstrap-checkpoint-path``` flag to determine if +checkpointing is enabled. Finally, the kubelet will perform an atomic +write of a ```Pod_UID.yaml``` file when the afore mentioned annotation exists. +The scope of this annotation is bounded and will not be promoted to a field. + +#### Pod Deletion +- On detected deletion of a Pod, the kubelet will remove the associated +checkpoint from the filesystem. Any failure to remove a pod, or file, will +result in an error notification in the kubelet logs. + +#### Cold Start +- On a cold start, the kubelet will check the value of +```--bootstrap-checkpoint-path```. If the value is specified, it will read in +the contents of the that directory and startup the appropriate Pod. Lastly, +the kubelet will then pull the list of pods from the api-server and rectify +what is supposed to be running according to what is bound, and will go through +its normal startup procedure. + +### Implementation Constraints +Due to its opt-in behavior, administrators will need to take the same precautions +necessary in segregating master nodes, when enabling the bootstrap annotation. + +Please see [WIP Implementation][4] for more details. + +## Graduation Criteria + +Graduating this feature is a responsibility of sig-cluster-lifecycle and +sig-node to determine over the course of the 1.10 and 1.11 releases. History +has taught us that initial implementations often have a tendency overlook use +cases and require refinement. It is the goal of this proposal to have an +initial alpha implementation of of bootstrap checkpoining in the 1.9 cycle, +and further refinement will occur after we have validated it across several +deployments. + +## Testing +Testing of this feature will occur in three parts. +- Unit testing of standard code behavior +- Simple node-e2e test to ensure restart recovery +- (TODO) E2E test w/kubeadm self hosted master restart recovery of an apiserver. + +## Implementation History + +- 20171020 - 1.9 draft proposal +- ? - accepted proposal +- ? - alpha implementation code complete + +## Unresolved Questions + +* None at this time. + +[0]: /contributors/design-proposals/cluster-lifecycle/self-hosted-kubernetes.md +[1]: https://github.com/kubernetes/community/pull/825 +[2]: https://docs.google.com/document/d/1hhrCa_nv0Sg4O_zJYOnelE8a5ClieyewEsQM6c7-5-o/edit?ts=5988fba8# +[3]: https://docs.google.com/document/d/1qmK0Iq4fqxnd8COBFZHpip27fT-qSPkOgy1x2QqjYaQ/edit?ts=599b797c# +[4]: https://github.com/kubernetes/kubernetes/pull/50984 diff --git a/contributors/design-proposals/dramatically-simplify-cluster-creation.md b/contributors/design-proposals/cluster-lifecycle/dramatically-simplify-cluster-creation.md index 78a1089a..3472115d 100644 --- a/contributors/design-proposals/dramatically-simplify-cluster-creation.md +++ b/contributors/design-proposals/cluster-lifecycle/dramatically-simplify-cluster-creation.md @@ -3,7 +3,7 @@ > ***Please note: this proposal doesn't reflect final implementation, it's here for the purpose of capturing the original ideas.*** > ***You should probably [read `kubeadm` docs](http://kubernetes.io/docs/getting-started-guides/kubeadm/), to understand the end-result of this effor.*** -Luke Marsden & many others in [SIG-cluster-lifecycle](https://github.com/kubernetes/community/tree/master/sig-cluster-lifecycle). +Luke Marsden & many others in [SIG-cluster-lifecycle](/sig-cluster-lifecycle). 17th August 2016 @@ -259,8 +259,3 @@ It also doesn't require kubelet to do TLS bootstrapping - kubeadm handles that. ## See also * [Joe Beda's "K8s the hard way easier"](https://docs.google.com/document/d/1lJ26LmCP-I_zMuqs6uloTgAnHPcuT7kOYtQ7XSgYLMA/edit#heading=h.ilgrv18sg5t) which combines Kelsey's "Kubernetes the hard way" with history of proposed UX at the end (scroll all the way down to the bottom). - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/ha_master.md b/contributors/design-proposals/cluster-lifecycle/ha_master.md index b225e667..3d0de1f1 100644 --- a/contributors/design-proposals/ha_master.md +++ b/contributors/design-proposals/cluster-lifecycle/ha_master.md @@ -231,6 +231,3 @@ will be in the same version. * Apiserver talks only to a local etcd replica which will be in a compatible version * We assume we will introduce this setup after we upgrade to etcd v3 so we don't need to cover upgrading database. -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/cluster-lifecycle/high-availability.md b/contributors/design-proposals/cluster-lifecycle/high-availability.md new file mode 100644 index 00000000..d893c597 --- /dev/null +++ b/contributors/design-proposals/cluster-lifecycle/high-availability.md @@ -0,0 +1,4 @@ +# High Availability of Scheduling and Controller Components in Kubernetes + +This document is deprecated. For more details about running a highly available +cluster master, please see the [admin instructions document](https://kubernetes.io/docs/admin/high-availability/). diff --git a/contributors/design-proposals/kubelet-tls-bootstrap.md b/contributors/design-proposals/cluster-lifecycle/kubelet-tls-bootstrap.md index fbd98413..f725b1a9 100644 --- a/contributors/design-proposals/kubelet-tls-bootstrap.md +++ b/contributors/design-proposals/cluster-lifecycle/kubelet-tls-bootstrap.md @@ -135,7 +135,7 @@ type CertificateSigningRequestList struct { ## Certificate Request Process -### Node intialization +### Node initialization When the kubelet executes it checks a location on disk for TLS assets (currently `/var/run/kubernetes/kubelet.{key,crt}` by default). If it finds @@ -237,7 +237,3 @@ auth. - supplemental policy (e.g. cluster CA only issues 30-day certs for hostnames *.k8s.example.com, each new cert must have fresh keys, ...) - fully automated provisioning (using a handshake protocol or external list of authorized machines) - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/local-cluster-ux.md b/contributors/design-proposals/cluster-lifecycle/local-cluster-ux.md index c78a51b7..8dac84bd 100644 --- a/contributors/design-proposals/local-cluster-ux.md +++ b/contributors/design-proposals/cluster-lifecycle/local-cluster-ux.md @@ -81,10 +81,10 @@ The final name of this tool is TBD. Suggestions are welcome! Minikube will provide a unified CLI to interact with the local cluster. The CLI will support only a few operations: - - **Start** - creates & starts a local cluster along with setting up kubectl & networking (if necessary) - - **Stop** - suspends the local cluster & preserves cluster state - - **Delete** - deletes the local cluster completely - - **Upgrade** - upgrades internal components to the latest available version (upgrades are not guaranteed to preserve cluster state) +- **Start** - creates & starts a local cluster along with setting up kubectl & networking (if necessary) +- **Stop** - suspends the local cluster & preserves cluster state +- **Delete** - deletes the local cluster completely +- **Upgrade** - upgrades internal components to the latest available version (upgrades are not guaranteed to preserve cluster state) For running and managing the kubernetes components themselves, we can re-use [Spread's localkube](https://github.com/redspread/localkube). Localkube is a self-contained go binary that includes all the master components including DNS and runs them using multiple go threads. @@ -154,8 +154,3 @@ minikube -> docker -> localkube - The latest version of Minikube is guaranteed to support the latest release of Kubernetes, including documentation. - The Google Cloud SDK will package minikube and provide utilities for configuring kubectl to use it, but will not in any other way wrap minikube. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/runtimeconfig.md b/contributors/design-proposals/cluster-lifecycle/runtimeconfig.md index b2ed83dd..c247eff8 100644 --- a/contributors/design-proposals/runtimeconfig.md +++ b/contributors/design-proposals/cluster-lifecycle/runtimeconfig.md @@ -64,6 +64,3 @@ APIs and not flags ([#12245](https://issues.k8s.io/12245)). When that is added, could be handled by versioned component config and the component flags deprecated. -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/self-hosted-final-cluster.png b/contributors/design-proposals/cluster-lifecycle/self-hosted-final-cluster.png Binary files differindex e5302b07..e5302b07 100644 --- a/contributors/design-proposals/self-hosted-final-cluster.png +++ b/contributors/design-proposals/cluster-lifecycle/self-hosted-final-cluster.png diff --git a/contributors/design-proposals/self-hosted-kubelet.md b/contributors/design-proposals/cluster-lifecycle/self-hosted-kubelet.md index d2318bea..765086f2 100644 --- a/contributors/design-proposals/self-hosted-kubelet.md +++ b/contributors/design-proposals/cluster-lifecycle/self-hosted-kubelet.md @@ -128,8 +128,3 @@ register itself with a given taint when it first contacts the API server. Given that, a kubelet could register itself with a given taint such as “component=kubelet”, and a kubelet pod could exist that has a toleration to that taint, ensuring it is the only pod the “bootstrap” kubelet runs. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/self-hosted-kubernetes.md b/contributors/design-proposals/cluster-lifecycle/self-hosted-kubernetes.md index 26220180..9152f251 100644 --- a/contributors/design-proposals/self-hosted-kubernetes.md +++ b/contributors/design-proposals/cluster-lifecycle/self-hosted-kubernetes.md @@ -100,4 +100,4 @@ Kubernetes self-hosted is working today. Bootkube is an implementation of the "t - [Health check endpoints for components don't work correctly](https://github.com/kubernetes-incubator/bootkube/issues/64#issuecomment-228144345) - [kubeadm does do self-hosted, but isn't tested yet](https://github.com/kubernetes/kubernetes/pull/40075) -- The Kubernetes [versioning policy](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/versioning.md) allows for version skew of kubelet and control plane but not skew between control plane components themselves. We must add testing and validation to Kubernetes that this skew works. Otherwise the work to make Kubernetes HA is rather pointless if it can't be upgraded in an HA manner as well. +- The Kubernetes [versioning policy](/contributors/design-proposals/release/versioning.md) allows for version skew of kubelet and control plane but not skew between control plane components themselves. We must add testing and validation to Kubernetes that this skew works. Otherwise the work to make Kubernetes HA is rather pointless if it can't be upgraded in an HA manner as well. diff --git a/contributors/design-proposals/self-hosted-layers.png b/contributors/design-proposals/cluster-lifecycle/self-hosted-layers.png Binary files differindex 1dc3e06a..1dc3e06a 100644 --- a/contributors/design-proposals/self-hosted-layers.png +++ b/contributors/design-proposals/cluster-lifecycle/self-hosted-layers.png diff --git a/contributors/design-proposals/self-hosted-moving-parts.png b/contributors/design-proposals/cluster-lifecycle/self-hosted-moving-parts.png Binary files differindex 423add2e..423add2e 100644 --- a/contributors/design-proposals/self-hosted-moving-parts.png +++ b/contributors/design-proposals/cluster-lifecycle/self-hosted-moving-parts.png diff --git a/contributors/design-proposals/dir_struct.txt b/contributors/design-proposals/dir_struct.txt new file mode 100644 index 00000000..ef61ae75 --- /dev/null +++ b/contributors/design-proposals/dir_struct.txt @@ -0,0 +1,244 @@ +Uncategorized + create_sheet.py + design_proposal_template.md + dir_struct.txt + multi-platform.md + owners + readme.md +./sig-cli + get-describe-apiserver-extensions.md + kubectl-create-from-env-file.md + kubectl-extension.md + kubectl-login.md + kubectl_apply_getsetdiff_last_applied_config.md + multi-fields-merge-key.md + owners + preserve-order-in-strategic-merge-patch.md + simple-rolling-update.md +./network + command_execution_port_forwarding.md + external-lb-source-ip-preservation.md + flannel-integration.md + network-policy.md + networking.md + service-discovery.md + service-external-name.md +./resource-management + admission_control_limit_range.md + admission_control_resource_quota.md + device-plugin-overview.png + device-plugin.md + device-plugin.png + gpu-support.md + hugepages.md + resource-quota-scoping.md +./testing + flakiness-sla.md +./autoscaling + horizontal-pod-autoscaler.md + hpa-status-conditions.md + hpa-v2.md + initial-resources.md +./architecture + architecture.md + architecture.png + architecture.svg + identifiers.md + namespaces.md + principles.md +./api-machinery + add-new-patchstrategy-to-clear-fields-not-present-in-patch.md + admission_control.md + admission_control_event_rate_limit.md + admission_control_extension.md + aggregated-api-servers.md + api-chunking.md + api-group.md + apiserver-build-in-admission-plugins.md + apiserver-count-fix.md + apiserver-watch.md + auditing.md + bulk_watch.md + client-package-structure.md + controller-ref.md + csi-client-structure-proposal.md + csi-new-client-library-procedure.md + customresources-validation.md + dynamic-admission-control-configuration.md + event_compression.md + extending-api.md + garbage-collection.md + metadata-policy.md + protobuf.md + server-get.md + synchronous-garbage-collection.md + thirdpartyresources.md +./node + all-in-one-volume.md + annotations-downward-api.md + configmap.md + container-init.md + container-runtime-interface-v1.md + cpu-manager.md + cri-dockershim-checkpoint.md + disk-accounting.md + downward_api_resources_limits_requests.md + dynamic-kubelet-configuration.md + envvar-configmap.md + expansion.md + kubelet-auth.md + kubelet-authorizer.md + kubelet-cri-logging.md + kubelet-eviction.md + kubelet-hypercontainer-runtime.md + kubelet-rkt-runtime.md + kubelet-rootfs-distribution.md + kubelet-systemd.md + node-allocatable.md + optional-configmap.md + pleg.png + pod-cache.png + pod-lifecycle-event-generator.md + pod-pid-namespace.md + pod-resource-management.md + propagation.md + resource-qos.md + runtime-client-server.md + runtime-pod-cache.md + seccomp.md + secret-configmap-downwardapi-file-mode.md + selinux-enhancements.md + selinux.md + sysctl.md +./service-catalog + pod-preset.md +./instrumentation + core-metrics-pipeline.md + custom-metrics-api.md + metrics-server.md + monitoring_architecture.md + monitoring_architecture.png + performance-related-monitoring.md + resource-metrics-api.md + volume_stats_pvc_ref.md +./auth + access.md + apparmor.md + enhance-pluggable-policy.md + flex-volumes-drivers-psp.md + image-provenance.md + no-new-privs.md + pod-security-context.md + pod-security-policy.md + secrets.md + security.md + security_context.md + service_accounts.md +./multicluster + control-plane-resilience.md + federated-api-servers.md + federated-ingress.md + federated-placement-policy.md + federated-replicasets.md + federated-services.md + federation-clusterselector.md + federation-high-level-arch.png + federation-lite.md + federation-phase-1.md + federation.md + ubernetes-cluster-state.png + ubernetes-design.png + ubernetes-scheduling.png +./scalability + kubemark.md + kubemark_architecture.png + scalability-testing.md +./cluster-lifecycle + bootstrap-discovery.md + cluster-deployment.md + clustering.md + dramatically-simplify-cluster-creation.md + ha_master.md + high-availability.md + kubelet-tls-bootstrap.md + local-cluster-ux.md + runtimeconfig.md + self-hosted-final-cluster.png + self-hosted-kubelet.md + self-hosted-kubernetes.md + self-hosted-layers.png + self-hosted-moving-parts.png +./cluster-lifecycle/clustering + .gitignore + dockerfile + dynamic.png + dynamic.seqdiag + makefile + owners + readme.md + static.png + static.seqdiag +./release + release-notes.md + release-test-signal.md + versioning.md +./scheduling + hugepages.md + multiple-schedulers.md + nodeaffinity.md + pod-preemption.md + pod-priority-api.md + podaffinity.md + rescheduler.md + rescheduling-for-critical-pods.md + rescheduling.md + resources.md + scheduler_extender.md + taint-node-by-condition.md + taint-toleration-dedicated.md +./scheduling/images + .gitignore + owners + preemption_1.png + preemption_2.png + preemption_3.png + preemption_4.png +./apps + controller_history.md + cronjob.md + daemon.md + daemonset-update.md + deploy.md + deployment.md + indexed-job.md + job.md + obsolete_templates.md + selector-generation.md + stateful-apps.md + statefulset-update.md +./storage + containerized-mounter.md + containerized-mounter.md~ + default-storage-class.md + flexvolume-deployment.md + grow-volume-size.md + local-storage-overview.md + mount-options.md + owners + persistent-storage.md + pod-safety.md + volume-hostpath-qualifiers.md + volume-metrics.md + volume-ownership-management.md + volume-provisioning.md + volume-selectors.md + volume-snapshotting.md + volume-snapshotting.png + volumes.md +./aws + aws_under_the_hood.md +./gcp + gce-l4-loadbalancer-healthcheck.md +./cloud-provider + cloud-provider-refactoring.md + cloudprovider-storage-metrics.md diff --git a/contributors/design-proposals/federated-api-servers.md b/contributors/design-proposals/federated-api-servers.md deleted file mode 100644 index 2a6000e4..00000000 --- a/contributors/design-proposals/federated-api-servers.md +++ /dev/null @@ -1,8 +0,0 @@ -# Federated API Servers - -Moved to [aggregated-api-servers.md](./aggregated-api-servers.md) since cluster -federation stole the word "federation" from this effort and it was very confusing. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/gcp/OWNERS b/contributors/design-proposals/gcp/OWNERS new file mode 100644 index 00000000..cd2232f4 --- /dev/null +++ b/contributors/design-proposals/gcp/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-gcp-leads +approvers: + - sig-gcp-leads +labels: + - sig/gcp diff --git a/contributors/design-proposals/gce-l4-loadbalancer-healthcheck.md b/contributors/design-proposals/gcp/gce-l4-loadbalancer-healthcheck.md index 22e6d17b..22e6d17b 100644 --- a/contributors/design-proposals/gce-l4-loadbalancer-healthcheck.md +++ b/contributors/design-proposals/gcp/gce-l4-loadbalancer-healthcheck.md diff --git a/contributors/design-proposals/high-availability.md b/contributors/design-proposals/high-availability.md deleted file mode 100644 index ecaa4a99..00000000 --- a/contributors/design-proposals/high-availability.md +++ /dev/null @@ -1,8 +0,0 @@ -# High Availability of Scheduling and Controller Components in Kubernetes - -This document is deprecated. For more details about running a highly available -cluster master, please see the [admin instructions document](https://github.com/kubernetes/kubernetes/blob/master/docs/admin/high-availability.md). - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/instrumentation/OWNERS b/contributors/design-proposals/instrumentation/OWNERS new file mode 100644 index 00000000..8e29eafa --- /dev/null +++ b/contributors/design-proposals/instrumentation/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-instrumentation-leads +approvers: + - sig-instrumentation-leads +labels: + - sig/instrumentation diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/instrumentation/core-metrics-pipeline.md index 8e668065..1c9d9f70 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/instrumentation/core-metrics-pipeline.md @@ -8,7 +8,6 @@ This document proposes a design for the set of metrics included in an eventual Core Metrics Pipeline. -<!-- BEGIN MUNGE: GENERATED_TOC --> - [Core Metrics in kubelet](#core-metrics-in-kubelet) - [Introduction](#introduction) @@ -23,29 +22,28 @@ This document proposes a design for the set of metrics included in an eventual C - [On-Demand Design:](#on-demand-design) - [Future Work](#future-work) -<!-- END MUNGE: GENERATED_TOC --> ## Introduction ### Definitions "Kubelet": The daemon that runs on every kubernetes node and controls pod and container lifecycle, among many other things. ["cAdvisor":](https://github.com/google/cadvisor) An open source container monitoring solution which only monitors containers, and has no concept of kubernetes constructs like pods or volumes. -["Summary API":](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) A kubelet API which currently exposes node metrics for use by both system components and monitoring systems. -["CRI":](https://github.com/kubernetes/community/blob/master/contributors/devel/container-runtime-interface.md) The Container Runtime Interface designed to provide an abstraction over runtimes (docker, rkt, etc). -"Core Metrics": A set of metrics described in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) whose purpose is to provide metrics for first-class resource isolation and untilization features, including [resource feasibility checking](https://github.com/eBay/Kubernetes/blob/master/docs/design/resources.md#the-resource-model) and node resource management. +["Summary API":](https://git.k8s.io/kubernetes/pkg/kubelet/apis/stats/v1alpha1/types.go) A kubelet API which currently exposes node metrics for use by both system components and monitoring systems. +["CRI":](/contributors/devel/container-runtime-interface.md) The Container Runtime Interface designed to provide an abstraction over runtimes (docker, rkt, etc). +"Core Metrics": A set of metrics described in the [Monitoring Architecture](/contributors/design-proposals/instrumentation/monitoring_architecture.md) whose purpose is to provide metrics for first-class resource isolation and utilization features, including [resource feasibility checking](https://github.com/eBay/Kubernetes/blob/master/docs/design/resources.md#the-resource-model) and node resource management. "Resource": A consumable element of a node (e.g. memory, disk space, CPU time, etc). "First-class Resource": A resource critical for scheduling, whose requests and limits can be (or soon will be) set via the Pod/Container Spec. "Metric": A measure of consumption of a Resource. ### Background -The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal contains a blueprint for a set of metrics referred to as "Core Metrics". The purpose of this proposal is to specify what those metrics are, to enable work relating to the collection, by the kubelet, of the metrics. +The [Monitoring Architecture](/contributors/design-proposals/instrumentation/monitoring_architecture.md) proposal contains a blueprint for a set of metrics referred to as "Core Metrics". The purpose of this proposal is to specify what those metrics are, to enable work relating to the collection, by the kubelet, of the metrics. -Kubernetes vendors cAdvisor into its codebase, and the kubelet uses cAdvisor as a library that enables it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of kubernetes constructs (e.g. pods) to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go). cAdvisor works by collecting metrics at an interval (10 seconds, by default), and the kubelet then simply queries these cached metrics whenever it has a need for them. +Kubernetes vendors cAdvisor into its codebase, and the kubelet uses cAdvisor as a library that enables it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of kubernetes constructs (e.g. pods) to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the [Summary API](https://git.k8s.io/kubernetes/pkg/kubelet/apis/stats/v1alpha1/types.go). cAdvisor works by collecting metrics at an interval (10 seconds, by default), and the kubelet then simply queries these cached metrics whenever it has a need for them. -Currently, cAdvisor collects a large number of metrics related to system and container performance. However, only some of these metrics are consumed by the kubelet summary API, and many are not used. The kubelet [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) is published to the kubelet summary API endpoint (stats/summary). Some of the metrics provided by the summary API are consumed by kubernetes system components, but many are included for the sole purpose of providing metrics for monitoring. +Currently, cAdvisor collects a large number of metrics related to system and container performance. However, only some of these metrics are consumed by the kubelet summary API, and many are not used. The kubelet [Summary API](https://git.k8s.io/kubernetes/pkg/kubelet/apis/stats/v1alpha1/types.go) is published to the kubelet summary API endpoint (stats/summary). Some of the metrics provided by the summary API are consumed by kubernetes system components, but many are included for the sole purpose of providing metrics for monitoring. ### Motivations -The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal explains why a separate monitoring pipeline is required. +The [Monitoring Architecture](/contributors/design-proposals/instrumentation/monitoring_architecture.md) proposal explains why a separate monitoring pipeline is required. By publishing core metrics, the kubelet is relieved of its responsibility to provide metrics for monitoring. The third party monitoring pipeline also is relieved of any responsibility to provide these metrics to system components. @@ -58,7 +56,7 @@ This proposal is to use this set of core metrics, collected by the kubelet, and The target "Users" of this set of metrics are kubernetes components (though not necessarily directly). This set of metrics itself is not designed to be user-facing, but is designed to be general enough to support user-facing components. ### Non Goals -Everything covered in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) design doc will not be covered in this proposal. This includes the third party metrics pipeline, and the methods by which the metrics found in this proposal are provided to other kubernetes components. +Everything covered in the [Monitoring Architecture](/contributors/design-proposals/instrumentation/monitoring_architecture.md) design doc will not be covered in this proposal. This includes the third party metrics pipeline, and the methods by which the metrics found in this proposal are provided to other kubernetes components. Integration with CRI will not be covered in this proposal. In future proposals, integrating with CRI may provide a better abstraction of information required by the core metrics pipeline to collect metrics. @@ -84,7 +82,7 @@ Metrics requirements for "First Class Resource Isolation and Utilization Feature - Kubelet - Node-level usage metrics for Filesystems, CPU, and Memory - Pod-level usage metrics for Filesystems and Memory - - Metrics Server (outlined in [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md)), which exposes the [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md) to the following system components: + - Metrics Server (outlined in [Monitoring Architecture](/contributors/design-proposals/instrumentation/monitoring_architecture.md)), which exposes the [Resource Metrics API](/contributors/design-proposals/instrumentation/resource-metrics-api.md) to the following system components: - Scheduler - Node-level usage metrics for Filesystems, CPU, and Memory - Pod-level usage metrics for Filesystems, CPU, and Memory @@ -150,7 +148,3 @@ Suggested, tentative future work, which may be covered by future proposals: - Decide on the format, name, and kubelet endpoint for publishing these metrics. - Integrate with the CRI to allow compatibility with a greater number of runtimes, and to create a better runtime abstraction. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/custom-metrics-api.md b/contributors/design-proposals/instrumentation/custom-metrics-api.md index e01848e3..ec944cf7 100644 --- a/contributors/design-proposals/custom-metrics-api.md +++ b/contributors/design-proposals/instrumentation/custom-metrics-api.md @@ -257,7 +257,7 @@ Relationship to HPA v2 ---------------------- The URL paths in this API are designed to correspond to different source -types in the [HPA v2](hpa-v2.md). Specifially, the `pods` source type +types in the [HPA v2](hpa-v2.md). Specifically, the `pods` source type corresponds to a URL of the form `/namespaces/$NS/pods/*/$METRIC_NAME?labelSelector=foo`, while the `object` source type corresponds to a URL of the form @@ -307,7 +307,7 @@ repository will most likely also house other metrics-related APIs for Kubernetes (e.g. historical metrics API definitions, the resource metrics API definitions, etc). -Note that there will not be a canonical implemenation of the custom +Note that there will not be a canonical implementation of the custom metrics API under Kubernetes, just the types and clients. Implementations will be left up to the monitoring pipelines. @@ -323,7 +323,7 @@ overhead, but makes the API line up nicely with other Kubernetes APIs. ### Labeled Metrics ### -Many metric systems support labeled metrics, allowing for dimenisionality +Many metric systems support labeled metrics, allowing for dimensionality beyond the Kubernetes object hierarchy. Since the HPA currently doesn't support specifying metric labels, this is not supported via this API. We may wish to explore this in the future. diff --git a/contributors/design-proposals/instrumentation/events-redesign.md b/contributors/design-proposals/instrumentation/events-redesign.md new file mode 100644 index 00000000..6540d61f --- /dev/null +++ b/contributors/design-proposals/instrumentation/events-redesign.md @@ -0,0 +1,384 @@ +# Make Kubernetes Events useful and safe + +Status: Pending + +Version: Beta + +Implementation Owner: gmarek@google.com + +Approvers: +- [X] thockin - API changes +- [X] briangrant - API changes +- [X] konryd - API changes from UI/UX side +- [X] pszczesniak - logging team side +- [X] wojtekt - performance side +- [ ] derekwaynecarr - "I told you so" Events person:) + +## Overview +This document describes an effort which aims at fixing few issues in current way Events are structured and implemented. This effort has two main goals - reduce performance impact that Events have on the rest of the cluster and add more structure to the Event object which is first and necessary step to make it possible to automate Event analysis. + +This doc combines those two goals in a single effort, which includes both API changes and changes in EventRecorder library. To finish this effort audit of all the Events in the system has to be done, "event style guide" needs to be written, but those are not a part of this proposal. + +Doc starts with more detailed description of the background and motivation for this change. After that introduction we describe our proposal in detail, including both API changes and EventRecorder/deduplication logic updates. Later we consider various effects of this proposal, including performance impact and backward compatibility. We finish with describing considered alternatives and presenting the work plan. + +## Background/motivation +There's a relatively wide agreement that current implementation of Events in Kubernetes is problematic. Events are supposed to give app developer insight into what's happening with his/her app. Important requirement for Event library is that it shouldn't cause/worsen performance problems in the cluster. + +The problem is that neither of those requirements are actually met. Currently Events are extremely spammy (e.g. Event is emitted when Pod is unable to schedule every few seconds) with unclear semantics (e.g. Reason was understood by developers as "reason for taking action" or "reason for emitting event"). Also there are well known performance problems caused by Events (e.g. #47366, #47899) - Events can overload API server if there's something wrong with the cluster (e.g. some correlated crashloop on many Nodes, user created way more Pods that fit on the cluster which fail to schedule repeatedly). This was raised by the community on number of occasions. + +Our goal is to solve both those problems, i.e.: +Update Event semantics such that they'll be considered useful by app developers. +Reduce impact that Events have on the system's performance and stability. + +Those two loosely coupled efforts will drastically improve users experience when they'll need more insight into what's happening with an application. + +In the rest of document I'll shortly characterize both efforts and explain where they interact. + +Style guide for writing Events will be created as a part of this effort and all new Events will need to go through API-like review (process will be described in the style guide). + +### Non goals +It's not a goal of this effort to persist Events outside of etcd or for longer time. + +### Current Event API +Current Event object consists of: +- InvolvedObject (ObjectRef) +- First/LastSeenTimestamp (1s precision time when Event in a given group was first/last seen) +- Reason (short, machine understandable description of what happened that Event was emitted, e.g. ImageNotFound) +- Message (longer description of what happened, e.g. "failed to start a Pod <PodName>" +- Source (component that emits event + its host) +- Standard object stuff (ObjectMeta) +- Type (Normal/Warning) + +Deduplication logic groups together Event which have the same: +- Source Component and Host +- InvolvedObject Kind, Namespace, Name, API version and UID, +- Type (works as Severity) +- Reason + +In particular it ignores Message. It starts grouping events with different messages only after 10 single ones are emitted, which is confusing. + +Current deduplication can be split into two kinds: deduplication happening when "Messages" are different, and one happening when "Messages" are the same. + +First one occurs mostly in Controllers, which create Events for creating single Pods, by putting Pod data inside message. Current Event logic means that we'll create 10 separate Events, with InvolvedObject being, e.g. ReplicationController, and messages saying "RC X created Pod Y" or something equivalent. Afterwards we'll have single Event object with InvolvedObject being the same ReplicationController, but with the message saying "those events were deduped" and count set to size of RC minus 10. Because part of semantics of a given Event is included in the `message` field. + +Deduplication on identical messages can be seen in retry loops. + +### Usability issues with the current API +Users would like to be able to use Events also for debugging and trace analysis of Kubernetes clusters. Current implementation makes it hard for the following reasons: +1s granularity of timestamps (system reacts much quicker than that, making it more or less unusable), +deduplication, that leaves only count and, first and last timestamps (e.g. when Controller is creating a number of Pods information about it is deduplicated), +`InvolvedObject`, `Message`, `Reason` and `Source` semantics are far from obvious. If we treat `Event` as a sentence object of this sentence is stored either in `Message` (if the subject is a Kubernetes object (e.g. Controller)), or in `InvolvedObject`, if the subject is some kind of a controller (e.g. Kubelet). +hard to query for interesting series using standard tools (e.g. all Events mentioning given Pod is pretty much impossible because of deduplication logic) +As semantic information is passed in the message, which in turn is ignored by the deduplication logic it is not clear that this mechanism will not cause deduplication of Events that are completely different. + +## Proposal + +### High level ideas (TL;DR): +When this proposal is implemented users and administrators: +will be able to better track interesting changes in the state of objects they're interested in +will be convinced that Events do not destabilize their clusters + +### API changes goals +We want to achieve following things: +Make it easy to list all interesting Events in common scenarios using kubectl: +Listing Events mentioning given Pod, +Listing Events emitted by a given component (e.g. Kubelet on a given machine, NodeController), +Make timestamps precise enough to allow better events correlation, +Update the field names to better indicate their function. + +### API changes +Make all semantic information about events first-class fields, allowing better deduplication and querying +Add "action" to "reason" to reduce confusion about the semantics of them, +Add "related" field to denote second object taking part in the action, +Increase timestamp precision. + +### Performance changes +"Event series" detection and sending only "series start" and "series finish" Events, +Add more aggressive backoff policy for Events, +API changes +We'd like to propose following structure in Events object in the new events API group: + +```golang +type Event struct { + // <type and object metadata> + + // Time when this Event was first observed. + EventTime metav1.MicroTime + + // Data about the Event series this event represents or nil if it's + // a singleton Event. + // +optional + Series *EventSeries + + // Name of the controller that emitted this Event, e.g. `kubernetes.io/kubelet`. + ReportingController string + + // ID of the controller instance, e.g. `kubelet-xyzf`. + ReportingInstance string + + // What action was taken or what failed regarding the Regarding object. + Action string + + // Why the action was taken or why the operation failed. + Reason string + + // The object this Event is “about”. In most cases it's the object that the + // given controller implements. + // +optional + Regarding ObjectReference + + // Optional secondary object for more complex actions. + // +optional + Related *ObjectReference + + // Human readable description of the Event. Possibly discarded when and + // Event series is being deduplicated. + // +optional + Note string + + // Type of this event (Normal, Warning), new types could be added in the + // future. + // +optional + Type string +} + +type EventSeries struct { + Count int32 + LastObservedTime MicroTime + State EventSeriesState +} + +const ( + EventSeriesStateOngoing = "Ongoing" + EventSeriesStateFinished = "Finished" + EventSeriesStateUnknown = "Unknown" +) +``` + +### Few examples: + +| Regarding | Action | Reason | ReportingController | Related | +| ----------| -------| -------| --------------------|---------| +| Node X | BecameUnreachable | HeartbeatTooOld | kubernetes.io/node-ctrl | <nil> | +| Node Y | FailedToAttachVolume | Unknown | kubernetes.io/pv-attach-ctrl | PVC X | +| ReplicaSet X | FailedToInstantiatePod | QuotaExceeded | kubernetes.io/replica-set-ctrl | <nil> | +| ReplicaSet X | InstantiatedPod | | kubernetes.io/replica-set-ctrl | Pod Y | +| Ingress X | CreatedLoadBalancer | | kubernetes.io/ingress-ctrl | <nil> | +| Pod X | ScheduledOn | | kubernetes.io/scheduler | Node Y | +| Pod X | FailedToSchedule | FitResourcesPredicateFailed | kubernetes.io/scheduler | <nil> | + +### Comparison between old and new API: + +| Old | New | +|-------------|-------------| +| Old Event { | New Event { | +| TypeMeta | TypeMeta | +| ObjectMeta | ObjectMeta | +| InvolvedObject ObjectReference | Regarding ObjectReference | +| | Related *ObjectReference | +| | Action string | +| Reason string | Reason string | +| Message string | Note string | +| Source EventSource | | +| | ReportingController string | +| | ReportingInstance string | +| FirstTimestamp metav1.Time | | +| LastTimestamp metav1.Time | | +| | EventTime metav1.MicroTime | +| Count int32 | | +| | Series EventSeries | +| Type string | Type string | +| } | } | + +Namespace in which Event will live will be equal to +- Namespace of Regarding object, if it's namespaced, +- NamespaceSystem, if it's not. + +Note that this means that if Event has both Regarding and Related objects, and only one of them is namespaced, it should be used as Regarding object. + +The biggest change is the semantics of the Event object in case of loops. If Series is nil it means that Event is a singleton, i.e. it happened only once and the semantics is exactly the same as currently in Events with `count = 1`. If Series is not nil it means that the Event is either beginning or the end of an Event series - equivalence of current Events with `count > 1`. Events for ongoing series have Series.State set to EventSeriesStateOngoing, while endings have Series.State set to EventSeriesStateFinished. + +This change is better described in the section below. + +## Performance improvements design +We want to replace current behavior, where EventRecorder patches Event object every time when deduplicated Event occurs with an approach where being in the loop is treated as a state, hence Events only should be updated only when system enters or exits loop state (or is a singleton Event). + +Because Event object TTL in etcd we can't have above implemented cleanly, as we need to update Event objects periodically to prevent etcd garbage collection from removing ongoing series. We can use this need to update users with new data about number of occurrences. + +The assumption we make for deduplication logic after API changes is that Events with the same <Regarding, Action, Reason, ReportingController, ReportingInstance, Related> tuples are considered isomorphic. This allows us to define notion of "event series", which is series of isomorphic events happening not farther away from each other than some defined threshold. E.g. Events happening every second are considered a series, but Events happening every hour are not. + +The main goal of this change is to limit number of API requests sent to the API server to the minimum. This is important as overloading the API server can severely impact usability of the system. + +In the absence of errors in the system (all Pods are happily running/starting, Nodes are healthy, etc.) the number of Events is easily manageable by the system. This means that it's enough to concentrate on erroneous states and limit number of Events published when something's wrong with the cluster. + +There are two cases to consider: Event series, which result in ~1 API call per ~30 minutes, so won't cause a problem until there's a huge number of them; and huge number of non-series Events. To improve the latter we require that no high-cardinality data are put into any of Regarding, Action, Reason, ReportingController, ReportingInstance, Related fields. Which bound the number of Events to O(number of objects in the system^2). + +## Changes in EventRecorder +EventRecorder is our client library for Events that are used in components to emit Events. The main function in this library is `Eventf`, which takes the data and passes it to the EventRecorder backend, which does deduplication and forwards it to the API server. + +We need to write completely new deduplication logic for new Events, preserving the old one to avoid necessity to rewrite all places when Events are used together with this change. Additionally we need to add a new `Eventf`-equivalent function to the interface that will handle creation of new kind of events. + +New deduplication logic will work in the following way: +- When event is emitted for the first time it's written to the API server without series field set. +- When isomorphic event is emitted within the threshold from the original one EventRecorder detects the start of the series, updates the Event object, with the Series field set carrying count and sets State to EventSeriesStateOngoing. In the EventRecorder it also creates an entry in `activeSeries` map with the timestamp of last observed Event in the series. +- All subsequent isomorphic Events don't result in any API calls, they only update last observed timestamp value and count in the EventRecorder. +- For all active series every 30 minutes EventRecorder will create a "heartbeat" call. Goal of this update is to periodically update user on number of occurrences and prevent garbage collection in etcd. The heartbeat will be an Event update that updates the count and last observed time fields in the series field. +- For all active series every 6 minutes (longer that the longest backoff period) EventRecorder will check if it noticed any attempts to emit isomorphic Event. If there were, it'll check again after aforementioned period (6 minutes). If there weren't it assumes that series is finished and emits closing Event call. This updates the Event by setting state to EventSeriesStateFinished to true and updating the count and last observed time fields in the series field. + +### Short example: +After first occurrence, Event looks like: +``` +{ + regarding: A, + action: B, + reportingController: C, + ..., +} +``` +After second occurrence, Event looks like: +``` +{ + regarding: A, + action: B, + reportingController: C, + ..., + series: {count: 2, state: "Ongoing"}, +} +``` +After half an hour of crashlooping, Event looks like: +``` +{ + regarding: A, + action: B, + reportingController: C, + ..., + series: {count: 4242, state: "Ongoing"}, +} +``` +Minute after crashloop stopped, Event looks like: +``` +{ + regarding: A, + action: B, + reportingController: C, + ..., + series: {count: 424242, state: "Finished"}, +} +``` + +### Client side changes +All clients will need to eventually migrate to use new Events, but no other actions are required from them. Deduplication logic change will be completely transparent after the move to the new API. + +### Restarts +Event recorder library will list all Events emitted by corresponding components and reconstruct internal activeSeries map from it. + +## Defence in depth +Because Events proved problematic we want to add multiple levels of protection in the client library to reduce chances that Events will be overloading API servers in the future. We propose to do two things. + +### Aggressive backoff +We need to make sure that kubernetes client used by EventRecorder uses properly configured and backoff pretty aggressively. Events should not destabilize the cluster, so if EventRecorder receives 429 response it should exponentially back off for non-negligible amount of time, to let API server recover. + +### Other related changes +To allow easier querying we need to make following fields selectable for Events: +- event.reportingComponent +- event.reportingInstance +- event.action +- event.reason +- event.regarding... +- event.related... +- event.type + +Kubectl will need to be updated to use new Events if present. + +## Considerations + +### Performance impact +We're not changing how Events are stored in the etcd (except adding new fields to the storage type). We'll keep current TTL for all Event objects. + +Proposed changes alone will have possibly three effects on performance: we will emit more Events for Pod creation (disable deduplication for "Create Pod" Event emitted by controllers), we will emit fewer Events for hotloops (3 API calls + 1 call/30min per hotloop series, instead of 1/iteration), and Events will be bigger. This means that Event memory footprint will grow slightly, but in the unhealthy looping state number of API calls will be reduced significantly. + +We looked at the amount of memory used in our performance tests in cluster of various size. The results are following: + +| | 5 nodes | 100 nodes | 500 nodes | 5000 nodes | +|-|---------|-----------|-----------|------------| +| event-etcd | 28MB | 65MB | 161MB | n/a | +| All master component | 530MB | 1,2GB | 3,9GB | n/a | +| Excess resources in default config | 3,22GB | 13,8GB | 56,1GB | n/a | + +The difference in size of the Event object comes from new Action and Related fields. We can safely estimate the increase to be smaller than 30%. We'll also emit additional Event per Pod creation, as currently Events for that are being deduplicated. There are currently at least 6 Events emitted when Pod is started, so impact of this change can be bounded by 20%. This means that in the worst case the increase in Event size can be bounded by 56%. As can be seen in the table above we can easily afford such increase. + +### Backward compatibility +Kubernetes API machinery moves towards moving all resources for which it make sense to separate API groups e.g. to allow defining separate storage for it. For this reason we're going to create a new `events` API group in which Event resources will live. + +In the same time we can't stop emitting v1.Events from the Core group as this is considered breaking API change. For this reason we decided to create a new API group for events but map it to the same internal type as core Events. + +As objects are stored in the versioned format we need to add new fields to the Core group, as we're going to use Core group as storage format for new Events. + +After the change we'll have three types of Event objects. Internal representation (denoted internal), "old" core API group type (denoted core) and "new" events API group (denoted events). They will look in the following way - green color denotes added fields: + +| internal.Event | core.Event | events.Event | +|----------------|------------|--------------| +| TypeMeta | TypeMeta | TypeMeta | +| ObjectMeta | ObjectMeta | ObjectMeta | +| InvolvedObject ObjectReference | InvolvedObject ObjectReference | Regarding ObjectReference | +| Related *ObjectReference | Related *ObjectReference | Related *ObjectReference | +| Action string | Action string | Action string | +| Reason string | Reason string | Reason string | +| Message string | Message string | Note string | +| Source.Component string | Source.Component string | ReportingController string | +| Source.Host string | Source.Host string | DeprecatedHost string | +| ReportingInstance string | ReportingInstance string | ReportingInstance string | +| FirstTimestamp metav1.Time | FirstTimestamp metav1.Time | DeprecatedFirstTimestamp metav1.Time | +| LastTimestamp metav1.Time | LastTimestamp metav1.Time | DeprecatedLastTimestamp metav1.Time | +| EventTime metav1.MicroTime | EventTime metav1.MicroTime | EventTime metav1.MicroTime | +| Count int32 | Count int32 | DeprecatedCount int32 | +| Series.Count int32 | Series.Count int32 | Series.Count int32 | +| Series.LastObservedTime | Series.LastObservedTime | Series.LastObservedTime | +| Series.State string | Series.State string | Series.State string | +| Type string | Type string | Type string | + +Considered alternative was to create a separate type that will hold all additional fields in core.Event type. It was dropped, as it's not clear it would help with the clarity of the API. + +There will be conversion functions that'll allow reading/writing Events as both core.Event and events.Event types. As we don't want to officially extend core.Event type, new fields will be set only if Event would be written through events.Event endpoint (e.g. if Event will be created by core.Event endpoint EventTime won't be set). + +This solution gives us clean(-ish) events.Event API and possibility to implement separate storage for Events in the future. The cost is adding more fields to core.Event type. We think that this is not a big price to pay, as the general direction would be to use separate API groups more and core group less in the future. + +`Events` API group will be added directly as beta API, as otherwise kubernetes component's wouldn't be allowed to use it. + +### Sample queries with "new" Events + +#### Get all NodeController Events +List Events from the NamespaceSystem with field selector `reportingController = "kubernetes.io/node-controller"` + +#### Get all Events from lifetime of a given Pod +List all Event with field selector `regarding.name = podName, regarding.namespace = podNamespace`, and `related.name = podName, related.namespace = podNamespace`. You need to join results outside of the kubernetes API. + +### Related work +There's ongoing effort for adding Event deduplication and teeing to the server side. It will allow even easier usage of Events, but in principle it's independent work that should not interfere with one proposed here. + +Another effort to protect API server from too many Events by dropping requests servers side in admission plugin is worked on by @staebler. +## Considered alternatives for API changes +### Leaving current dedup mechanism but improve backoff behavior +As we're going to move all semantic informations to fields, instead of passing some of them in message, we could just call it a day, and leave the deduplication logic as is. When doing that we'd need to depend on the client-recorder library on protecting API server, by using number of techniques, like batching, aggressive backing off and allowing admin to reduce number of Events emitted by the system. This solution wouldn't drastically reduce number of API requests and we'd need to hope that small incremental changes would be enough. + +### Timestamp list as a dedup mechanism +Another considered solution was to store timestamps of Events explicitly instead of only count. This gives users more information, as people complain that current dedup logic is too strong and it's hard to "decompress" Event if needed. This change has clearly worse performance characteristic, but fixes the problem of "decompressing" Events and generally making deduplication lossless. We believe that individual repeated events are not interesting per se, what's interesting is when given series started and when it finished, which is how we ended with the current proposal. + +### Events as an aggregated object +We considered adding nested information about occurrences into the Event. In other words we'd have single Event object per Subject and instead of having only `Count`, we could have stored slice of `timestamp-object` pairs, as a slightly heavier deduplication information. This would have non-negligible impact on size of the event-etcd, and additional price for it would be much harder query logic (querying nested slices is currently not implemented in kubernetes API), e.g. "Give me all Events that refer Pod X" would be hard. + +### Using new API group for storing data +Instead of adding "new" fields to the "old" versioned type, we could have change the version in which we store Events to the new group and use annotations to store "deprecated" fields. This would allow us to avoid having "hybrid" type, as `v1.Events` became, but the change would have a much higher risk (we would have been moving battle-tested and simple `v1.Event` store to new `events.Event` store with some of the data present only in annotations). Additionally performance would degrade, as we'd need to parse JSONs from annotations to get values for "old" fields. +Adding panic button that would stop creation/update of Events +If all other prevention mechanism fail we’d like a way for cluster admin to disable Events in the cluster, to stop them overloading the server. However, we dropped this idea, as it's currently possible to achieve the similar result by changing RBAC rules. + +### Pivoting towards more machine readable Events by introducing stricter structure +We considered making easier for automated systems to use Events by enforcing "active voice" for Event objects. This would allow us to assure which field in the Event points to the active component, and which to direct and indirect objects. We dropped this idea because Events are supposed to be consumed only by humans. + +### Pivoting towards making Events more helpful for cluster operator during debugging +We considered exposing more data that cluster operator would need to use Events for debugging, e.g. making ReportingController more central to the semantics of Event and adding some way to easily grep though the logs of appropriate component when looking for context of a given Event. This idea was dropped because Events are supposed to give application developer who's running his application on the cluster a rough understanding what was happening with his app. + +## Proposed implementation plan +- 1.9 - create a beta api-group Events with the new Event type, +- 1.10 - migrate at controllers running in controller manager to use events API group, +- 1.11 - finish migrating Events in all components and move and move events storage representation to the new type. diff --git a/contributors/design-proposals/metrics-server.md b/contributors/design-proposals/instrumentation/metrics-server.md index 80cefaf9..344addf6 100644 --- a/contributors/design-proposals/metrics-server.md +++ b/contributors/design-proposals/instrumentation/metrics-server.md @@ -5,7 +5,7 @@ Resource Metrics API is an effort to provide a first-class Kubernetes API (stable, versioned, discoverable, available through apiserver and with client support) that serves resource usage metrics for pods and nodes. The use cases were discussed and the API was proposed a while ago in -[another proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md). +[another proposal](/contributors/design-proposals/instrumentation/resource-metrics-api.md). This document describes the architecture and the design of the second part of this effort: making the mentioned API available in the same way as the other Kubernetes APIs. @@ -43,18 +43,18 @@ Previously metrics server was blocked on this dependency. ### Design ### Metrics server will be implemented in line with -[Kubernetes monitoring architecture](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/monitoring_architecture.md) +[Kubernetes monitoring architecture](/contributors/design-proposals/instrumentation/monitoring_architecture.md) and inspired by [Heapster](https://github.com/kubernetes/heapster). It will be a cluster level component which periodically scrapes metrics from all Kubernetes nodes served by Kubelet through Summary API. Then metrics will be aggregated, stored in memory (see Scalability limitations) and served in -[Metrics API](https://github.com/kubernetes/metrics/blob/master/pkg/apis/metrics/v1alpha1/types.go) format. +[Metrics API](https://git.k8s.io/metrics/pkg/apis/metrics/v1alpha1/types.go) format. Metrics server will use apiserver library to implement http server functionality. The library offers common Kubernetes functionality like authorization/authentication, versioning, support for auto-generated client. To store data in memory we will replace the default storage layer (etcd) by introducing in-memory store which will implement -[Storage interface](https://github.com/kubernetes/apiserver/blob/master/pkg/registry/rest/rest.go). +[Storage interface](https://git.k8s.io/apiserver/pkg/registry/rest/rest.go). Only the most recent value of each metric will be remembered. If a user needs an access to historical data they should either use 3rd party monitoring solution or @@ -71,13 +71,13 @@ due to security reasons (our policy allows only connection in the opposite direc There will be only one instance of metrics server running in each cluster. In order to handle high metrics volume, metrics server will be vertically autoscaled by -[addon-resizer](https://github.com/kubernetes/contrib/tree/master/addon-resizer). +[addon-resizer](https://git.k8s.io/contrib/addon-resizer). We will measure its resource usage characteristic. Our experience from profiling Heapster shows that it scales vertically effectively. If we hit performance limits we will consider scaling it horizontally, though it’s rather complicated and is out of the scope of this doc. Metrics server will be Kubernetes addon, create by kube-up script and managed by -[addon-manager](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/addon-manager). +[addon-manager](https://git.k8s.io/kubernetes/cluster/addons/addon-manager). Since there is a number of dependant components, it will be marked as a critical addon. In the future when the priority/preemption feature is introduced we will migrate to use this proper mechanism for marking it as a high-priority, system component. diff --git a/contributors/design-proposals/monitoring_architecture.md b/contributors/design-proposals/instrumentation/monitoring_architecture.md index 5def02b5..1b4cd206 100644 --- a/contributors/design-proposals/monitoring_architecture.md +++ b/contributors/design-proposals/instrumentation/monitoring_architecture.md @@ -98,7 +98,7 @@ These sources are scraped by a component we call *metrics-server* which is like version of today's Heapster. metrics-server stores locally only latest values and has no sinks. metrics-server exposes the master metrics API. (The configuration described here is similar to the current Heapster in “standalone” mode.) -[Discovery summarizer](aggregated-api-servers.md) +[Discovery summarizer](../api-machinery/aggregated-api-servers.md) makes the master metrics API available to external clients such that from the client's perspective it looks the same as talking to the API server. @@ -196,8 +196,3 @@ Prometheus format.  - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/monitoring_architecture.png b/contributors/design-proposals/instrumentation/monitoring_architecture.png Binary files differindex 570996b7..570996b7 100644 --- a/contributors/design-proposals/monitoring_architecture.png +++ b/contributors/design-proposals/instrumentation/monitoring_architecture.png diff --git a/contributors/design-proposals/performance-related-monitoring.md b/contributors/design-proposals/instrumentation/performance-related-monitoring.md index 7937dbdd..f2b75813 100644 --- a/contributors/design-proposals/performance-related-monitoring.md +++ b/contributors/design-proposals/instrumentation/performance-related-monitoring.md @@ -32,7 +32,7 @@ how careful we need to be. ### Huge number of handshakes slows down API server It was a long standing issue for performance and is/was an important bottleneck for scalability (https://github.com/kubernetes/kubernetes/issues/13671). The bug directly -causing this problem was incorrect (from the golangs standpoint) handling of TCP connections. Secondary issue was that elliptic curve encryption (only one available in go 1.4) +causing this problem was incorrect (from the golang's standpoint) handling of TCP connections. Secondary issue was that elliptic curve encryption (only one available in go 1.4) is unbelievably slow. ## Proposed metrics/statistics to gather/compute to avoid problems @@ -42,8 +42,8 @@ is unbelievably slow. Basic ideas: - number of Pods/ReplicationControllers/Services in the cluster - number of running replicas of master components (if they are replicated) -- current elected master of ectd cluster (if running distributed version) -- nuber of master component restarts +- current elected master of etcd cluster (if running distributed version) +- number of master component restarts - number of lost Nodes ### Logging monitoring @@ -57,7 +57,7 @@ Basic ideas: ### REST call monitoring We do measure REST call duration in the Density test, but we need an API server monitoring as well, to avoid false failures caused e.g. by the network traffic. We already have -some metrics in place (https://github.com/kubernetes/kubernetes/blob/master/pkg/apiserver/metrics/metrics.go), but we need to revisit the list and add some more. +some metrics in place (https://git.k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go), but we need to revisit the list and add some more. Basic ideas: - number of calls per verb, client, resource type @@ -110,7 +110,3 @@ We should monitor other aspects of the system, which may indicate saturation of Basic ideas: - queue length for queues in the system, - wait time for WaitGroups. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/resource-metrics-api.md b/contributors/design-proposals/instrumentation/resource-metrics-api.md index fee416e0..075c6180 100644 --- a/contributors/design-proposals/resource-metrics-api.md +++ b/contributors/design-proposals/instrumentation/resource-metrics-api.md @@ -20,7 +20,7 @@ Use cases which are not listed below are out of the scope of MVP version of Reso HPA uses the latest value of cpu usage as an average aggregated across 1 minute (the window may change in the future). The data for a given set of pods -(defined either by pod list or label selector) should be accesible in one request +(defined either by pod list or label selector) should be accessible in one request due to performance issues. #### Scheduler @@ -72,7 +72,7 @@ directly using some custom API there. ## Proposed API -Initially the metrics API will be in a separate [API group](api-group.md) called ```metrics```. +Initially the metrics API will be in a separate [API group](../api-machinery/api-group.md) called ```metrics```. Later if we decided to have Node and Pod in different API groups also NodeMetrics and PodMetrics should be in different API groups. @@ -146,6 +146,3 @@ Depending on the further requirements the following features may be added: - possibility to query for window sizes and aggregation functions (though single window size/aggregation function per request) - cluster level metrics -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/instrumentation/volume_stats_pvc_ref.md b/contributors/design-proposals/instrumentation/volume_stats_pvc_ref.md new file mode 100644 index 00000000..1b2f599b --- /dev/null +++ b/contributors/design-proposals/instrumentation/volume_stats_pvc_ref.md @@ -0,0 +1,57 @@ +# Add PVC reference in Volume Stats + +## Background +Pod volume stats tracked by kubelet do not currently include any information about the PVC (if the pod volume was referenced via a PVC) + +This prevents exposing (and querying) volume metrics labeled by PVC name which is preferable for users, given that PVC is a top-level API object. + +## Proposal + +Modify ```VolumeStats``` tracked in Kubelet and populate with PVC info: + +``` +// VolumeStats contains data about Volume filesystem usage. +type VolumeStats struct { + // Embedded FsStats + FsStats + // Name is the name given to the Volume + // +optional + Name string `json:"name,omitempty"` ++ // PVCRef is a reference to the measured PVC. ++ // +optional ++ PVCRef PVCReference `json:"pvcRef"` +} + ++// PVCReference contains enough information to describe the referenced PVC. ++type PVCReference struct { ++ Name string `json:"name"` ++ Namespace string `json:"namespace"` ++} +``` + +## Implementation +2 options are described below. Option 1 supports current requirements/requested use cases. Option 2 supports an additional use case that was being discussed and is called out for completeness/discussion/feedback. + +### Option 1 +- Modify ```kubelet::server::stats::calcAndStoreStats()``` + - If the pod volume is referenced via a PVC, populate ```PVCRef``` in VolumeStats using the Pod spec + + - The Pod spec is already available in this method, so the changes are contained to this function. + +- The limitation of this approach is that we're limited to reporting only what is available in the pod spec (Pod namespace and PVC claimname) + +### Option 2 +- Modify the ```volumemanager::GetMountedVolumesForPod()``` (or add a new function) to return additional volume information from the actual/desired state-of-world caches + - Use this to populate PVCRef in VolumeStats + +- This allows us to get information not available in the Pod spec such as the PV name/UID which can be used to label metrics - enables exposing/querying volume metrics by PV name +- It's unclear whether this is a use case we need to/should support: + * Volume metrics are only refreshed for mounted volumes which implies a bound/available PVC + * We expect most user-storage interactions to be via the PVC +- Admins monitoring PVs (and not PVC's) so that they know when their users are running out of space or are over-provisioning would be a use case supporting adding PV information to + metrics + + + + + diff --git a/contributors/design-proposals/multi-platform.md b/contributors/design-proposals/multi-platform.md index dfc04f95..923472e6 100644 --- a/contributors/design-proposals/multi-platform.md +++ b/contributors/design-proposals/multi-platform.md @@ -73,7 +73,7 @@ This is a fairly long topic. If you're interested how to cross-compile, see [det The easiest way of running Kubernetes on another architecture at the time of writing is probably by using the docker-multinode deployment. Of course, you may choose whatever deployment you want, the binaries are easily downloadable from the URL above. -[docker-multinode](https://github.com/kubernetes/kube-deploy/tree/master/docker-multinode) is intended to be a "kick-the-tires" multi-platform solution with Docker as the only real dependency (but it's not production ready) +[docker-multinode](https://git.k8s.io/kube-deploy/docker-multinode) is intended to be a "kick-the-tires" multi-platform solution with Docker as the only real dependency (but it's not production ready) But when we (`sig-cluster-lifecycle`) have standardized the deployments to about three and made them production ready; at least one deployment should support **all platforms**. @@ -100,7 +100,7 @@ When it's possible to test Kubernetes using Kubernetes; volunteers should be giv ### Official support level When all e2e tests are passing for a given platform; the platform should be officially supported by the Kubernetes team. -At the time of writing, `amd64` is in the officially supported category category. +At the time of writing, `amd64` is in the officially supported category. When a platform is building and it's possible to set up a cluster with the core functionality, the platform is supported on a "best-effort" and experimental basis. At the time of writing, `arm`, `arm64` and `ppc64le` are in the experimental category; the e2e tests aren't cross-platform yet. @@ -121,7 +121,7 @@ For reference see [docker/docker#24739](https://github.com/docker/docker/issues/ This has been debated quite a lot about; how we should name non-amd64 docker images that are pushed to `gcr.io`. See [#23059](https://github.com/kubernetes/kubernetes/pull/23059) and [#23009](https://github.com/kubernetes/kubernetes/pull/23009). -This means that the naming `gcr.io/google_containers/${binary}:${version}` should contain a _manifest list_ for future tags. +This means that the naming `k8s.gcr.io/${binary}:${version}` should contain a _manifest list_ for future tags. The manifest list thereby becomes a wrapper that is pointing to the `-${arch}` images. This requires `docker-1.10` or newer, which probably means Kubernetes v1.4 and higher. @@ -140,7 +140,7 @@ Also, [the apiserver now exposes](https://github.com/kubernetes/kubernetes/pull/ ### Standardize all image Makefiles to follow the same pattern All Makefiles should push for all platforms when doing `make push`, and build for all platforms when doing `make build`. -Under the hood; they should compile binaries in a container for reproducability, and use QEMU for emulating Dockerfile `RUN` commands if necessary. +Under the hood; they should compile binaries in a container for reproducibility, and use QEMU for emulating Dockerfile `RUN` commands if necessary. ### Remove linux/amd64 hard-codings from the codebase @@ -215,7 +215,7 @@ Go 1.5 introduced many changes. To name a few that are relevant to Kubernetes: All release notes for Go 1.5 [are here](https://golang.org/doc/go1.5) -Go 1.6 didn't introduce as many changes as Go 1.5 did, but here are some of note: +Go 1.6 didn't introduce as many changes as Go 1.5 did, but here are some of notes: - It should perform a little bit better than Go 1.5. - `linux/mips64` and `linux/mips64le` were added as new ports. - Go < 1.6.2 for `ppc64le` had [bugs in it](https://github.com/kubernetes/kubernetes/issues/24922). @@ -377,7 +377,7 @@ In order to dynamically compile a go binary with `cgo`, we need `gcc` installed The only Kubernetes binary that is using C code is the `kubelet`, or in fact `cAdvisor` on which `kubelet` depends. `hyperkube` is also dynamically linked as long as `kubelet` is. We should aim to make `kubelet` statically linked. -The normal `x86_64-linux-gnu` can't cross-compile binaries, so we have to install gcc cross-compilers for every platform. We do this in the [`kube-cross`](https://github.com/kubernetes/kubernetes/blob/master/build/build-image/cross/Dockerfile) image, +The normal `x86_64-linux-gnu` can't cross-compile binaries, so we have to install gcc cross-compilers for every platform. We do this in the [`kube-cross`](https://git.k8s.io/kubernetes/build/build-image/cross/Dockerfile) image, and depend on the [`emdebian.org` repository](https://wiki.debian.org/CrossToolchains). Depending on `emdebian` isn't ideal, so we should consider using the latest `gcc` cross-compiler packages from the `ubuntu` main repositories in the future. Here's an example when cross-compiling plain C code: @@ -471,7 +471,7 @@ CMD ["/usr/local/bin/kube-apiserver"] ```console $ file kube-apiserver kube-apiserver: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped -$ docker build -t gcr.io/google_containers/kube-apiserver-arm:v1.x.y . +$ docker build -t k8s.gcr.io/kube-apiserver-arm:v1.x.y . Step 1 : FROM armel/busybox ---> 9bb1e6d4f824 Step 2 : ENV kubernetes true @@ -527,6 +527,3 @@ The 30th of November 2015, a tracking issue about making Kubernetes run on ARM w The 27th of April 2016, Kubernetes `v1.3.0-alpha.3` was released, and it became the first release that was able to run the [docker getting started guide](http://kubernetes.io/docs/getting-started-guides/docker/) on `linux/amd64`, `linux/arm`, `linux/arm64` and `linux/ppc64le` without any modification. -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/multicluster/OWNERS b/contributors/design-proposals/multicluster/OWNERS new file mode 100644 index 00000000..fca0e564 --- /dev/null +++ b/contributors/design-proposals/multicluster/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-multicluster-leads +approvers: + - sig-multicluster-leads +labels: + - sig/multicluster diff --git a/contributors/design-proposals/multicluster/cluster-registry/api-design.md b/contributors/design-proposals/multicluster/cluster-registry/api-design.md new file mode 100644 index 00000000..2133f499 --- /dev/null +++ b/contributors/design-proposals/multicluster/cluster-registry/api-design.md @@ -0,0 +1,164 @@ +# Cluster Registry API + +@perotinus, @madhusudancs + +Original draft: 08/16/2017 + +**Reviewed** in SIG multi-cluster meeting on 8/29 + +*This doc is a Markdown conversion of the original Cluster Registry API +[Google doc](https://docs.google.com/document/d/1Oi9EO3Jwtp69obakl-9YpLkP764GZzsz95XJlX1a960/edit). +That doc is deprecated, and this one is canonical; however, the old doc will be +preserved so as not to lose comment and revision history that it contains.* +## Table of Contents + +- [Purpose](#purpose) +- [Motivating use cases](#motivating-use-cases) +- [API](#api) +- [Authorization-based filtering of the list of clusters](#authorization-based-filtering-of-the-list-of-clusters) +- [Status](#status) +- [Auth](#auth) +- [Key differences vs existing Federation API `Cluster` object](#key-differences-vs-existing-federation-api-cluster-object) +- [Open questions](#open-questions) + +## Purpose + +The cluster registry API is intended to provide a common abstraction for other +tools that will perform operations on multiple clusters. It provides an +interface to a list of objects that will store metadata about clusters that can +be used by other tools. The cluster registry implementation is meant to remain +simple: we believe there is benefit in defining a common layer that can be used +by many different tools to solve different kinds of multi-cluster problems. + +It may be helpful to consider this API as an extension of the `kubeconfig` file. +The `kubeconfig` file contains a list of clusters with the auth data necessary +for kubectl to access them; the cluster registry API intends to provide this +data, plus some additional useful metadata, from a remote location instead of +from the user's local machine. + +## Motivating use cases + +These were presented at the SIG-Federation F2F meeting on 8/4/17 +([Atlassian](https://docs.google.com/document/d/1PH859COCWSkRxILrQd6wDdYLGJaBtWQkSN3I-Lnam3g/edit#heading=h.suxgoa67n1aw), +[CoreOS](https://docs.google.com/presentation/d/1InJagQNOxqA0ftK0peJLzyEFU2IZEXrJprDN6fcleMg/edit#slide=id.p), +[Google](https://docs.google.com/presentation/d/1Php_HnHI-Sy20ieyd_jBgr7XTs0fKT0Cq9z6dC4zOMc/), +[RedHat](https://docs.google.com/presentation/d/1dExjeSQTXI8_k00nqXRkSIFPTkzAzUTFtETU4Trg5yw/edit#slide=id.p)). +Each of the use cases presented assumes the ability to access a registry of +clusters, and so all are valid motivating use cases for the cluster registry +API. Note that these use cases will generally require more tooling than the +cluster registry itself. The cluster registry API will support what these other +tools will need in order to operate, but will not intrinsically support these +use cases. + +- Consistent configuration across clusters/replication of resources +- Federated Ingress: load balancing across multiple clusters, potentially + geo-aware +- Multi-cluster application distribution, with policy/balancing +- Disaster recovery/failover +- Human- and tool- parseable interaction with a list of clusters +- Monitoring/health checking/status reporting for clusters, potentially with + dashboards +- Policy-based and jurisdictional placement of workloads + +## API + +This document defines the cluster registry API. It is an evolution of the +[current Federation cluster API](https://git.k8s.io/federation/apis/federation/types.go#L99), +and is designed more specifically for the "cluster registry" use case in +contrast to the Federation `Cluster` object, which was made for the +active-control-plane Federation. + +The API is a Kubernetes-style REST API that supports the following operations: + +1. `POST` - to create new objects. +1. `GET` - to retrieve both lists and individual objects. +1. `PUT` - to update or create an object. +1. `DELETE` - to delete an object. +1. `PATCH` - to modify the fields of an object. + +Optional API operations: + +1. `WATCH` - to receive a stream of changes made to a given object. As `WATCH` + is not a standard HTTP method, this operation will be implemented as `GET + /<resource>&watch=true`. We believe that it's not always necessary to + support WATCH for this API. Implementations can choose to support or not + support this operation. An implementation that does not support the + operation should return HTTP error 405, StatusMethodNotAllowed, per the + [relevant Kubernetes API conventions](/contributors/devel/api-conventions.md#error-codes). + +We also intend to support a use case where the server returns a file that can be +stored for later use. We expect this to be doable with the standard API +machinery; and if the API is implemented not using the Kubernetes API machinery, +that the returned file must be interoperable with the response from a Kubernetes +API server. + +[The API](https://git.k8s.io/cluster-registry/pkg/apis/clusterregistry/v1alpha1/types.go) +is defined in the cluster registry repo, and is not replicated here in order to +avoid mismatches. + +All top-level objects that define resources in Kubernetes embed a +`meta.ObjectMeta` that in-turn contains a number of fields. All the fields in +that struct are potentially useful with the exception of the `ClusterName` and +the `Namespace` fields. Having a `ClusterName` field alongside a `Name` field in +the cluster registry API will be confusing to our users. Therefore, in the +initial API implementation, we will add validation logic that rejects `Cluster` +objects that contain a value for the `ClusterName` field. The `Cluster` object's +`Namespace` field will be disabled by making the object be root scoped instead +of namespace scoped. + +The `Cluster` object will have `Spec` and `Status` fields, following the +[Kubernetes API conventions](/contributors/devel/api-conventions.md#spec-and-status). +There was argument in favor of a `State` field instead of `Spec` and `Status` +fields, since the `Cluster` in the registry does not necessarily hold a user's +intent about the cluster being represented, but instead may hold descriptive +information about the cluster and information about the status of the cluster; +and because the cluster registry provides no controller that performs +reconciliation on `Cluster` objects. However, after +[discussion with SIG-arch](https://groups.google.com/forum/#!topic/kubernetes-sig-architecture/ptK2mVtha38), +the decision was made in favor of spec and status. + +## Authorization-based filtering of the list of clusters + +The initial version of the cluster registry supports a cluster list API that +does not take authorization rules into account. It returns a list of clusters +similar to how other Kubernetes List APIs list the objects in the presence of +RBAC rules. A future version of this API will take authorization rules into +account and only return the subset of clusters a user is authorized to access in +the registry. + +## Status + +There are use cases for the cluster registry that call for storing status that +is provided by more active controllers, e.g. health checks and cluster capacity. +At this point, these use cases are not as well-defined as the use cases that +require a data store, and so we do not intend to propose a complete definition +for the `ClusterStatus` type. We recognize the value of conventions, so as these +use cases become more clearly defined, the API of the `ClusterStatus` will be +extended appropriately. + +## Auth + +The cluster registry API will not provide strongly-typed objects for returning +auth info. Instead, it will provide a generic type that clients can use as they +see fit. This is intended to mirror what `kubectl` does with its +[AuthProviderConfig](https://git.k8s.io/client-go/tools/clientcmd/api/types.go#L144). +As open standards are developed for cluster auth, the API can be extended to +provide first-class support for these. We want to avoid baking non-open +standards into the API, and so having to support potentially a multiplicity of +them as they change. The cluster registry itself is not intended to be a +credential store, but instead to provide "pointers" that will provide the +information needed by callers to authenticate to a cluster. There is some more +context +[here](https://docs.google.com/a/google.com/document/d/1cxKV4Faywsn_to49csN0S0TZLYuHgExusEsmgKQWc28/edit?usp=sharing). + +## Key differences vs existing Federation API `Cluster` object + +- Active controller is not required; the registry can be used without any + controllers +- `WATCH` support is not required + +## Open questions + +All open questions have been +[migrated](https://github.com/kubernetes/cluster-registry/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20%22Migrated%20from%20the%20Cluster%22) +to issues in the cluster registry repo. diff --git a/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md b/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md new file mode 100644 index 00000000..09685ffd --- /dev/null +++ b/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md @@ -0,0 +1,335 @@ +# Cluster registry design and plan + +@perotinus + +Updated: 11/2/17 + +*REVIEWED* in SIG-multicluster meeting on 10/24 + +*This doc is a Markdown conversion of the original Cluster registry design and +plan +[Google doc](https://docs.google.com/document/d/1bVvq9lDIbE-Glyr6GkSGWYkLb2cCNk9bR8LL7Wm-L6g/edit). +That doc is deprecated, and this one is canonical; however, the old doc will be +preserved so as not to lose comment and revision history that it contains.* + +## Table of Contents + +- [Background](#background) +- [Goal](#goal) +- [Technical requirements](#technical-requirements) + - [Alpha](#alpha) + - [Beta](#beta) + - [Later](#later) +- [Implementation design](#implementation-design) + - [Alternatives](#alternatives) + - [Using a CRD](#using-a-crd) +- [Tooling design](#tooling-design) + - [User tooling](#user-tooling) +- [Repository process](#repository-process) +- [Release strategy](#release-strategy) + - [Version skew](#version-skew) +- [Test strategy](#test-strategy) +- [Milestones and timelines](#milestones-and-timelines) + - [Alpha (targeting late Q4 '17)](#alpha-targeting-late-q4-'17) + - [Beta (targeting mid Q1 '18)](#beta-targeting-mid-q1-'18) + - [Stable (targeting mid Q2 '18)](#stable-targeting-mid-q2-'18) + - [Later](#later) + +## Background + +SIG-multicluster has identified a cluster registry as being a key enabling +component for multi-cluster use cases. The goal of the SIG in this project is +that the API defined for the cluster registry become a standard for +multi-cluster tools. The API design is being discussed in +[a separate doc](https://drive.google.com/a/google.com/open?id=1Oi9EO3Jwtp69obakl-9YpLkP764GZzsz95XJlX1a960). +A working prototype of the cluster registry has been assembled in +[a new repository](https://github.com/kubernetes/cluster-registry). + +## Goal + +This document intends to lay out an initial plan for moving the cluster registry +from the prototype state through alpha, beta and eventually to a stable release. + +## Technical requirements + +These requirements are derived mainly from the output of the +[August 9th multi-cluster SIG meeting](https://docs.google.com/document/d/11cB3HK67BZUb7aNOCpQK8JsA8Na8F-F_6bFu77KrKT4/edit#heading=h.cuvqls7pl9qc). +However, they also derive (at least indirectly) from the results of the +[SIG F2F meeting ](https://docs.google.com/document/d/1HkVBSm9L9UJC2f3wfs_8zt1PJmv6iepdtJ2fmkCOHys/edit) +that took place earlier, as well as the use cases presented by various parties +at that meeting. + +### Alpha + +- [API] Provides an HTTP server with a Kubernetes-style API for CRUD + operations on a registry of clusters + - Provides a way to filter across clusters by label + - Supports annotations +- [API] Provides information about each cluster's API server's + location/endpoint +- [API] Provides the ability to assign user-friendly names for clusters +- [API] Provides pointers to authentication information for clusters' API + servers + - This implies that it does not support storing credentials directly +- [Implementation] Supports both independent and aggregated deployment models + - Supports delegated authn/authz in aggregated mode + - Supports integration into Federation via aggregated deployment model + +### Beta + +- Supports providing a flat file of clusters for storage and later use + - This may be provided by the ecosystem rather than the registry API + implementation directly +- Supports independent authz for reading/writing clusters +- `kubectl` integration with the cluster registry API server is first-class + and on par with kubectl integration with core Kubernetes APIs +- Supports grouping of clusters +- Supports specifying and enforcing read and/or write authorization for groups + of clusters in the registry +- Working Federation integration +- Supports status from active controllers + +### Later + +- Supports an HA deployment strategy +- Supports guarantees around immutability/identity of clusters in list +- Version skew between various components is understood and supported skews + are defined and tested + +## Implementation design + +The cluster registry will be implemented using the +[Kubernetes API machinery](https://github.com/kubernetes/apimachinery). The +cluster registry API server will be a fork and rework of the existing Federation +API server, scaled down and simplified to match the simpler set of requirements +for the cluster registry. It will use the +[apiserver](https://github.com/kubernetes/apiserver) library, plus some code +copied from the core Kubernetes repo that provides scaffolding for certain +features in the API server. This is currently implemented in a prototype form +in the [cluster-registry repo](https://github.com/kubernetes/cluster-registry). + +The API will be implemented using the Kubernetes API machinery, as a new API +with two objects, `Cluster` and `ClusterList`. Other APIs may be added in the +future to support future use cases, but the intention is that the cluster +registry API server remain minimal and only provide the APIs that users of a +cluster registry would want for the cluster registry. + +The cluster registry will not be responsible for storing secrets. It will +contain pointers to other secret stores which will need to be implemented +independently. The cluster registry API will not provide proxy access to +clusters, and will not interact with clusters on a user's behalf. Storing secret +information in the cluster registry will be heavily discouraged by its +documentation, and will be considered a misuse of the registry. This allows us +to sidestep the complexity of implementing a secure credential storage. + +The expectation is that Federation and other programs will use the +cluster-registry as an aggregated API server rather than via direct code +integration. Therefore, the cluster registry will explicitly support being +deployed as an API server that can be aggregated by other API servers. + +### Alternatives + +#### Using a CRD + +The cluster registry could be implemented as a CRD that is registered with a +Kubernetes API server. This implementation is lighter weight than running a full +API server. If desired, the administrator could then disable the majority of the +Kubernetes APIs for non-admin users and so make it appear as if the API server +only supports cluster objects. It should be possible for a user to migrate +without much effort from a CRD-based to an API-server-based implementation of +the cluster registry, but the cluster-registry project is not currently planning +to spend time supporting this use case. CRDs do not support (and may never +support) versioning, which is very desirable for the cluster registry API. Users +who wish to use a CRD implementation will have to design and maintain it +themselves. + +## Tooling design + +### User tooling + +In the alpha stage, the cluster registry will repurpose the kubefed tool from +Federation and use it to initialize a cluster registry. In early stages, this +will only create a deployment with one replica, running the API server and etcd +in a `Pod`. As the HA requirements for the cluster registry are fleshed out, +this tool may need to be updated or replaced to support deploying a cluster +registry in multiple clusters and with multiple etcd replicas. + +Since the cluster registry is a Kubernetes API server that serves a custom +resource type, it will be usable by `kubectl`. We expect that the kubectl +experience for custom APIs will soon be on par with that of core Kubernetes +APIs, since there has been a significant investment in making `kubectl` provide +very detailed output from its describe subcommand; and `kubectl` 1.9 is expected +to introduce API server-originated columns. Therefore, we will not initially +implement any special tooling for interacting with the registry, and will tell +users to use `kubectl` or generated client libraries. + +## Repository process + +The cluster registry is a top-level Kubernetes repository, and thus it requires +some process to ensure stability for dependent projects and accountability. +Since the Federation project wants to use the cluster registry instead of its +current cluster API, there is a requirement to establish process even though the +project is young and does not yet have a lot of contributors. + +The standard Kubernetes Prow bots have been enabled on the repo. There is +currently some functionality around managing PR approval and reviewer assignment +and such that does not yet live in Prow, but given the limited number of +contributors at this point it seems reasonable to wait for sig-testing to +implement this functionality in Prow rather than enabling the deprecated tools. +In most cases where a process is necessary, we will defer to the spirit of the +Kubernetes process (if not the letter) though we will modify it as necessary for +the scope and scale of the cluster registry project. There is not yet a merge +queue, and until there is a clear need for one we do not intend to add one. + +The code in the repository will use bazel as its build system. This is in-line +with what the Kubernetes project is attempting to move towards, and since the +cluster registry has a similar but more limited set of needs than Kubernetes, we +expect that bazel will support our needs adequately. The structure that bazel +uses is meant to be compatible with go tooling, so if necessary, we can migrate +away from bazel in the future without having to entirely revamp the repository +structure. + +There is not currently a good process in Kubernetes for keeping vendored +dependencies up-to-date. The strategy we expect to take with the cluster +registry is to update only when necessary, and to use the same dependency +versions that Kubernetes uses unless there is some particular incompatibility. + +## Release strategy + +For early versions, the cluster registry release will consist of a container +image and a client tool. The container image will contain the cluster registry +API server, and the tool will be used to bootstrap this image plus an etcd image +into a running cluster. The container will be published to GCR, and the client +tool releases will be stored in a GCS bucket, following a pattern used by other +k/ projects. + +For now, the release process will be managed mostly manually by repository +maintainers. We will create a release branch that will be used for releases, and +use GitHub releases along with some additional per-release documentation +(`CHANGELOG`, etc). `CHANGELOG`s and release notes will be collected manually +until the volume of work becomes too great. We do not intend to create multiple +release branches until the project is more stable. Releases will undergo a +more-detailed set of tests that will ensure compatibility with recent released +versions of `kubectl` (for the registry) and Kubernetes (for the cluster +registry as an aggregated API server). Having a well-defined release process +will be a stable release requirement, and by that point we expect to have gained +some practical experience that will make it easier to codify the requirements +around doing releases. The cluster registry will use semantic versioning, but +its versions will not map to Kubernetes versions. Cluster registry releases will +not follow the Kubernetes release cycle, though Kubernetes releases may trigger +cluster registry releases if there are compatibility issues that need to be +fixed. + +Projects that want to vendor the cluster registry will be able to do so. We +expect that these projects will vendor from the release branch if they want a +stable version, or from a desired SHA if they are comfortable using a version +that has not necessarily been fully vetted. + +As the project matures, we expect the tool to evolve (or be replaced) in order +to support deployment against an existing etcd instance (potentially provided by +an etcd operator), and to provide a HA story for hosting a cluster registry. +This is considered future work and will not be addressed directly in this +document. + +Cross-compilation support in bazel is still a work in progress, so the cluster +registry will not be able to easily provide binary releases for every platform +until this is supported by bazel. If it becomes necessary to provide +cross-platform binaries before bazel cross-compilation is available, the +repository is setup to support common go tooling, so we should be able to devise +a process for doing so. + +### Version skew + +There are several participants in the cluster registry ecosystem whose versions +will be conceptually able to float relative to each other: + +- The cluster registry API server +- The host cluster's API server +- `kubectl` + +We will need to define the version skew restraints between these components and +ensure that our testing validates key skews that we care about. + +## Test strategy + +The cluster registry is a simple component, and so should be able to be tested +extensively without too much difficulty. The API machinery code is already +tested by its owning teams, and since the cluster registry is a straightforward +API server, it should not require much testing of its own. The bulk of the tests +written will be integration tests, to ensure that it runs correctly in +aggregated and independent modes in a Kubernetes cluster; to verify that various +versions of kubectl can interact with it; and to verify that it can be upgraded +safely. A full testing strategy is a requirement for a GA launch; we expect +development of a test suite to be an ongoing effort in the early stages of +development. + +The command-line tool will require testing. It should be E2E tested against +recent versions of Kubernetes, to ensure that a simple cluster registry can be +created in a Kubernetes cluster. The multiplicity of configuration options it +provides cannot conveniently be tested in E2E tests, and so will be validated in +unit tests. + +## Milestones and timelines + +### Alpha (targeting late Q4 '17) + +- Test suite is running on each PR, with reasonable unit test coverage and + minimal integration/E2E testing +- Repository processes (OWNERship, who can merge, review lag standards, + project planning, issue triaging, etc.) established +- Contributor documentation written +- User documentation drafted +- Cluster registry API also alpha +- All Alpha technical requirements met + +### Beta (targeting mid Q1 '18) + +- Full suite of integration/E2E tests running on each PR +- API is beta +- Preparatory tasks for GA +- All Beta technical requirements met +- User documentation complete and proofed, content-wise +- Enough feedback solicited from users, or inferred from download + statistics/repository issues + +### Stable (targeting mid Q2 '18) + +- Fully fleshed-out user documentation +- User documentation is published in a finalized location +- First version of API is GA +- Documented upgrade test procedure, with appropriate test tooling implemented +- Plan for and documentation about Kubernetes version support (e.g., which + versions of Kubernetes a cluster registry can be a delegated API server + with) +- Releases for all platforms +- Well-defined and documented release process + +### Later + +- Documented approach to doing a HA deployment +- Work on Later technical requirements + +## Questions + +- How does RBAC work in this case? +- Generated code. How does it work? How do we make it clean? Do we check it + in? +- Do we need/want an example client that uses the cluster registry for some + basic operations? A demo, as it were? +- Labels vs. fields. Versioning? Do we graduate? Do we define a policy for + label names and have the server validate it? +- Where should the user documentation for the cluster registry live? + kubernetes.io doesn't quite seem appropriate, but perhaps that's the right + place? +- Is there a reasonable way to support a CRD-based implementation? Should this + project support it directly, or work to not prevent it from working, or + ignore it? + +## History + +| Date | Details | +|--|--| +| 10/9/17 | Initial draft | +| 10/16/17 | Minor edits based on comments | +| 10/19/17 | Added section specifically about version skew; change P0/P1/P2 to alpha/beta/stable; added some milestone requirements | +| 11/2/17 | Resolved most comments and added minor tweaks to text in order to do so | diff --git a/contributors/design-proposals/control-plane-resilience.md b/contributors/design-proposals/multicluster/control-plane-resilience.md index 95a1d93c..174e7de0 100644 --- a/contributors/design-proposals/control-plane-resilience.md +++ b/contributors/design-proposals/multicluster/control-plane-resilience.md @@ -112,9 +112,9 @@ well-bounded time period. 1. cluster persistent state (i.e. etcd disks) is either: 1. truly persistent (i.e. remote persistent disks), or 1. reconstructible (e.g. using etcd [dynamic member - addition](https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md#add-a-new-member) + addition](https://github.com/coreos/etcd/blob/master/Documentation/v2/runtime-configuration.md#add-a-new-member) or [backup and - recovery](https://github.com/coreos/etcd/blob/master/Documentation/admin_guide.md#disaster-recovery)). + recovery](https://github.com/coreos/etcd/blob/master/Documentation/v2/admin_guide.md#disaster-recovery)). 1. and boot disks are either: 1. truly persistent (i.e. remote persistent disks), or 1. reconstructible (e.g. using boot-from-snapshot, @@ -194,14 +194,13 @@ to do three things: <li>allocate a new node (not necessary if running etcd as a pod, in which case specific measures are required to prevent user pods from interfering with system pods, for example using node selectors as -described in <A HREF="), +described in <A HREF="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector">nodeSelector</A>), <li>start an etcd replica on that new node, and <li>have the new replica recover the etcd state. </ol> In the case of local disk (which fails in concert with the machine), the etcd state must be recovered from the other replicas. This is called -<A HREF="https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md#add-a-new-member"> -dynamic member addition</A>. +<A HREF="https://github.com/coreos/etcd/blob/master/Documentation/op-guide/runtime-configuration.md#add-a-new-member">dynamic member addition</A>. In the case of remote persistent disk, the etcd state can be recovered by attaching the remote persistent disk to the replacement node, thus the state is @@ -210,8 +209,7 @@ recoverable even if all other replicas are down. There are also significant performance differences between local disks and remote persistent disks. For example, the <A HREF="https://cloud.google.com/compute/docs/disks/#comparison_of_disk_types"> -sustained throughput local disks in GCE is approximately 20x that of remote -disks</A>. +sustained throughput local disks in GCE is approximately 20x that of remote disks</A>. Hence we suggest that self-healing be provided by remotely mounted persistent disks in non-performance critical, single-zone cloud deployments. For @@ -235,7 +233,3 @@ be automated and continuously tested. </tr> </table> - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/multicluster/federated-api-servers.md b/contributors/design-proposals/multicluster/federated-api-servers.md new file mode 100644 index 00000000..00b1c23b --- /dev/null +++ b/contributors/design-proposals/multicluster/federated-api-servers.md @@ -0,0 +1,4 @@ +# Federated API Servers + +Moved to [aggregated-api-servers.md](../api-machinery/aggregated-api-servers.md) since cluster +federation stole the word "federation" from this effort and it was very confusing. diff --git a/contributors/design-proposals/federated-ingress.md b/contributors/design-proposals/multicluster/federated-ingress.md index 07e75b0c..c2cb3793 100644 --- a/contributors/design-proposals/federated-ingress.md +++ b/contributors/design-proposals/multicluster/federated-ingress.md @@ -188,7 +188,3 @@ Assuming that GCP-only (see above) is complete: In theory the same approach as "Cross-cloud via GCP" above could be used, except that AWS infrastructure would be used to get traffic first to an AWS cluster, and then proxied onwards to non-AWS and/or on-prem clusters. Detail docs TBD. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/federated-placement-policy.md b/contributors/design-proposals/multicluster/federated-placement-policy.md index 8b3d4b49..c30374ea 100644 --- a/contributors/design-proposals/federated-placement-policy.md +++ b/contributors/design-proposals/multicluster/federated-placement-policy.md @@ -28,7 +28,7 @@ A simple example of a placement policy is > compliance. The [Kubernetes Cluster -Federation](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/federation.md#policy-engine-and-migrationreplication-controllers) +Federation](/contributors/design-proposals/multicluster/federation.md#policy-engine-and-migrationreplication-controllers) design proposal includes a pluggable policy engine component that decides how applications/resources are placed across federated clusters. @@ -283,7 +283,7 @@ When the remediator component (in the sidecar) receives the notification it sends a PATCH request to the federation-apiserver to update the affected resource. This way, the actual rebalancing of ReplicaSets is still handled by the [Rescheduling -Algorithm](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/federated-replicasets.md) +Algorithm](/contributors/design-proposals/multicluster/federated-replicasets.md) in the Federated ReplicaSet controller. The remediator component must be deployed with a kubeconfig for the @@ -368,4 +368,4 @@ engine could implement. ## Future Work - This proposal uses ConfigMaps to store and manage policies. In the future, we - want to introduce a first-class **Policy** API resource.
\ No newline at end of file + want to introduce a first-class **Policy** API resource. diff --git a/contributors/design-proposals/federated-replicasets.md b/contributors/design-proposals/multicluster/federated-replicasets.md index 8b48731c..9ed57bc7 100644 --- a/contributors/design-proposals/federated-replicasets.md +++ b/contributors/design-proposals/multicluster/federated-replicasets.md @@ -506,8 +506,3 @@ target cluster, to watch changes in every target cluster etcd regarding the posted LRS's and if any violation from the scheduled number of replicase is detected the scheduling code is re-called for re-scheduling purposes. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/federated-services.md b/contributors/design-proposals/multicluster/federated-services.md index 8ec9ca29..a43726d4 100644 --- a/contributors/design-proposals/federated-services.md +++ b/contributors/design-proposals/multicluster/federated-services.md @@ -513,7 +513,3 @@ state, and apply changes to the underlying kubernetes clusters accordingly. They also have the anti-entropy mechanism for reconciling Cluster Federation "desired desired" state against kubernetes "actual desired" state. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/federation-clusterselector.md b/contributors/design-proposals/multicluster/federation-clusterselector.md index 0a68424a..c12e4233 100644 --- a/contributors/design-proposals/federation-clusterselector.md +++ b/contributors/design-proposals/multicluster/federation-clusterselector.md @@ -22,7 +22,7 @@ A few examples... 1. Deploy the baz service to all prod clusters globally Currently, it's possible to control placement decision of Federated ReplicaSets -using the `federation.kubernetes.io/replica-set-preferences` annotation. This provides functionaility to change the number of ReplicaSets created on each Federated Cluster, by setting the quantity for each Cluster by Cluster Name. Since cluster names are required, in situations where clusters are add/removed from Federation it would require the object definitions to change in order to maintain the same configuration. From the example above, if a new cluster is created in Europe and added to federation, then the replica-set-preferences would need to be updated to include the new cluster name. +using the `federation.kubernetes.io/replica-set-preferences` annotation. This provides functionality to change the number of ReplicaSets created on each Federated Cluster, by setting the quantity for each Cluster by Cluster Name. Since cluster names are required, in situations where clusters are add/removed from Federation it would require the object definitions to change in order to maintain the same configuration. From the example above, if a new cluster is created in Europe and added to federation, then the replica-set-preferences would need to be updated to include the new cluster name. This proposal is to provide placement decision support for all object types using Labels on the Federated Clusters as opposed to cluster names. The matching language currently used for nodeAffinity placement decisions onto nodes can be leveraged. @@ -34,7 +34,7 @@ Carrying forward the examples from above... ## Design -The proposed design uses a ClusterSelector annotation that has a value that is parsed into a struct definition that follows the same design as the [NodeSelector type used w/ nodeAffinity](https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/types.go#L1798) and will also use the [Matches function](https://github.com/kubernetes/apimachinery/blob/master/pkg/labels/selector.go#L172) of the apimachinery project to determine if an object should be sent on to federated clusters or not. +The proposed design uses a ClusterSelector annotation that has a value that is parsed into a struct definition that follows the same design as the [NodeSelector type used w/ nodeAffinity](https://git.k8s.io/kubernetes/pkg/api/types.go#L1972) and will also use the [Matches function](https://git.k8s.io/apimachinery/pkg/labels/selector.go#L172) of the apimachinery project to determine if an object should be sent on to federated clusters or not. In situations where objects are not to be forwarded to federated clusters, instead a delete api call will be made using the object definition. If the object does not exist it will be ignored. @@ -78,4 +78,4 @@ The logic to determine if an object is sent to a Federated Cluster will have two ## Open Questions 1. Should there be any special considerations for when dependant resources would not be forwarded together to a Federated Cluster. -1. How to improve usability of this feature long term. It will certainly help to give first class API support but easier ways to map labels or requirements to objects may be required.
\ No newline at end of file +1. How to improve usability of this feature long term. It will certainly help to give first class API support but easier ways to map labels or requirements to objects may be required. diff --git a/contributors/design-proposals/federation-high-level-arch.png b/contributors/design-proposals/multicluster/federation-high-level-arch.png Binary files differindex 8a416cc1..8a416cc1 100644 --- a/contributors/design-proposals/federation-high-level-arch.png +++ b/contributors/design-proposals/multicluster/federation-high-level-arch.png diff --git a/contributors/design-proposals/federation-lite.md b/contributors/design-proposals/multicluster/federation-lite.md index 549f98df..3dd1f4d8 100644 --- a/contributors/design-proposals/federation-lite.md +++ b/contributors/design-proposals/multicluster/federation-lite.md @@ -86,7 +86,7 @@ service of type LoadBalancer. The native cloud load-balancers on both AWS & GCE are region-level, and support load-balancing across instances in multiple zones (in the same region). For both clouds, the behaviour of the native cloud load-balancer is reasonable in the face of failures (indeed, this is why clouds -provide load-balancing as a primitve). +provide load-balancing as a primitive). For multi-AZ clusters we will therefore simply rely on the native cloud provider load balancer behaviour, and we do not anticipate substantial code changes. @@ -170,8 +170,8 @@ GCE. If you had two volumes both named `myvolume` in two different GCE zones, this would not be ambiguous when Kubernetes is operating only in a single zone. But, when operating a cluster across multiple zones, `myvolume` is no longer sufficient to specify a volume uniquely. Worse, the fact that a volume happens -to be unambigious at a particular time is no guarantee that it will continue to -be unambigious in future, because a volume with the same name could +to be unambiguous at a particular time is no guarantee that it will continue to +be unambiguous in future, because a volume with the same name could subsequently be created in a second zone. While perhaps unlikely in practice, we cannot automatically enable multi-AZ clusters for GCE users if this then causes volume mounts to stop working. @@ -195,7 +195,3 @@ Initially therefore, the GCE changes will be to: 1. change the kubernetes cloud provider to iterate through relevant zones when resolving items 1. tag GCE PD volumes with the appropriate zone information - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/federation-phase-1.md b/contributors/design-proposals/multicluster/federation-phase-1.md index 3aa16dd8..25d27ee6 100644 --- a/contributors/design-proposals/federation-phase-1.md +++ b/contributors/design-proposals/multicluster/federation-phase-1.md @@ -7,7 +7,7 @@ In this document we propose a design for the “Control Plane” of Kubernetes (K8S) federation (a.k.a. “Ubernetes”). For background of this work please refer to -[this proposal](../../docs/proposals/federation.md). +[this proposal](federation.md). The document is arranged as following. First we briefly list scenarios and use cases that motivate K8S federation work. These use cases drive the design and they also verify the design. We summarize the @@ -400,8 +400,3 @@ This part has been included in the section “Federated Service” of document “[Federated Cross-cluster Load Balancing and Service Discovery Requirements and System Design](federated-services.md))”. Please refer to that document for details. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/federation.md b/contributors/design-proposals/multicluster/federation.md index fc595123..21c159d7 100644 --- a/contributors/design-proposals/federation.md +++ b/contributors/design-proposals/multicluster/federation.md @@ -641,8 +641,3 @@ does each replica find the other replicas and how do clients find their primary zookeeper replica? And now how do I do a shared, highly available redis database? Use a few common specific use cases like this to flesh out the detailed API and semantics of Cluster Federation. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/ubernetes-cluster-state.png b/contributors/design-proposals/multicluster/ubernetes-cluster-state.png Binary files differindex 56ec2df8..56ec2df8 100644 --- a/contributors/design-proposals/ubernetes-cluster-state.png +++ b/contributors/design-proposals/multicluster/ubernetes-cluster-state.png diff --git a/contributors/design-proposals/ubernetes-design.png b/contributors/design-proposals/multicluster/ubernetes-design.png Binary files differindex 44924846..44924846 100644 --- a/contributors/design-proposals/ubernetes-design.png +++ b/contributors/design-proposals/multicluster/ubernetes-design.png diff --git a/contributors/design-proposals/ubernetes-scheduling.png b/contributors/design-proposals/multicluster/ubernetes-scheduling.png Binary files differindex 01774882..01774882 100644 --- a/contributors/design-proposals/ubernetes-scheduling.png +++ b/contributors/design-proposals/multicluster/ubernetes-scheduling.png diff --git a/contributors/design-proposals/network/OWNERS b/contributors/design-proposals/network/OWNERS new file mode 100644 index 00000000..1939ca5c --- /dev/null +++ b/contributors/design-proposals/network/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-network-leads +approvers: + - sig-network-leads +labels: + - sig/network diff --git a/contributors/design-proposals/command_execution_port_forwarding.md b/contributors/design-proposals/network/command_execution_port_forwarding.md index a7175403..b4545662 100644 --- a/contributors/design-proposals/command_execution_port_forwarding.md +++ b/contributors/design-proposals/network/command_execution_port_forwarding.md @@ -152,7 +152,3 @@ forwarding connections from different clients are not able to see each other's data. This can most likely be achieved via SELinux labeling and unique process contexts. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/network/coredns.md b/contributors/design-proposals/network/coredns.md new file mode 100644 index 00000000..50217366 --- /dev/null +++ b/contributors/design-proposals/network/coredns.md @@ -0,0 +1,223 @@ +# Add CoreDNS for DNS-based Service Discovery + +Status: Pending + +Version: Alpha + +Implementation Owner: @johnbelamaric + +## Motivation + +CoreDNS is another CNCF project and is the successor to SkyDNS, which kube-dns is based on. It is a flexible, extensible +authoritative DNS server and directly integrates with the Kubernetes API. It can serve as cluster DNS, +complying with the [dns spec](https://git.k8s.io/dns/docs/specification.md). + +CoreDNS has fewer moving parts than kube-dns, since it is a single executable and single process. It is written in Go so +it is memory-safe (kube-dns includes dnsmasq which is not). It supports a number of use cases that kube-dns does not +(see below). As a general-purpose authoritative DNS server it has a lot of functionality that kube-dns could not reasonably +be expected to add. See, for example, the [intro](https://docs.google.com/presentation/d/1v6Coq1JRlqZ8rQ6bv0Tg0usSictmnN9U80g8WKxiOjQ/edit#slide=id.g249092e088_0_181) or [coredns.io](https://coredns.io) or the [CNCF webinar](https://youtu.be/dz9S7R8r5gw). + +## Proposal + +The proposed solution is to enable the selection of CoreDNS as an alternate to Kube-DNS during cluster deployment, with the +intent to make it the default in the future. + +## User Experience + +### Use Cases + + * Standard DNS-based service discovery + * Federation records + * Stub domain support + * Adding custom DNS entries + * Making an alias for an external name [#39792](https://github.com/kubernetes/kubernetes/issues/39792) + * Dynamically adding services to another domain, without running another server [#55](https://github.com/kubernetes/dns/issues/55) + * Adding an arbitrary entry inside the cluster domain (for example TXT entries [#38](https://github.com/kubernetes/dns/issues/38)) + * Verified pod DNS entries (ensure pod exists in specified namespace) + * Experimental server-side search path to address latency issues [#33554](https://github.com/kubernetes/kubernetes/issues/33554) + * Limit PTR replies to the cluster CIDR [#125](https://github.com/kubernetes/dns/issues/125) + * Serve DNS for selected namespaces [#132](https://github.com/kubernetes/dns/issues/132) + * Serve DNS based on a label selector + * Support for wildcard queries (e.g., `*.namespace.svc.cluster.local` returns all services in `namespace`) + +By default, the user experience would be unchanged. For more advanced uses, existing users would need to modify the +ConfigMap that contains the CoreDNS configuration file. + +### Configuring CoreDNS + +The CoreDNS configuration file is called a `Corefile` and syntactically is the same as a +[Caddyfile](https://caddyserver.com/docs/caddyfile). The file consists of multiple stanzas called _server blocks_. +Each of these represents a set of zones for which that server block should respond, along with the list +of plugins to apply to a given request. More details on this can be found in the +[Corefile Explained](https://coredns.io/2017/07/23/corefile-explained/) and +[How Queries Are Processed](https://coredns.io/2017/06/08/how-queries-are-processed-in-coredns/) blog +entries. + +### Configuration for Standard Kubernetes DNS + +The intent is to make configuration as simple as possible. The following Corefile will behave according +to the spec, except that it will not respond to Pod queries. It assumes the cluster domain is `cluster.local` +and the cluster CIDRs are all within 10.0.0.0/8. + +``` +. { + errors + log + cache 30 + health + prometheus + kubernetes 10.0.0.0/8 cluster.local + proxy . /etc/resolv.conf +} + +``` + +The `.` means that queries for the root zone (`.`) and below should be handled by this server block. Each +of the lines within `{ }` represent individual plugins: + + * `errors` enables [error logging](https://coredns.io/plugins/errors) + * `log` enables [query logging](https://coredns.io/plugins/log/) + * `cache 30` enables [caching](https://coredns.io/plugins/cache/) of positive and negative responses for 30 seconds + * `health` opens an HTTP port to allow [health checks](https://coredns.io/plugins/health) from Kubernetes + * `prometheus` enables Prometheus [metrics](https://coredns.io/plugins/metrics) + * `kubernetes 10.0.0.0/8 cluster.local` connects to the Kubernetes API and [serves records](https://coredns.io/plugins/kubernetes/) for the `cluster.local` domain and reverse DNS for 10.0.0.0/8 per the [spec](https://git.k8s.io/dns/docs/specification.md) + * `proxy . /etc/resolv.conf` [forwards](https://coredns.io/plugins/proxy) any queries not handled by other plugins (the `.` means the root domain) to the nameservers configured in `/etc/resolv.conf` + +### Configuring Stub Domains + +To configure stub domains, you add additional server blocks for those domains: + +``` +example.com { + proxy example.com 8.8.8.8:53 +} + +. { + errors + log + cache 30 + health + prometheus + kubernetes 10.0.0.0/8 cluster.local + proxy . /etc/resolv.conf +} +``` + +### Configuring Federation + +Federation is implemented as a separate plugin. You simply list the federation names and +their corresponding domains. + +``` +. { + errors + log + cache 30 + health + prometheus + kubernetes 10.0.0.0/8 cluster.local + federation cluster.local { + east east.example.com + west west.example.com + } + proxy . /etc/resolv.conf +} +``` + +### Reverse DNS + +Reverse DNS is supported for Services and Endpoints. It is not for Pods. + +You have to configure the reverse zone to make it work. That means knowing the service CIDR and configuring that +ahead of time (until [#25533](https://github.com/kubernetes/kubernetes/issues/25533) is implemented). + +Since reverse DNS zones are on classful boundaries, if you have a classless CIDR for your service CIDR +(say, a /12), then you have to widen that to the containing classful network. That leaves a subset of that network +open to the spoofing described in [#125](https://github.com/kubernetes/dns/issues/125); this is to be fixed +in [#1074](https://github.com/coredns/coredns/issues/1074). + +PTR spoofing by manual endpoints +([#124](https://github.com/kubernetes/dns/issues/124)) would +still be an issue even with [#1074](https://github.com/coredns/coredns/issues/1074) solved (as it is in kube-dns). This could be resolved in the case +where `pods verified` is enabled but that is not done at this time. + +### Deployment and Operations + +Typically when deployed for cluster DNS, CoreDNS is managed by a Deployment. The +CoreDNS pod only contains a single container, as opposed to kube-dns which requires three +containers. This simplifies troubleshooting. + +The Kubernetes integration is stateless and so multiple pods may be run. Each pod will have its +own connection to the API server. If you (like OpenShift) run a DNS pod for each node, you should not enable +`pods verified` as that could put a high load on the API server. Instead, if you wish to support +that functionality, you can run another central deployment and configure the per-node +instances to proxy `pod.cluster.local` to the central deployment. + +All logging is to standard out, and may be disabled if +desired. In very high queries-per-second environments, it is advisable to disable query logging to +avoid I/O for every query. + +CoreDNS can be configured to provide an HTTP health check endpoint, so that it can be monitored +by a standard Kubernetes HTTP health check. Readiness checks are not currently supported but +are in the works (see [#588](https://github.com/coredns/coredns/issues/588)). For Kubernetes, a +CoreDNS instance will be considered ready when it has finished syncing with the API. + +CoreDNS performance metrics can be published for Prometheus. + +When a change is made to the Corefile, you can send each CoreDNS instance a SIGUSR1, which will +trigger a graceful reload of the Corefile. + +### Performance and Resource Load + +The performance test was done in GCE with the following components: + + * CoreDNS system with machine type : n1-standard-1 ( 1 CPU, 2.3 GHz Intel Xeon E5 v3 (Haswell)) + * Client system with machine type: n1-standard-1 ( 1 CPU, 2.3 GHz Intel Xeon E5 v3 (Haswell)) + * Kubemark Cluster with 5000 nodes + +CoreDNS and client are running out-of-cluster (due to it being a Kubemark cluster). + +The following is the summary of the performance of CoreDNS. CoreDNS cache was disabled. + +Services (with 1% change per minute\*) | Max QPS\*\* | Latency (Median) | CoreDNS memory (at max QPS) | CoreDNS CPU (at max QPS) | +------------ | ------------- | -------------- | --------------------- | ----------------- | +1,000 | 18,000 | 0.1 ms | 38 MB | 95 % | +5,000 | 16,000 | 0.1 ms | 73 MB | 93 % | +10,000 | 10,000 | 0.1 ms | 115 MB | 78 % | + +\* We simulated service change load by creating and destroying 1% of services per minute. + +\** Max QPS with < 1 % packet loss + +## Implementation + +Each distribution project (kubeadm, minikube, kubespray, and others) will implement CoreDNS as an optional +add-on as appropriate for that project. + +### Client/Server Backwards/Forwards compatibility + +No changes to other components are needed. + +The method for configuring the DNS server will change. Thus, in cases where users have customized +the DNS configuration, they will need to modify their configuration if they move to CoreDNS. +For example, if users have configured stub domains, they would need to modify that configuration. + +When serving SRV requests for headless services, some responses are different from kube-dns, though still within +the specification (see [#975](https://github.com/coredns/coredns/issues/975)). In summary, these are: + + * kube-dns uses endpoint names that have an opaque identifier. CoreDNS instead uses the pod IP with dashes. + * kube-dns returns a bogus SRV record with port = 0 when no SRV prefix is present in the query. + coredns returns all SRV record for the service (see also [#140](https://github.com/kubernetes/dns/issues/140)) + +Additionally, federation may return records in a slightly different manner (see [#1034](https://github.com/coredns/coredns/issues/1034)), +though this may be changed prior to completing this proposal. + +In the plan for the Alpha, there will be no automated conversion of the kube-dns configuration. However, as +part of the Beta, code will be provided that will produce a proper Corefile based upon the existing kube-dns +configuration. + +## Alternatives considered + +Maintain existing kube-dns, add functionality to meet the currently unmet use cases above, and fix underlying issues. +Ensuring the use of memory-safe code would require replacing dnsmasq with another (memory-safe) caching DNS server, +or implementing caching within kube-dns. diff --git a/contributors/design-proposals/external-lb-source-ip-preservation.md b/contributors/design-proposals/network/external-lb-source-ip-preservation.md index 4735c135..50140a0e 100644 --- a/contributors/design-proposals/external-lb-source-ip-preservation.md +++ b/contributors/design-proposals/network/external-lb-source-ip-preservation.md @@ -1,5 +1,3 @@ -<!-- BEGIN MUNGE: GENERATED_TOC --> - - [Overview](#overview) - [Motivation](#motivation) - [Alpha Design](#alpha-design) @@ -27,7 +25,6 @@ - [Future work](#future-work) - [Appendix](#appendix) -<!-- END MUNGE: GENERATED_TOC --> # Overview @@ -165,7 +162,7 @@ For the 1.4 release, this feature will be implemented for the GCE cloud provider - Node: On the node, we expect to see the real source IP of the client. Destination IP will be the Service Virtual External IP. -- Pod: For processes running inside the Pod network namepsace, the source IP will be the real client source IP. The destination address will the be Pod IP. +- Pod: For processes running inside the Pod network namespace, the source IP will be the real client source IP. The destination address will the be Pod IP. #### GCE Expected Packet Destination IP (HealthCheck path) diff --git a/contributors/design-proposals/flannel-integration.md b/contributors/design-proposals/network/flannel-integration.md index 465ee5e6..3448ab28 100644 --- a/contributors/design-proposals/flannel-integration.md +++ b/contributors/design-proposals/network/flannel-integration.md @@ -126,7 +126,3 @@ This proposal is really just a call for community help in writing a Kubernetes x * Investigate flannel server running on every node (as done in the reference implementation mentioned above) * Use flannel reservation mode to support node controller podcidr allocation - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/network-policy.md b/contributors/design-proposals/network/network-policy.md index 32f7c016..6a4b01a8 100644 --- a/contributors/design-proposals/network-policy.md +++ b/contributors/design-proposals/network/network-policy.md @@ -297,8 +297,3 @@ spec: ## References - https://github.com/kubernetes/kubernetes/issues/22469 tracks network policy in kubernetes. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/networking.md b/contributors/design-proposals/network/networking.md index df8c8f82..ff97aa83 100644 --- a/contributors/design-proposals/networking.md +++ b/contributors/design-proposals/network/networking.md @@ -123,10 +123,11 @@ to serve the purpose outside of GCE. overlay network, primarily aiming at Docker integration. - [Calico](https://github.com/Metaswitch/calico) uses BGP to enable real container IPs. + - [Cilium](https://github.com/cilium/cilium) supports Overlay Network mode (IPv4/IPv6) and Direct Routing model (IPv6) ## Pod to service -The [service](../user-guide/services.md) abstraction provides a way to group pods under a +The [service](https://kubernetes.io/docs/concepts/services-networking/service/) abstraction provides a way to group pods under a common access policy (e.g. load-balanced). The implementation of this creates a virtual IP which clients can access and which is transparently proxied to the pods in a Service. Each node runs a kube-proxy process which programs @@ -185,7 +186,3 @@ AWS started rolling out basic but direct ipv6 assignment to instances doesn't appear to be supported by other major cloud providers (e.g. GCE) yet. We'd happily take pull requests from people running Kubernetes on bare metal, though. :-) - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/network/pod-resolv-conf.md b/contributors/design-proposals/network/pod-resolv-conf.md new file mode 100644 index 00000000..ed6e090f --- /dev/null +++ b/contributors/design-proposals/network/pod-resolv-conf.md @@ -0,0 +1,210 @@ +# Custom /etc/resolv.conf + +* Status: pending +* Version: alpha +* Implementation owner: Bowei Du <[bowei@google.com](mailto:bowei@google.com)>, + Zihong Zheng <[zihongz@google.com](mailto:zihongz@google.com)> + +# Overview + +The `/etc/resolv.conf` in a pod is managed by Kubelet and its contents are +generated based on `pod.dnsPolicy`. For `dnsPolicy: Default`, the `search` and +`nameserver` fields are taken from the `resolve.conf` on the node where the pod +is running. If the `dnsPolicy` is `ClusterFirst`, the search contents of the +resolv.conf is the hosts `resolv.conf` augmented with the following options: + +* Search paths to add aliases for domain names in the same namespace and + cluster suffix. +* `options ndots` to 5 to ensure the search paths are searched for all + potential matches. + +The configuration of both search paths and `ndots` results in query +amplification of five to ten times for non-cluster internal names. This is due +to the fact that each of the search path expansions must be tried before the +actual result is found. This order of magnitude increase of query rate imposes a +large load on the kube-dns service. At the same time, there are user +applications do not need the convenience of the name aliases and do not wish to +pay this performance cost. + + +## Existing workarounds + +The current work around for this problem is to specify an FQDN for name +resolution. Any domain name that ends with a period (e.g. `foo.bar.com.`) will +not be search path expanded. However, use of FQDNs is not well-known practice +and imposes application-level changes. Cluster operators may not have the luxury +of enforcing such a change to applications that run on their infrastructure. + +It is also possible for the user to insert a short shell script snippet that +rewrites `resolv.conf` on container start-up. This has the same problems as the +previous approach and is also awkward for the user. This also forces the +container to have additional executable code such as a shell or scripting engine +which increases the applications security surface area. + + +# Proposal sketch + +This proposal gives users a way to overlay tweaks into the existing +`DnsPolicy`. A new PodSpec field `dnsParams` will contains fields that are +merged with the settings currently selected with `DnsPolicy`. + +The fields of `DnsParams` are: + +* `nameservers` is a list of additional nameservers to use for resolution. On + `resolv.conf` platforms, these are entries to `nameserver`. +* `search` is a list of additional search path subdomains. On `resolv.conf` + platforms, these are entries to the `search` setting. These domains will be + appended to the existing search path. +* `options` that are an OS-dependent list of (name, value) options. These values + are NOT expected to be generally portable across platforms. For containers that + use `/etc/resolv.conf` style configuration, these correspond to the parameters + passed to the `option` lines. Options will override if their names coincide, + i.e, if the `DnsPolicy` sets `ndots:5` and `ndots:1` appears in the `Spec`, + then the final value will be `ndots:1`. + +For users that want to completely customize their resolution configuration, we +add a new `DnsPolicy: Custom` that does not define any settings. This is +essentially an empty `resolv.conf` with no fields defined. + +## Pod API examples + +### Host `/etc/resolv.conf` + +Assume in the examples below that the host has the following `/etc/resolv.conf`: + +```bash +nameserver 10.1.1.10 +search foo.com +options ndots:1 +``` + +### Override DNS server and search paths + +In the example below, the user wishes to use their own DNS resolver and add the +pod namespace and a custom expansion to the search path, as they do not use the +other name aliases: + +```yaml +# Pod spec +apiVersion: v1 +kind: Pod +metadata: {"namespace": "ns1", "name": "example"} +spec: + ... + dnsPolicy: Custom + dnsParams: + nameservers: ["1.2.3.4"] + search: + - ns1.svc.cluster.local + - my.dns.search.suffix + options: + - name: ndots + value: 2 + - name: edns0 +``` + +The pod will get the following `/etc/resolv.conf`: + +```bash +nameserver 1.2.3.4 +search ns1.svc.cluster.local my.dns.search.suffix +options ndots:2 edns0 +``` + +## Overriding `ndots` + +Override `ndots:5` in `ClusterFirst` with `ndots:1`. This keeps all of the +settings intact: + +```yaml +dnsPolicy: ClusterFirst +dnsParams: +- options: + - name: ndots + - value: 1 +``` + +Resulting `resolv.conf`: + +```bash +nameserver 10.0.0.10 +search default.svc.cluster.local svc.cluster.local cluster.local foo.com +options ndots:1 +``` + +# API changes + +```go +type PodSpec struct { + ... + DNSPolicy string `json:"dnsPolicy,omitempty"` + DNSParams *PodDNSParams `json:"dnsParams,omitempty"` + ... +} + +type PodDNSParams struct { + Nameservers []string `json:"nameservers,omitempty"` + Search []string `json:"search,omitempty"` + Options []PodDNSParamsOption `json:"options,omitempty" patchStrategy:"merge" patchMergeKey:"name"` +} + +type PodDNSParamsOption struct { + Name string `json:"name"` + Value *string `json:"value,omitempty"` +} +``` + +## Semantics + +Let the following be the Go representation of the `resolv.conf`: + +```go +type ResolvConf struct { + Nameserver []string // "nameserver" entries + Search []string // "search" entries + Options []PodDNSParamsOption // "options" entries +} +``` + +Let `var HostResolvConf ResolvConf` be the host `resolv.conf`. + +Then the final Pod `resolv.conf` will be: + +```go +func podResolvConf() ResolvConf { + var podResolv ResolvConf + + switch (pod.DNSPolicy) { + case "Default": + podResolv = HostResolvConf + case "ClusterFirst: + podResolv.Nameservers = []string{ KubeDNSClusterIP } + podResolv.Search = ... // populate with ns.svc.suffix, svc.suffix, suffix, host entries... + podResolv.Options = []PodDNSParamsOption{{"ndots","5" }} + case "Custom": // start with empty `resolv.conf` + break + } + + // Append the additional nameservers. + podResolv.Nameservers = append(Nameservers, pod.DNSParams.Nameservers...) + // Append the additional search paths. + podResolv.Search = append(Search, pod.DNSParams.Search...) + // Merge the DnsParams.Options with the options derived from the given DNSPolicy. + podResolv.Options = mergeOptions(pod.Options, pod.DNSParams.Options) + + return podResolv +} +``` + +### Invalid configurations + +The follow configurations will result in an invalid Pod spec: + +* Nameservers or search paths exceed system limits. (Three nameservers, six + search paths, 256 characters for `glibc`). +* Invalid option appears for the given platform. + +# References + +* [Kubernetes DNS name specification](https://git.k8s.io/dns/docs/specification.md) +* [`/etc/resolv.conf manpage`](http://manpages.ubuntu.com/manpages/zesty/man5/resolv.conf.5.html) diff --git a/contributors/design-proposals/service-discovery.md b/contributors/design-proposals/network/service-discovery.md index 66e497e7..c3da6b2b 100644 --- a/contributors/design-proposals/service-discovery.md +++ b/contributors/design-proposals/network/service-discovery.md @@ -60,10 +60,3 @@ The fragment below is taken from the service section of the kubernetes.json were Five service annotations are proposed as a standard way to describe a service endpoint. These five annotation are promoted as a Kubernetes standard, so that services can be discovered and a service catalog can be build to facilitate service consumers. - - - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/service-external-name.md b/contributors/design-proposals/network/service-external-name.md index eaab4c51..69073f8b 100644 --- a/contributors/design-proposals/service-external-name.md +++ b/contributors/design-proposals/network/service-external-name.md @@ -154,8 +154,3 @@ result in a failure during server certificate validation. This is acknowledged and left for future consideration. For the time being, users and administrators might need to ensure that the server certificates also mentions the Kubernetes name as an alternate host name. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/node/OWNERS b/contributors/design-proposals/node/OWNERS new file mode 100644 index 00000000..ab6d8dd5 --- /dev/null +++ b/contributors/design-proposals/node/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-node-leads +approvers: + - sig-node-leads +labels: + - sig/node diff --git a/contributors/design-proposals/node/accelerator-monitoring.md b/contributors/design-proposals/node/accelerator-monitoring.md new file mode 100644 index 00000000..984ce656 --- /dev/null +++ b/contributors/design-proposals/node/accelerator-monitoring.md @@ -0,0 +1,101 @@ +# Monitoring support for hardware accelerators + +Version: Alpha + +Owner: @mindprince (agarwalrohit@google.com) + +## Motivation + +We have had alpha support for running containers with GPUs attached in Kubernetes for a while. To take this to beta and GA, we need to provide GPU monitoring, so that users can get insights into how their GPU jobs are performing. + +## Detailed Design + +The current metrics pipeline for Kubernetes is: +- Container level metrics are collected by [cAdvisor](https://github.com/google/cadvisor). +- Kubelet embeds cAdvisor as a library. It uses its knowledge of pod-to-container mappings and the metrics from cAdvisor to expose pod level metrics as the summary API. +- [Heapster](https://github.com/kubernetes/heapster) uses kubelet’s summary API and pushes metrics to the some sink. +There are plans to change this pipeline but the details for that are still not finalized. + +To expose GPU metrics to Kubernetes users, we would need to make changes to all these components. + +First up is cAdvisor: we need to make cAdvisor collect metrics for GPUs that are attached to a container. + +The source for getting metrics for NVIDIA GPUs is [NVIDIA Management Library (NVML)](https://developer.nvidia.com/nvidia-management-library-nvml). NVML is a closed source C library [with a documented API](http://docs.nvidia.com/deploy/nvml-api/index.html). Because we want to use NVML from cAdvisor (which is written in Go), we need to [write a cgo wrapper for NVML](https://github.com/mindprince/gonvml). + +The cAdvisor binary is statically linked currently. Because we can’t statically link the closed source NVML code, we would need to make cAdvisor a dynamically linked binary. We would use `dlopen` in the cgo wrapper to dynamically load NVML. Because kubelet embeds cAdvisor, kubelet will also need to be a dynamically linked binary. In my testing, kubelet running on GCE 1.7.x clusters was found to be a dynamically linked binary already but now being dynamically linked will become a requirement. + +When cAdvisor starts up, it would read the vendor files in `/sys/bus/pci/devices/*` to see if any NVIDIA devices (vendor ID: `0x10de`) are attached to the node. +- If no NVIDIA devices are found, this code path would become dormant for the rest of cAdvisor/kubelet lifetime. +- If NVIDIA devices are found, we would start a goroutine that would check for the presence of NVML by trying to dynamically load it at regular intervals (say every minute or every 5 minutes). We need to do this regular checking instead of doing it just once because it may happen that cAdvisor is started before the nvidia drivers and nvml are installed. Once the NVML dynamic loading succeeds, we would use NVML’s query methods to find out how many devices exist on the node and create a map from their minor numbers to their handles and cache that map. The goroutine would exit at this point. + +If we detected the presence of NVML in the previous step, whenever a new container is detected by cAdvisor, cAdvisor would read the `devices.list` file from the container [devices cgroup](https://www.kernel.org/doc/Documentation/cgroup-v1/devices.txt). The `devices.list` file lists the major:minor number of all the devices that the container is allowed to access. If we find any device with major number `195` ([which is the major number assigned to NVIDIA devices](https://github.com/torvalds/linux/blob/v4.13/Documentation/admin-guide/devices.txt#L2583)), we would cache the list of corresponding minor numbers for that container. + +During every housekeeping operation, in addition to collecting all the existing metrics, we will use the cached nvidia device minor numbers and the map from minor numbers to device handles to get metrics for GPU devices attached to the container. + +The following new metrics would be exposed per container from cAdvisor: + +``` +type ContainerStats struct { +... + // Metrics for Accelerators. + // Each Accelerator corresponds to one element in the array. + Accelerators []AcceleratorStats `json:"accelerators,omitempty"` +... +} + +type AcceleratorStats struct { + // Make of the accelerator (nvidia, amd, google etc.) + Make string `json:"make"` + + // Model of the accelerator (tesla-p100, tesla-k80) + Model string `json:"model"` + + // ID of the accelerator. device minor number? Or UUID? + ID string `json:"id"` + + // Total acclerator memory. + // unit: bytes + MemoryTotal uint64 `json:"memory_total"` + + // Total accelerator memory allocated. + // unit: bytes + MemoryUsed uint64 `json:"memory_used"` + + // Percent of time over the past sample period during which + // the accelerator was actively processing. + DutyCycle uint64 `json:"duty_cycle"` +} +``` + +The API is generic to add support for different types of accelerators in the future even though we will only add support for NVIDIA GPUs initially. The API is inspired by what Google has in borg. + +We will update kubelet’s summary API to also add these metrics. + +From the summary API, they will flow to heapster and stackdriver. + +## Caveats +- As mentioned before, this would add a requirement that cAdvisor and kubelet are dynamically linked. +- We would need to make sure that kubelet is able to access the nvml libraries. Some existing container based nvidia driver installers install drivers in a special directory. We would need to make sure that that directory is in kubelet’s `LD_LIBRARY_PATH`. + +## Testing Plan +- Adding unit tests and e2e tests to cAdvisor for this code. +- Manually testing various scenarios with nvml installed and not installed; containers running with nvidia devices attached and not attached. +- Performance/Utilization testing: impact on cAdvisor/kubelet resource usage. Impact on GPU performance when we collect metrics. + +## Alternatives Rejected +Why collect GPU metrics in cAdvisor? Why not collect them in [device plugins](/contributors/design-proposals/resource-management/device-plugin.md)? The path forward if we collected GPU metrics in device plugin is not clear and may take a lot of time to get finalized. + +Here’s a rough sketch of how things could work: + +(1) device plugin -> kubelet summary API -> heapster -> ... +- device plugin collects GPU metrics using the cgo wrapper. This is straightforward, in fact, this may even be easier because we don’t have to worry about making kubelet dynamically linked. +- device plugin exposes a new container-level metrics API. This is complicated. There's no good way to have a device plugin metrics API. All we can have is a device plugin metrics endpoint. We can't really define how the metrics inside that will look like because different device types can have wildly different metrics. We can't have a metrics structure that will work well both for GPUs and NICs for example. +- We would have to make the kubelet understand whatever metrics are exposed in the device plugin metrics endpoint and expose it though the summary API. This is not ideal because device plugins are out-of-tree and controlled by vendors, so there can’t a mapping between the metrics exposed by the device plugins and what’s exposed in the kubelet’s summary API. If we try to define such a mapping, it becomes an implicit API that new device plugins have to follow to get their metrics exposed by the kubelet or they would have to update the mapping. + +(2) device plugin -> heapster -> ... +- If we don’t go through the kubelet, we can make heapster directly talk to the metrics endpoint exposed by the device plugin. This has the same problem as the last bullet point: how would heapster understand the metrics exposed by the device plugins so that it [can expose them to its backends](https://github.com/kubernetes/heapster/blob/v1.4.3/docs/storage-schema.md). In addition, we would have to solve the issue of how to map containers to their pods. + +(3) device plugin -> … +- If we don’t go through kubelet or heapster. We can have the device plugins directly expose metrics to the monitoring agent. For example, device plugins can expose a /metrics endpoint in prometheus format and prometheus can scrape it directly or a prom-to-sd container can send metrics from that endpoint directly to stackdriver. This becomes a more DIY solution, where there’s no real monitoring support provided by kubernetes and device plugin vendors are expected to add metrics to their plugins and users/operators are expected to plumb those metrics to their metrics storage backends. This approach also requires a way to map containers to their pods. + +Once the new monitoring architecture plans are implemented, we can revisit this and maybe collect GPU metrics in device plugins instead of cAdvisor. diff --git a/contributors/design-proposals/all-in-one-volume.md b/contributors/design-proposals/node/all-in-one-volume.md index 9c91f102..e1796817 100644 --- a/contributors/design-proposals/all-in-one-volume.md +++ b/contributors/design-proposals/node/all-in-one-volume.md @@ -299,8 +299,3 @@ spec: path: my-group/my-password mode: 511 ``` - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/annotations-downward-api.md b/contributors/design-proposals/node/annotations-downward-api.md index 8b6933d5..dcad5ab1 100644 --- a/contributors/design-proposals/annotations-downward-api.md +++ b/contributors/design-proposals/node/annotations-downward-api.md @@ -62,6 +62,3 @@ env: In general, this environment downward API part will be implemented in the same place as the other metadata - as a label conversion function. -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/container-init.md b/contributors/design-proposals/node/container-init.md index d4e8e83b..e26f92b4 100644 --- a/contributors/design-proposals/container-init.md +++ b/contributors/design-proposals/node/container-init.md @@ -426,7 +426,7 @@ container status `reason` and `message` fields while the pod is in the - mountPath: /var/run/docker.sock volumeName: dockersocket -## Backwards compatibilty implications +## Backwards compatibility implications Since this is a net new feature in the API and Kubelet, new API servers during upgrade may not be able to rely on Kubelets implementing init containers. The management of feature skew between @@ -438,7 +438,3 @@ master and Kubelet is tracked in issue [#4855](https://github.com/kubernetes/kub * Unify pod QoS class with init containers * Implement container / image volumes to make composition of runtime from images efficient - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/container-runtime-interface-v1.md b/contributors/design-proposals/node/container-runtime-interface-v1.md index e8e55c00..9e89abf5 100644 --- a/contributors/design-proposals/container-runtime-interface-v1.md +++ b/contributors/design-proposals/node/container-runtime-interface-v1.md @@ -260,8 +260,3 @@ can potentially become a very thin daemon. * Metrics: [#27097](https://issues.k8s.io/27097) * Log management: [#24677](https://issues.k8s.io/24677) - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/cpu-manager.md b/contributors/design-proposals/node/cpu-manager.md index f54c52c5..2dde3b6f 100644 --- a/contributors/design-proposals/cpu-manager.md +++ b/contributors/design-proposals/node/cpu-manager.md @@ -35,7 +35,7 @@ _Solution requirements:_ sure that the pod gets its cpu quota primarily from reserved core(s), resulting in fewer context switches and higher cache affinity". 1. Support the case where in a given pod, one container is latency-critical - and another is not (e.g. auxillary side-car containers responsible for + and another is not (e.g. auxiliary side-car containers responsible for log forwarding, metrics collection and the like.) 1. Do not cap CPU quota for guaranteed containers that are granted exclusive cores, since that would be antithetical to (1) above. @@ -45,6 +45,8 @@ _Solution requirements:_ * Feature: [Further differentiate performance characteristics associated with pod level QoS](https://github.com/kubernetes/features/issues/276) +* Feature: [Add CPU Manager for pod cpuset + assignment](https://github.com/kubernetes/features/issues/375) ## Proposed changes @@ -58,14 +60,14 @@ processor resource. The kuberuntime notifies the CPU manager when containers come and go. The first such notification occurs in between the container runtime interface calls to create and start the container. The second notification -occurs after the container is destroyed by the container runtime. The CPU +occurs after the container is stopped by the container runtime. The CPU Manager writes CPU settings for containers using a new CRI method named [`UpdateContainerResources`](https://github.com/kubernetes/kubernetes/pull/46105). This new method is invoked from two places in the CPU manager: during each -call to `RegisterContainer` and also periodically from a separate +call to `AddContainer` and also periodically from a separate reconciliation loop. - + _CPU Manager block diagram. `Policy`, `State`, and `Topology` types are factored out of the CPU Manager to promote reuse and to make it easier @@ -110,26 +112,23 @@ type State interface { GetCPUSet(containerID string) (cpuset.CPUSet, bool) GetDefaultCPUSet() cpuset.CPUSet GetCPUSetOrDefault(containerID string) cpuset.CPUSet - GetPresure() bool SetCPUSet(containerID string, cpuset CPUSet) SetDefaultCPUSet(cpuset CPUSet) Delete(containerID string) - SetPresure(value bool) } type Manager interface { - Start() - RegisterContainer(p *Pod, c *Container, containerID string) error - UnregisterContainer(containerID string) error - IsUnderCPUPresure() bool + Start(ActivePodsFunc, status.PodStatusProvider, runtimeService) + AddContainer(p *Pod, c *Container, containerID string) error + RemoveContainer(containerID string) error State() state.Reader } type Policy interface { Name() string Start(s state.State) - RegisterContainer(s State, pod *Pod, container *Container, containerID string) error - UnregisterContainer(s State, containerID string) error + AddContainer(s State, pod *Pod, container *Container, containerID string) error + RemoveContainer(s State, containerID string) error } type CPUSet map[int]struct{} // set operations and parsing/formatting helpers @@ -146,19 +145,12 @@ configuration. The three policies are **none**, **static** and **dynamic**. The active CPU manager policy is set through a new Kubelet configuration value `--cpu-manager-policy`. The default value is `none`. -The number of CPUs that pods may run on can be implicitly controlled using the -existing node-allocatable configuration settings. See the [node allocatable -proposal document][node-allocatable] for details. The CPU manager will claim -`ceiling(node.status.allocatable.cpu)` as the number of CPUs available to -assign to pods, starting from the highest-numbered physical core and -descending topologically. It is recommended to configure `kube-reserved` -and `system-reserved` such that their sum is an integer when the CPU manager -is enabled. This ensures that `node.status.allocatable.cpu` is also an -integer. - -Operator documentation will be updated to explain how to configure the -system to use the low-numbered physical cores for kube-reserved and -system-reserved cgroups. +The CPU manager periodically writes resource updates through the CRI in +order to reconcile in-memory cpuset assignments with cgroupfs. The +reconcile frequency is set through a new Kubelet configuration value +`--cpu-manager-reconcile-period`. If not specified, it defaults to the +same duration as `--node-status-update-frequency` (which itself defaults +to 10 seconds at time of writing.) Each policy is described below. @@ -182,6 +174,14 @@ node. Once allocated at pod admission time, an exclusive CPU remains assigned to a single container for the lifetime of the pod (until it becomes terminal.) +The Kubelet requires the total CPU reservation from `--kube-reserved` +and `--system-reserved` to be greater than zero when the static policy is +enabled. This is because zero CPU reservation would allow the shared pool to +become empty. The set of reserved CPUs is taken in order of ascending +physical core ID. Operator documentation will be updated to explain how to +configure the system to use the low-numbered physical cores for kube-reserved +and system-reserved cgroups. + Workloads that need to know their own CPU mask, e.g. for managing thread-level affinity, can read it from the virtual file `/proc/self/status`: @@ -198,6 +198,35 @@ allocated or deallocated.) ##### Implementation sketch +The static policy maintains the following sets of logical CPUs: + +- **SHARED:** Burstable, BestEffort, and non-integral Guaranteed containers + run here. Initially this contains all CPU IDs on the system. As + exclusive allocations are created and destroyed, this CPU set shrinks + and grows, accordingly. This is stored in the state as the default + CPU set. + +- **RESERVED:** A subset of the shared pool which is not exclusively + allocatable. The membership of this pool is static for the lifetime of + the Kubelet. The size of the reserved pool is the ceiling of the total + CPU reservation from `--kube-reserved` and `--system-reserved`. + Reserved CPUs are taken topologically starting with lowest-indexed + physical core, as reported by cAdvisor. + +- **ASSIGNABLE:** Equal to `SHARED - RESERVED`. Exclusive CPUs are allocated + from this pool. + +- **EXCLUSIVE ALLOCATIONS:** CPU sets assigned exclusively to one container. + These are stored as explicit assignments in the state. + +When an exclusive allocation is made, the static policy also updates the +default cpuset in the state abstraction. The CPU manager's periodic +reconcile loop takes care of updating the cpuset in cgroupfs for any +containers that may be running in the shared pool. For this reason, +applications running within exclusively-allocated containers must tolerate +potentially sharing their allocated CPUs for up to the CPU manager +reconcile period. + ```go func (p *staticPolicy) Start(s State) { fullCpuset := cpuset.NewCPUSet() @@ -209,7 +238,7 @@ func (p *staticPolicy) Start(s State) { s.SetDefaultCPUSet(fullCpuset.Difference(reserved)) } -func (p *staticPolicy) RegisterContainer(s State, pod *Pod, container *Container, containerID string) error { +func (p *staticPolicy) AddContainer(s State, pod *Pod, container *Container, containerID string) error { if numCPUs := numGuaranteedCPUs(pod, container); numCPUs != 0 { // container should get some exclusively allocated CPUs cpuset, err := p.allocateCPUs(s, numCPUs) @@ -222,10 +251,10 @@ func (p *staticPolicy) RegisterContainer(s State, pod *Pod, container *Container return nil } -func (p *staticPolicy) UnregisterContainer(s State, containerID string) error { +func (p *staticPolicy) RemoveContainer(s State, containerID string) error { if toRelease, ok := s.GetCPUSet(containerID); ok { s.Delete(containerID) - p.releaseCPUs(s, toRelease) + s.SetDefaultCPUSet(s.GetDefaultCPUSet().Union(toRelease)) } return nil } @@ -246,8 +275,8 @@ func (p *staticPolicy) UnregisterContainer(s State, containerID string) error { 1. _A container arrives that requires exclusive cores._ 1. Kuberuntime calls the CRI delegate to create the container. - 1. Kuberuntime registers the container with the CPU manager. - 1. CPU manager registers the container to the static policy. + 1. Kuberuntime adds the container with the CPU manager. + 1. CPU manager adds the container to the static policy. 1. Static policy acquires CPUs from the default pool, by topological-best-fit. 1. Static policy updates the state, adding an assignment for the new @@ -257,8 +286,8 @@ func (p *staticPolicy) UnregisterContainer(s State, containerID string) error { 1. Kuberuntime calls the CRI delegate to start the container. 1. _A container that was assigned exclusive cores terminates._ - 1. Kuberuntime unregisters the container with the CPU manager. - 1. CPU manager unregisters the container with the static policy. + 1. Kuberuntime removes the container with the CPU manager. + 1. CPU manager removes the container with the static policy. 1. Static policy adds the container's assigned CPUs back to the default pool. 1. Kuberuntime calls the CRI delegate to remove the container. @@ -266,14 +295,13 @@ func (p *staticPolicy) UnregisterContainer(s State, containerID string) error { cpuset for all containers running in the shared pool. 1. _The shared pool becomes empty._ - 1. The CPU manager adds a node condition with effect NoSchedule, - NoExecute that prevents BestEffort and Burstable QoS class pods from - running on the node. BestEffort and Burstable QoS class pods are - evicted from the node. - -1. _The shared pool becomes nonempty._ - 1. The CPU manager removes the node condition with effect NoSchedule, - NoExecute for BestEffort and Burstable QoS class pods. + 1. This cannot happen. The size of the shared pool is greater than + the number of exclusively allocatable CPUs. The Kubelet requires the + total CPU reservation from `--kube-reserved` and `--system-reserved` + to be greater than zero when the static policy is enabled. The number + of exclusively allocatable CPUs is + `floor(capacity.cpu - allocatable.cpu)` and the shared pool initially + contains all CPUs in the system. #### Policy 3: "dynamic" cpuset control @@ -295,11 +323,11 @@ func (p *dynamicPolicy) Start(s State) { // TODO } -func (p *dynamicPolicy) RegisterContainer(s State, pod *Pod, container *Container, containerID string) error { +func (p *dynamicPolicy) AddContainer(s State, pod *Pod, container *Container, containerID string) error { // TODO } -func (p *dynamicPolicy) UnregisterContainer(s State, containerID string) error { +func (p *dynamicPolicy) RemoveContainer(s State, containerID string) error { // TODO } ``` @@ -332,15 +360,6 @@ func (p *dynamicPolicy) UnregisterContainer(s State, containerID string) error { directly from the shared pool, is too simplistic. 1. Mitigation: defer supporting this until a new policy tailored for use with `isolcpus` can be added. -1. CPU exhaustion. Terminology: a no-CPU pod is defined here as having - at least one container with no or zero-valued CPU request. If all available - CPUs are allocated exclusively, additional steps must be taken to remove - any no-CPU pods from the node. In addition, the system must prevent further - no-CPU pods from being bound to the node. - 1. Mitigation: Introduce a new CPUPressure node condition. This - condition causes any no-CPU pods to fail scheduler predicates for this - node and also fail node-level admission checks. Also evict no-CPU pods - when CPUPressure occurs. ## Implementation roadmap @@ -362,21 +381,22 @@ func (p *dynamicPolicy) UnregisterContainer(s State, containerID string) error { * Performance metrics for one or more plausible synthetic workloads show benefit over none policy. -### Phase 3: Cache allocation +### Phase 3: Beta support [TARGET: Kubernetes v1.9] -* Static policy also manages [cache allocation][cat] on supported platforms. +* Container CPU assignments are durable across Kubelet restarts. +* Expanded user and operator docs and tutorials. -### Phase 4: Dynamic policy +### Later phases [TARGET: After Kubernetes v1.9] +* Static policy also manages [cache allocation][cat] on supported platforms. * Dynamic policy is implemented. * Unit tests for dynamic policy pass. * e2e tests for dynamic policy pass. * Performance metrics for one or more plausible synthetic workloads show benefit over none policy. - -### Phase 5: NUMA - -* Kubelet can discover "advanced" CPU topology (NUMA). +* Kubelet can discover "advanced" topology (NUMA). +* Node-level coordination for NUMA-dependent resource allocations, for example + devices, CPUs, memory-backed volumes including hugepages. ## Appendix A: cpuset pitfalls @@ -398,7 +418,7 @@ func (p *dynamicPolicy) UnregisterContainer(s State, containerID string) error { [cpuset-files]: http://man7.org/linux/man-pages/man7/cpuset.7.html#FILES [ht]: http://www.intel.com/content/www/us/en/architecture-and-technology/hyper-threading/hyper-threading-technology.html [hwloc]: https://www.open-mpi.org/projects/hwloc -[node-allocatable]: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node-allocatable.md#phase-2---enforce-allocatable-on-pods +[node-allocatable]: /contributors/design-proposals/node/node-allocatable.md#phase-2---enforce-allocatable-on-pods [procfs]: http://man7.org/linux/man-pages/man5/proc.5.html -[qos]: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-qos.md +[qos]: /contributors/design-proposals/node/resource-qos.md [topo]: http://github.com/intelsdi-x/swan/tree/master/pkg/isolation/topo diff --git a/contributors/design-proposals/cri-dockershim-checkpoint.md b/contributors/design-proposals/node/cri-dockershim-checkpoint.md index 85db4c89..85db4c89 100644 --- a/contributors/design-proposals/cri-dockershim-checkpoint.md +++ b/contributors/design-proposals/node/cri-dockershim-checkpoint.md diff --git a/contributors/design-proposals/disk-accounting.md b/contributors/design-proposals/node/disk-accounting.md index d11c7a6f..eaee0ead 100755..100644 --- a/contributors/design-proposals/disk-accounting.md +++ b/contributors/design-proposals/node/disk-accounting.md @@ -580,8 +580,3 @@ Capacity in MB = 1638400 * 512 * 128 bytes = 100 GB * If you use a non-default location for docker storage, change `/var/lib/docker` in the examples to your storage location. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/downward_api_resources_limits_requests.md b/contributors/design-proposals/node/downward_api_resources_limits_requests.md index bebf05ab..e8cf4438 100644 --- a/contributors/design-proposals/downward_api_resources_limits_requests.md +++ b/contributors/design-proposals/node/downward_api_resources_limits_requests.md @@ -162,7 +162,7 @@ metadata: spec: containers: - name: test-container - image: gcr.io/google_containers/busybox + image: k8s.gcr.io/busybox command: [ "/bin/sh","-c", "env" ] resources: requests: @@ -186,7 +186,7 @@ metadata: spec: containers: - name: client-container - image: gcr.io/google_containers/busybox + image: k8s.gcr.io/busybox command: ["sh", "-c", "while true; do if [[ -e /etc/labels ]]; then cat /etc/labels; fi; if [[ -e /etc/annotations ]]; then cat /etc/annotations; fi;sleep 5; done"] resources: requests: @@ -308,7 +308,7 @@ metadata: spec: containers: - name: test-container - image: gcr.io/google_containers/busybox + image: k8s.gcr.io/busybox command: [ "/bin/sh","-c", "env" ] resources: requests: @@ -332,7 +332,7 @@ metadata: spec: containers: - name: client-container - image: gcr.io/google_containers/busybox + image: k8s.gcr.io/busybox command: ["sh", "-c", "while true; do if [[ -e /etc/labels ]]; then cat /etc/labels; fi; if [[ -e /etc/annotations ]]; then cat /etc/annotations; fi; sleep 5; done"] resources: requests: @@ -470,7 +470,7 @@ metadata: spec: containers: - name: test-container - image: gcr.io/google_containers/busybox + image: k8s.gcr.io/busybox command: [ "/bin/sh","-c", "env" ] resources: requests: @@ -501,7 +501,7 @@ metadata: spec: containers: - name: client-container - image: gcr.io/google_containers/busybox + image: k8s.gcr.io/busybox command: ["sh", "-c","while true; do if [[ -e /etc/labels ]]; then cat /etc/labels; fi; if [[ -e /etc/annotations ]]; then cat /etc/annotations; fi; sleep 5; done"] resources: requests: @@ -586,7 +586,7 @@ metadata: spec: containers: - name: test-container - image: gcr.io/google_containers/busybox + image: k8s.gcr.io/busybox command: [ "/bin/sh","-c", "env" ] resources: requests: @@ -616,7 +616,3 @@ and export GOMAXPROCS=$(CPU_LIMIT)" ``` - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/dynamic-kubelet-configuration.md b/contributors/design-proposals/node/dynamic-kubelet-configuration.md index 8ca0e99c..3dc8870f 100644 --- a/contributors/design-proposals/dynamic-kubelet-configuration.md +++ b/contributors/design-proposals/node/dynamic-kubelet-configuration.md @@ -75,7 +75,7 @@ There are two types of configuration that are stored on disk: - cached configurations from a remote source, e.g. `ConfigMaps` from etcd. - the local "init" configuration, e.g. the set of config files the node is provisioned with. -The Kubelet should accept a `--dynamic-config-dir` flag, which specifies a directory for storing all of the information necessary for dynamic configuraiton from remote sources; e.g. the cached configurations, which configuration is currently in use, which configurations are known to be bad, etc. +The Kubelet should accept a `--dynamic-config-dir` flag, which specifies a directory for storing all of the information necessary for dynamic configuration from remote sources; e.g. the cached configurations, which configuration is currently in use, which configurations are known to be bad, etc. - When the Kubelet downloads a `ConfigMap`, it will checkpoint a serialization of the `ConfigMap` object to a file at `{dynamic-config-dir}/checkpoints/{UID}`. - We checkpoint the entire object, rather than unpacking the contents to disk, because the former is less complex and reduces chance for errors during the checkpoint process. @@ -308,6 +308,3 @@ There is discussion in [#10179](https://github.com/kubernetes/kubernetes/issues/ + Reconciling configuration with objects added after the completion of a rollout, e.g. new `Nodes`. + Pausing/resuming a rollout. -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/envvar-configmap.md b/contributors/design-proposals/node/envvar-configmap.md index 5a398873..9464a1af 100644 --- a/contributors/design-proposals/envvar-configmap.md +++ b/contributors/design-proposals/node/envvar-configmap.md @@ -182,7 +182,3 @@ cm2_key2="b" Add similar support for Secrets. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/expansion.md b/contributors/design-proposals/node/expansion.md index 4f2f6f40..2647e85c 100644 --- a/contributors/design-proposals/expansion.md +++ b/contributors/design-proposals/node/expansion.md @@ -381,7 +381,7 @@ metadata: spec: containers: - name: test-container - image: gcr.io/google_containers/busybox + image: k8s.gcr.io/busybox command: [ "/bin/sh", "-c", "env" ] env: - name: PUBLIC_URL @@ -399,7 +399,7 @@ metadata: spec: containers: - name: test-container - image: gcr.io/google_containers/busybox + image: k8s.gcr.io/busybox command: [ "/bin/sh", "-c", "env" ] env: - name: POD_NAMESPACE @@ -410,8 +410,3 @@ spec: value: "http://gitserver.$(POD_NAMESPACE):$(SERVICE_PORT)" restartPolicy: Never ``` - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/kubelet-auth.md b/contributors/design-proposals/node/kubelet-auth.md index c4d35dd9..cb34f65d 100644 --- a/contributors/design-proposals/kubelet-auth.md +++ b/contributors/design-proposals/node/kubelet-auth.md @@ -101,6 +101,3 @@ This mode allows any authenticated request. * Add support for CRL revocation for x509 client certificate authentication (http://issue.k8s.io/18982) -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/kubelet-authorizer.md b/contributors/design-proposals/node/kubelet-authorizer.md index f3c24417..0352ea94 100644 --- a/contributors/design-proposals/kubelet-authorizer.md +++ b/contributors/design-proposals/node/kubelet-authorizer.md @@ -180,5 +180,5 @@ Future work could further limit a kubelet's API access: Features that expand or modify the APIs or objects accessed by the kubelet will need to involve the node authorizer. Known features in the design or development stages that might modify kubelet API access are: * [Dynamic kubelet configuration](https://github.com/kubernetes/features/issues/281) -* [Local storage management](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/local-storage-overview.md) +* [Local storage management](/contributors/design-proposals/storage/local-storage-overview.md) * [Bulk watch of secrets/configmaps](https://github.com/kubernetes/community/pull/443) diff --git a/contributors/design-proposals/kubelet-cri-logging.md b/contributors/design-proposals/node/kubelet-cri-logging.md index 0cda9901..a19ff3f5 100644 --- a/contributors/design-proposals/kubelet-cri-logging.md +++ b/contributors/design-proposals/node/kubelet-cri-logging.md @@ -59,7 +59,7 @@ the format and the metadata (i.e., timestamps) of the logs. In the current implementation, kubelet calls `docker logs` with parameters to return the log content. As of now, docker only supports `log` operations for the “journal” and “json-file” drivers [2]. In other words, *the support of `kubectl logs` is not -universal in all kuernetes deployments*. +universal in all kubernetes deployments*. **Cluster logging support** @@ -137,7 +137,7 @@ Because kubelet determines where the logs are stores and can access them directly, this meets requirement (1). As for requirement (2), the log collector can easily extract basic pod metadata (e.g., pod UID, container name) from the paths, and watch the directly for any changes. In the future, we can -extend this by maintaining a metada file in the pod directory. +extend this by maintaining a metadata file in the pod directory. **Log format** @@ -199,7 +199,7 @@ clients attaching as well. There are ad-hoc solutions/discussions that addresses one or two of the requirements, but no comprehensive solution for CRI specifically has been -proposed so far (with the excpetion of @tmrtfs's proposal +proposed so far (with the exception of @tmrtfs's proposal [#33111](https://github.com/kubernetes/kubernetes/pull/33111), which has a much wider scope). It has come up in discussions that kubelet can delegate all the logging management to the runtime to allow maximum flexibility. However, it is @@ -233,8 +233,3 @@ evolves. [4] rkt support: https://github.com/systemd/systemd/pull/4179 - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/kubelet-eviction.md b/contributors/design-proposals/node/kubelet-eviction.md index 68b39ec1..a96702cc 100644 --- a/contributors/design-proposals/kubelet-eviction.md +++ b/contributors/design-proposals/node/kubelet-eviction.md @@ -241,6 +241,19 @@ the `kubelet` will select a subsequent pod. ## Eviction Strategy +The `kubelet` will implement an eviction strategy oriented around +[Priority](/contributors/design-proposals/scheduling/pod-priority-api.md) +and pod usage relative to requests. It will target pods that are the lowest +Priority, and are the largest consumers of the starved resource relative to +their scheduling request. + +It will target pods whose usage of the starved resource exceeds its requests. +Of those pods, it will rank by priority, then usage - requests. If system +daemons are exceeding their allocation (see [Strategy Caveat](strategy-caveat) below), +and all pods are using less than their requests, then it must evict a pod +whose usage is less than requests, based on priority, then usage - requests. + +Prior to v1.9: The `kubelet` will implement a default eviction strategy oriented around the pod quality of service class. @@ -258,14 +271,16 @@ starved resource. relative to their request are killed first. If no pod has exceeded its request, the strategy targets the largest consumer of the starved resource. -A guaranteed pod is guaranteed to never be evicted because of another pod's -resource consumption. That said, guarantees are only as good as the underlying -foundation they are built upon. If a system daemon +### Strategy Caveat + +A pod consuming less resources than its requests is guaranteed to never be +evicted because of another pod's resource consumption. That said, guarantees +are only as good as the underlying foundation they are built upon. If a system daemon (i.e. `kubelet`, `docker`, `journald`, etc.) is consuming more resources than -were reserved via `system-reserved` or `kube-reserved` allocations, and the node -only has guaranteed pod(s) remaining, then the node must choose to evict a -guaranteed pod in order to preserve node stability, and to limit the impact -of the unexpected consumption to other guaranteed pod(s). +were reserved via `system-reserved` or `kube-reserved` allocations, then the node +must choose to evict a pod, even if it is consuming less than its requests. +It must take action in order to preserve node stability, and to limit the impact +of the unexpected consumption to other well-behaved pod(s). ## Disk based evictions @@ -425,15 +440,6 @@ from placing new best effort pods on the node since they will be rejected by the On the other hand, the `DiskPressure` condition if true should dissuade the scheduler from placing **any** new pods on the node since they will be rejected by the `kubelet` in admission. -## Enforcing Node Allocatable - -To enforce [Node Allocatable](./node-allocatable.md), Kubelet primarily uses cgroups. -However `storage` cannot be enforced using cgroups. - -Once Kubelet supports `storage` as an `Allocatable` resource, Kubelet will perform evictions whenever the total storage usage by pods exceed node allocatable. - -If a pod cannot tolerate evictions, then ensure that requests is set and it will not exceed `requests`. - ## Best Practices ### DaemonSet @@ -457,15 +463,3 @@ for eviction. Instead `DaemonSet` should ideally include Guaranteed pods only. The pod eviction may evict more pods than needed due to stats collection timing gap. This can be mitigated by adding the ability to get root container stats on an on-demand basis (https://github.com/google/cadvisor/issues/1247) in the future. - -### How kubelet ranks pods for eviction in response to inode exhaustion - -At this time, it is not possible to know how many inodes were consumed by a particular container. If the `kubelet` observes -inode exhaustion, it will evict pods by ranking them by quality of service. The following issue has been opened in cadvisor -to track per container inode consumption (https://github.com/google/cadvisor/issues/1422) which would allow us to rank pods -by inode consumption. For example, this would let us identify a container that created large numbers of 0 byte files, and evict -that pod over others. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/kubelet-hypercontainer-runtime.md b/contributors/design-proposals/node/kubelet-hypercontainer-runtime.md index c3da7d9a..8aba0b1a 100644 --- a/contributors/design-proposals/kubelet-hypercontainer-runtime.md +++ b/contributors/design-proposals/node/kubelet-hypercontainer-runtime.md @@ -38,8 +38,3 @@ runtime. The HyperContainer runtime is maintained by <https://github.com/kubernetes/frakti>. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/kubelet-rkt-runtime.md b/contributors/design-proposals/node/kubelet-rkt-runtime.md index 84aac8cc..1bc6435b 100644 --- a/contributors/design-proposals/kubelet-rkt-runtime.md +++ b/contributors/design-proposals/node/kubelet-rkt-runtime.md @@ -74,7 +74,7 @@ In addition, the rkt cli has historically been the primary interface to the rkt The initial integration will execute the rkt binary directly for app creation/start/stop/removal, as well as image pulling/removal. -The creation of pod sanbox is also done via rkt command line, but it will run under `systemd-run` so it's monitored by the init process. +The creation of pod sandbox is also done via rkt command line, but it will run under `systemd-run` so it's monitored by the init process. In the future, some of these decisions are expected to be changed such that rkt is vendored as a library dependency for all operations, and other init systems will be supported as well. @@ -97,7 +97,3 @@ In the future, some of these decisions are expected to be changed such that rkt 5. Revendor rktlet into `pkg/kubelet/rktshim`, and start deprecating the `pkg/kubelet/rkt` package. 6. Eventually replace the current `pkg/kubelet/rkt` package. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/kubelet-rootfs-distribution.md b/contributors/design-proposals/node/kubelet-rootfs-distribution.md index db51eaae..16c29404 100644 --- a/contributors/design-proposals/kubelet-rootfs-distribution.md +++ b/contributors/design-proposals/node/kubelet-rootfs-distribution.md @@ -30,7 +30,7 @@ running the Kubelet intentionally is using a chroot and is neither a container n The kubelet chroot will essentially operate as follows: ``` -container-download-and-extract gcr.io/google_containers/hyperkube:v1.4.0 /path/to/chroot +container-download-and-extract k8s.gcr.io/hyperkube:v1.4.0 /path/to/chroot mount --make-shared /var/lib/kubelet mount --rbind /var/lib/kubelet /path/to/chroot/var/lib/kubelet # And many more mounts, omitted @@ -65,7 +65,7 @@ This work will also only initially target the GCE provider and `kube-up` method #### Hyperkube Image Packaging -The Hyperkube image is distributed as part of an official release to the `gcr.io/google_containers` registry, but is not included along with the `kube-up` artifacts used for deployment. +The Hyperkube image is distributed as part of an official release to the `k8s.gcr.io` registry, but is not included along with the `kube-up` artifacts used for deployment. This will need to be remediated in order to complete this proposal. diff --git a/contributors/design-proposals/kubelet-systemd.md b/contributors/design-proposals/node/kubelet-systemd.md index b4277cfa..cef68d2a 100644 --- a/contributors/design-proposals/kubelet-systemd.md +++ b/contributors/design-proposals/node/kubelet-systemd.md @@ -135,7 +135,7 @@ The `kubelet` should associate node bootstrapping semantics to the configured ### Node allocatable The proposal makes no changes to the definition as presented here: -https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/node-allocatable.md +https://git.k8s.io/kubernetes/docs/proposals/node-allocatable.md The node will report a set of allocatable compute resources defined as follows: @@ -401,7 +401,3 @@ Some OS distributions will fix this bug in versions of docker <= 1.0.9, so opera be aware of how their version of `docker` was packaged when using this feature. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/node-allocatable.md b/contributors/design-proposals/node/node-allocatable.md index 09efa7a9..4fe22c37 100644 --- a/contributors/design-proposals/node-allocatable.md +++ b/contributors/design-proposals/node/node-allocatable.md @@ -90,7 +90,7 @@ Together, evictions and node allocatable help improve node stability. As of v1.5, evictions are based on overall node usage relative to `Capacity`. Kubelet evicts pods based on QoS and user configured eviction thresholds. -More deails in [this doc](./kubelet-eviction.md#enforce-node-allocatable) +More details in [this doc](./kubelet-eviction.md#enforce-node-allocatable) From v1.6, if `Allocatable` is enforced by default across all pods on a node using cgroups, pods cannot exceed `Allocatable`. Memory and CPU limits are enforced using cgroups, but there exists no easy means to enforce storage limits though. @@ -295,7 +295,7 @@ The former disables Node Allocatable enforcement on all pods and the latter avoi This rollout in v1.6 might cause the following symptoms: 1. If `--kube-reserved` and/or `--system-reserved` flags are also specified, OOM kills of containers and/or evictions of pods. This can happen primarily to `Burstable` and `BestEffort` pods since they can no longer use up all the resource available on the node. -1. Total allocatable capadity in the cluster reduces resulting in pods staying `Pending` because Hard Eviction Thresholds are included in Node Allocatable. +1. Total allocatable capacity in the cluster reduces resulting in pods staying `Pending` because Hard Eviction Thresholds are included in Node Allocatable. ##### Proposed Timeline @@ -334,8 +334,3 @@ according to `KubeReserved`. The community should be notified that an update to schedulers is recommended, but if a scheduler is not updated it falls under the above case of "scheduler is not allocatable-resources aware". - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/optional-configmap.md b/contributors/design-proposals/node/optional-configmap.md index 715ac6a7..715ac6a7 100644 --- a/contributors/design-proposals/optional-configmap.md +++ b/contributors/design-proposals/node/optional-configmap.md diff --git a/contributors/design-proposals/pleg.png b/contributors/design-proposals/node/pleg.png Binary files differindex f15c5d83..f15c5d83 100644 --- a/contributors/design-proposals/pleg.png +++ b/contributors/design-proposals/node/pleg.png diff --git a/contributors/design-proposals/pod-cache.png b/contributors/design-proposals/node/pod-cache.png Binary files differindex dee86c40..dee86c40 100644 --- a/contributors/design-proposals/pod-cache.png +++ b/contributors/design-proposals/node/pod-cache.png diff --git a/contributors/design-proposals/pod-lifecycle-event-generator.md b/contributors/design-proposals/node/pod-lifecycle-event-generator.md index 207d6a17..00f4e10c 100644 --- a/contributors/design-proposals/pod-lifecycle-event-generator.md +++ b/contributors/design-proposals/node/pod-lifecycle-event-generator.md @@ -186,7 +186,7 @@ redundant syncs. to provide all necessary information to detect container state changes in `GetPods()` (#13571). -- Benchmark docker to adjust relising frequency. +- Benchmark docker to adjust relisting frequency. - Fix/adapt features that rely on frequent, periodic pod syncing. * Liveness/Readiness probing: Create a separate probing manager using @@ -194,8 +194,3 @@ redundant syncs. * Instruct pod workers to set up a wake-up call if syncing failed, so that it can retry. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/pod-pid-namespace.md b/contributors/design-proposals/node/pod-pid-namespace.md index 43c38f22..6ac16b3b 100644 --- a/contributors/design-proposals/pod-pid-namespace.md +++ b/contributors/design-proposals/node/pod-pid-namespace.md @@ -70,8 +70,3 @@ to disable the shared PID namespace in the subsequent release. [1]: https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/ - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/pod-resource-management.md b/contributors/design-proposals/node/pod-resource-management.md index ee1e2ed3..cc1c43a9 100644 --- a/contributors/design-proposals/pod-resource-management.md +++ b/contributors/design-proposals/node/pod-resource-management.md @@ -16,7 +16,7 @@ to Kubernetes. It outlines the implementation and associated rollout plan. ## Introduction Kubernetes supports container level isolation by allowing users -to specify [compute resource requirements](resources.md) via requests and +to specify [compute resource requirements](/contributors/design-proposals/scheduling/resources.md) via requests and limits on individual containers. The `kubelet` delegates creation of a cgroup sandbox for each container to its associated container runtime. @@ -172,7 +172,7 @@ pod<UID>/cpu.shares = 2 ## QoS level cgroups The `kubelet` defines a `--cgroup-root` flag that is used to specify the `ROOT` -node in the cgroup hierarchy below which the `kubelet` should manange individual +node in the cgroup hierarchy below which the `kubelet` should manage individual cgroup sandboxes. It is strongly recommended that users keep the default value for `--cgroup-root` as `/` in order to avoid deep cgroup hierarchies. The `kubelet` creates a cgroup sandbox under the specified path `ROOT/kubepods` per @@ -222,7 +222,7 @@ resource reservation model as is provided via [node allocatable](node-allocatabl for system and kubernetes daemons. For operators that have this concern, the `kubelet` with opt-in configuration -will attempt to limit the abilty for a pod in a lower QoS tier to burst utilization +will attempt to limit the ability for a pod in a lower QoS tier to burst utilization of a compressible resource that was requested by a pod in a higher QoS tier. The `kubelet` will support a flag `experimental-qos-reserved` that @@ -341,7 +341,7 @@ we are not enabling this function by default, but would like operators that want to value resource priority over resource utilization to gather real-world feedback on its utility. -As a best practice, oeprators that want to provide a similar resource +As a best practice, operators that want to provide a similar resource reservation model for Guaranteed pods as we offer via enforcement of node allocatable are encouraged to schedule their Guaranteed pods first as it will ensure the Burstable and BestEffort tiers have had their QoS diff --git a/contributors/design-proposals/node/propagation.md b/contributors/design-proposals/node/propagation.md new file mode 100644 index 00000000..20cf58d9 --- /dev/null +++ b/contributors/design-proposals/node/propagation.md @@ -0,0 +1,311 @@ +# HostPath Volume Propagation + +## Abstract + +A proposal to add support for propagation mode in HostPath volume, which allows +mounts within containers to visible outside the container and mounts after pods +creation visible to containers. Propagation [modes] (https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt) contains "shared", "slave", "private", +"unbindable". Out of them, docker supports "shared" / "slave" / "private". + +Several existing issues and PRs were already created regarding that particular +subject: +* Capability to specify mount propagation mode of per volume with docker [#20698] (https://github.com/kubernetes/kubernetes/pull/20698) +* Set propagation to "shared" for hostPath volume [#31504] (https://github.com/kubernetes/kubernetes/pull/31504) + +## Use Cases + +1. (From @Kaffa-MY) Our team attempts to containerize flocker with zfs as back-end +storage, and launch them in DaemonSet. Containers in the same flocker node need +to read/write and share the same mounted volume. Currently the volume mount +propagation mode cannot be specified between the host and the container, and then +the volume mount of each container would be isolated from each other. +This use case is also referenced by Containerized Volume Client Drivers - Design +Proposal [#22216] (https://github.com/kubernetes/kubernetes/pull/22216) + +1. (From @majewsky) I'm currently putting the [OpenStack Swift object storage] (https://github.com/openstack/swift) into +k8s on CoreOS. Swift's storage services expect storage drives to be mounted at +/srv/node/{drive-id} (where {drive-id} is defined by the cluster's ring, the topology +description data structure which is shared between all cluster members). Because +there are several such services on each node (about a dozen, actually), I assemble +/srv/node in the host mount namespace, and pass it into the containers as a hostPath +volume. +Swift is designed such that drives can be mounted and unmounted at any time (most +importantly to hot-swap failed drives) and the services can keep running, but if +the services run in a private mount namespace, they won't see the mounts/unmounts +performed on the host mount namespace until the containers are restarted. +The slave mount namespace is the correct solution for this AFAICS. Until this +becomes available in k8s, we will have to have operations restart containers manually +based on monitoring alerts. + +1. (From @victorgp) When using CoreOS Container Linux that does not provides external fuse systems +like, in our case, GlusterFS, and you need a container to do the mounts. The only +way to see those mounts in the host, hence also visible by other containers, is by +sharing the mount propagation. + +1. (From @YorikSar) For OpenStack project, Neutron, we need network namespaces +created by it to persist across reboot of pods with Neutron agents. Without it +we have unnecessary data plane downtime during rolling update of these agents. +Neutron L3 agent creates interfaces and iptables rules for each virtual router +in a separate network namespace. For managing them it uses ip netns command that +creates persistent network namespaces by calling unshare(CLONE_NEWNET) and then +bind-mounting new network namespace's inode from /proc/self/ns/net to file with +specified name in /run/netns dir. These bind mounts are the only references to +these namespaces that remain. +When we restart the pod, its mount namespace is destroyed with all these bind +mounts, so all network namespaces created by the agent are gone. For them to +survive we need to bind mount a dir from host mount namespace to container one +with shared flag, so that all bind mounts are propagated across mount namespaces +and references to network namespaces persist. + +1. (From https://github.com/kubernetes/kubernetes/issues/46643) I expect the + container to start and any fuse mounts it creates in a volume that exists on + other containers in the pod (that are using :slave) are available to those + other containers. + + In other words, two containers in the same pod share an EmptyDir. One + container mounts something in it and the other one can see it. The first + container must have (r)shared mount propagation to the EmptyDir, the second + one can have (r)slave. + + +## Implementation Alternatives + +### Add an option in VolumeMount API + +The new `VolumeMount` will look like: + +```go +type MountPropagationMode string + +const ( + // MountPropagationHostToContainer means that the volume in a container will + // receive new mounts from the host or other containers, but filesystems + // mounted inside the container won't be propagated to the host or other + // containers. + // Note that this mode is recursively applied to all mounts in the volume + // ("rslave" in Linux terminology). + MountPropagationHostToContainer MountPropagationMode = "HostToContainer" + // MountPropagationBidirectional means that the volume in a container will + // receive new mounts from the host or other containers, and its own mounts + // will be propagated from the container to the host or other containers. + // Note that this mode is recursively applied to all mounts in the volume + // ("rshared" in Linux terminology). + MountPropagationBidirectional MountPropagationMode = "Bidirectional" +) + +type VolumeMount struct { + // Required: This must match the Name of a Volume [above]. + Name string `json:"name"` + // Optional: Defaults to false (read-write). + ReadOnly bool `json:"readOnly,omitempty"` + // Required. + MountPath string `json:"mountPath"` + // mountPropagation is the mode how are mounts in the volume propagated from + // the host to the container and from the container to the host. + // When not set, MountPropagationHostToContainer is used. + // This field is alpha in 1.8 and can be reworked or removed in a future + // release. + // Optional. + MountPropagation *MountPropagationMode `json:"mountPropagation,omitempty"` +} +``` + +Default would be `HostToContainer`, i.e. `rslave`, which should not break +backward compatibility, `Bidirectional` must be explicitly requested. +Using enum instead of simple `PropagateMounts bool` allows us to extend the +modes to `private` or non-recursive `shared` and `slave` if we need so in +future. + +Only privileged containers are allowed to use `Bidirectional` for their volumes. +This will be enforced during validation. + +Opinion against this: + +1. This will affect all volumes, while only HostPath need this. It could be +checked during validation and any non-HostPath volumes with non-default +propagation could be rejected. + +1. This need API change, which is discouraged. + +### Add an option in HostPathVolumeSource + +The new `HostPathVolumeSource` will look like: + +```go +type MountPropagationMode string + +const ( + // MountPropagationHostToContainer means that the volume in a container will + // receive new mounts from the host or other containers, but filesystems + // mounted inside the container won't be propagated to the host or other + // containers. + // Note that this mode is recursively applied to all mounts in the volume + // ("rslave" in Linux terminology). + MountPropagationHostToContainer MountPropagationMode = "HostToContainer" + // MountPropagationBidirectional means that the volume in a container will + // receive new mounts from the host or other containers, and its own mounts + // will be propagated from the container to the host or other containers. + // Note that this mode is recursively applied to all mounts in the volume + // ("rshared" in Linux terminology). + MountPropagationBidirectional MountPropagationMode = "Bidirectional" +) + +type HostPathVolumeSource struct { + Path string `json:"path"` + // mountPropagation is the mode how are mounts in the volume propagated from + // the host to the container and from the container to the host. + // When not set, MountPropagationHostToContainer is used. + // This field is alpha in 1.8 and can be reworked or removed in a future + // release. + // Optional. + MountPropagation *MountPropagationMode `json:"mountPropagation,omitempty"` +} +``` + +Default would be `HostToContainer`, i.e. `rslave`, which should not break +backward compatibility, `Bidirectional` must be explicitly requested. +Using enum instead of simple `PropagateMounts bool` allows us to extend the +modes to `private` or non-recursive `shared` and `slave` if we need so in +future. + +Only privileged containers can use HostPath with `Bidirectional` mount +propagation - kubelet silently downgrades the propagation to `HostToContainer` +when running `Bidirectional` HostPath in a non-privileged container. This allows +us to use the same `HostPathVolumeSource` in a pod with two containers, one +non-privileged with `HostToContainer` propagation and second privileged with +`Bidirectional` that mounts stuff for the first one. + +Opinion against this: + +1. This need API change, which is discouraged. + +1. All containers use this volume will share the same propagation mode. + +1. Silent downgrade from `Bidirectional` to `HostToContainer` for non-privileged + containers. + +1. (From @jonboulle) May cause cross-runtime compatibility issue. + +1. It's not possible to validate a pod + mount propagation. Mount propagation + is stored in a HostPath PersistentVolume object, while privileged mode is + stored in Pod object. Validator sees only one object and we don't do + cross-object validation and can't reject non-privileged pod that uses a PV + with shared mount propagation. + +### Make HostPath shared for privileged containers, slave for non-privileged. + +Given only HostPath needs this feature, and CAP_SYS_ADMIN access is needed when +making mounts inside container, we can bind propagation mode with existing option +privileged, or we can introduce a new option in SecurityContext to control this. + +The propagation mode could be determined by the following logic: + +```go +// Environment check to ensure "shared" is supported. +if !dockerNewerThanV110 || !mountPathIsShared { + return "" +} +if container.SecurityContext.Privileged { + return "shared" +} else { + return "slave" +} +``` + +Opinion against this: + +1. This changes the behavior of existing config. + +1. (From @euank) "shared" is not correctly supported by some kernels, we need +runtime support matrix and when that will be addressed. + +1. This may cause silently fail and be a debuggability nightmare on many +distros. + +1. (From @euank) Changing those mountflags may make docker even less stable, +this may lock up kernel accidentally or potentially leak mounts. + +1. (From @jsafrane) Typical container that needs to mount something needs to +see host's `/dev` and `/sys` as HostPath volumes. This would make them shared +without any way to opt-out. Docker creates a new `/dev/shm` in the +container, which gets propagated to the host, shadowing host's `/dev/shm`. +Similarly, systemd running in a container is very picky about `/sys/fs/cgroup` +and something prevents it from starting if `/sys` is shared. + +## Decision + +* We will take 'Add an option in VolumeMount API' + * With an alpha feature gate in 1.8. + * Only privileged containers can use `rshared` (`Bidirectional`) mount + propagation (with a validator). + +* During alpha, all the behavior above must be explicitly enabled by + `kubelet --feature-gates=MountPropagation=true` + It will be used only for testing of volume plugins in e2e tests and + Mount propagation may be redesigned or even removed in any future release. + + When the feature is enabled: + + * The default mount propagation of **all** volumes (incl. GCE, AWS, Cinder, + Gluster, Flex, ...) will be `slave`, which is different to current + `private`. Extensive testing is needed! We may restrict it to HostPath + + EmptyDir in Beta. + + * **Any** volume in a privileged container can be `Bidirectional`. We may + restrict it to HostPath + EmptyDir in Beta. + + * Kubelet's Docker shim layer will check that it is able to run a container + with shared mount propagation on `/var/lib/kubelet` during startup and log + a warning otherwise. This ensures that both Docker and kubelet see the same + `/var/lib/kubelet` and it can be shared into containers. + E.g. Google COS-58 runs Docker in a separate mount namespace with slave + propagation and thus can't run a container with shared propagation on + anything. + + This will be done via simple docker version check (1.13 is required) when + the feature gate is enabled. + + * Node conformance suite will check that mount propagation in /var/lib/kubelet + works. + + * When running on a distro with `private` as default mount propagation + (probably anything that does not run systemd, such as Debian Wheezy), + Kubelet will make `/var/lib/kubelet` share-able into containers and it will + refuse to start if it's unsuccessful. + + It sounds complicated, but it's simple + `mount --bind --rshared /var/lib/kubelet /var/lib/kubelet`. See + kubernetes/kubernetes#45724 + + +## Extra Concerns + +@lucab and @euank has some extra concerns about pod isolation when propagation +modes are changed, listed below: + +1. how to clean such pod resources (as mounts are now crossing pod boundaries, +thus they can be kept busy indefinitely by processes outside of the pod) + +1. side-effects on restarts (possibly piling up layers of full-propagation mounts) + +1. how does this interacts with other mount features (nested volumeMounts may or +may not propagate back to the host, depending of ordering of mount operations) + +1. limitations this imposes on runtimes (RO-remounting may now affects the host, +is it on purpose or a dangerous side-effect?) + +1. A shared mount target imposes some constraints on its parent subtree (generally, +it has to be shared as well), which in turn prevents some mount operations when +preparing a pod (eg. MS_MOVE). + +1. The "on-by-default" nature means existing hostpath mounts, which used to be +harmless, could begin consuming kernel resources and cause a node to crash. Even +if a pod does not create any new mountpoints under its hostpath bindmount, it's +not hard to reach multiplicative explosions with shared bindmounts and so the +change in default + no cleanup could result in existing workloads knocking the +node over. + +These concerns are valid and we decide to limit the propagation mode to HostPath +volume only, in HostPath, we expect any runtime should NOT perform any additional +actions (such as clean up). This behavior is also consistent with current HostPath +logic: kube does not take care of the content in HostPath either. diff --git a/contributors/design-proposals/resource-qos.md b/contributors/design-proposals/node/resource-qos.md index 13ad0bd4..238a9dac 100644 --- a/contributors/design-proposals/resource-qos.md +++ b/contributors/design-proposals/node/resource-qos.md @@ -20,11 +20,11 @@ Borg increased utilization by about 20% when it started allowing use of such non ## Requests and Limits -For each resource, containers can specify a resource request and limit, `0 <= request <= `[`Node Allocatable`](../design-proposals/node-allocatable.md) & `request <= limit <= Infinity`. +For each resource, containers can specify a resource request and limit, `0 <= request <= `[`Node Allocatable`](node-allocatable.md) & `request <= limit <= Infinity`. If a pod is successfully scheduled, the container is guaranteed the amount of resources requested. Scheduling is based on `requests` and not `limits`. The pods and its containers will not be allowed to exceed the specified limit. -How the request and limit are enforced depends on whether the resource is [compressible or incompressible](resources.md). +How the request and limit are enforced depends on whether the resource is [compressible or incompressible](../scheduling/resources.md). ### Compressible Resource Guarantees @@ -41,7 +41,7 @@ How the request and limit are enforced depends on whether the resource is [compr ### Admission/Scheduling Policy -- Pods will be admitted by Kubelet & scheduled by the scheduler based on the sum of requests of its containers. The scheduler & kubelet will ensure that sum of requests of all containers is within the node's [allocatable](../proposals/node-allocatable.md) capacity (for both memory and CPU). +- Pods will be admitted by Kubelet & scheduled by the scheduler based on the sum of requests of its containers. The scheduler & kubelet will ensure that sum of requests of all containers is within the node's [allocatable](node-allocatable.md) capacity (for both memory and CPU). ## QoS Classes @@ -213,6 +213,3 @@ A strict hierarchy of user-specified numerical priorities is not desirable becau 1. Achieved behavior would be emergent based on how users assigned priorities to their pods. No particular SLO could be delivered by the system, and usage would be subject to gaming if not restricted administratively 2. Changes to desired priority bands would require changes to all user pod configurations. -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/runtime-client-server.md b/contributors/design-proposals/node/runtime-client-server.md index 16cc677c..b50e003d 100644 --- a/contributors/design-proposals/runtime-client-server.md +++ b/contributors/design-proposals/node/runtime-client-server.md @@ -200,7 +200,3 @@ This proposal is first filed by [@brendandburns](https://github.com/brendandburn * [kubernetes/13709](https://github.com/kubernetes/kubernetes/pull/13079) * [New container runtime interface](https://github.com/kubernetes/kubernetes/pull/25899) - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/runtime-pod-cache.md b/contributors/design-proposals/node/runtime-pod-cache.md index d4926c3e..752741f1 100644 --- a/contributors/design-proposals/runtime-pod-cache.md +++ b/contributors/design-proposals/node/runtime-pod-cache.md @@ -28,7 +28,7 @@ pod cache, we can further improve Kubelet's CPU usage by need to inspect containers with no state changes. ***Don't we already have a [container runtime cache] -(https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/container/runtime_cache.go)?*** +(https://git.k8s.io/kubernetes/pkg/kubelet/container/runtime_cache.go)?*** The runtime cache is an optimization that reduces the number of `GetPods()` calls from the workers. However, @@ -167,7 +167,3 @@ by such periods and should improve Kubelet's perceived responsiveness. - Deprecate the old runtime cache. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/seccomp.md b/contributors/design-proposals/node/seccomp.md index 794e758c..f3e5ec67 100644 --- a/contributors/design-proposals/seccomp.md +++ b/contributors/design-proposals/node/seccomp.md @@ -248,7 +248,7 @@ metadata: spec: containers: - name: explorer - image: gcr.io/google_containers/explorer:1.0 + image: k8s.gcr.io/explorer:1.0 args: ["-port=8080"] ports: - containerPort: 8080 @@ -261,6 +261,3 @@ spec: emptyDir: {} ``` -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/secret-configmap-downwarapi-file-mode.md b/contributors/design-proposals/node/secret-configmap-downwardapi-file-mode.md index f9b3b5af..85ee9ccc 100644 --- a/contributors/design-proposals/secret-configmap-downwarapi-file-mode.md +++ b/contributors/design-proposals/node/secret-configmap-downwardapi-file-mode.md @@ -180,7 +180,3 @@ The are two downside: the file permissions will be the same on all. This is already the case for Key mappings and doesn't seem like a big issue either. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/selinux-enhancements.md b/contributors/design-proposals/node/selinux-enhancements.md index 3b3e168a..aec5533b 100644 --- a/contributors/design-proposals/selinux-enhancements.md +++ b/contributors/design-proposals/node/selinux-enhancements.md @@ -203,7 +203,3 @@ ensure things work as expected under rkt. The `VolumeHost` abstraction is used in a couple of PV controllers as NOP implementations. These should be altered to no longer include `GetRootContext`. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/selinux.md b/contributors/design-proposals/node/selinux.md index ece83d44..6cde471d 100644 --- a/contributors/design-proposals/selinux.md +++ b/contributors/design-proposals/node/selinux.md @@ -310,8 +310,3 @@ to manage labels individually. This allows the volume plugins to determine when they do and don't want this type of support from the Kubelet, and allows the criteria each plugin uses to evolve without changing the Kubelet. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/sysctl.md b/contributors/design-proposals/node/sysctl.md index d24c18ee..8ab61b8c 100644 --- a/contributors/design-proposals/sysctl.md +++ b/contributors/design-proposals/node/sysctl.md @@ -28,7 +28,6 @@ for namespaced kernel parameters (sysctls) set for each pod. ## Table of Contents -<!-- BEGIN MUNGE: GENERATED_TOC --> - [Setting Sysctls on the Pod Level](#setting-sysctls-on-the-pod-level) - [Roadmap](#roadmap) @@ -63,7 +62,6 @@ for namespaced kernel parameters (sysctls) set for each pod. - [Use in a pod](#use-in-a-pod) - [Allowing only certain sysctls](#allowing-only-certain-sysctls) -<!-- END MUNGE: GENERATED_TOC --> ## Abstract @@ -126,7 +124,7 @@ Some real-world examples for the use of sysctls: - a containerized IPv6 routing daemon requires e.g. `/proc/sys/net/ipv6/conf/all/forwarding` and `/proc/sys/net/ipv6/conf/all/accept_redirects` (compare [docker#4717](https://github.com/docker/docker/issues/4717#issuecomment-98653017)) -- the [nginx ingress controller in kubernetes/contrib](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/examples/sysctl/change-proc-values-rc.yaml#L80) +- the [nginx ingress controller in kubernetes/contrib](https://git.k8s.io/contrib/ingress/controllers/nginx/examples/sysctl/change-proc-values-rc.yaml#L80) uses a privileged sidekick container to set `net.core.somaxconn` and `net.ipv4.ip_local_port_range`. - a huge software-as-a-service provider uses shared memory (`kernel.shm*`) and message queues (`kernel.msg*`) to communicate between containers of their web-serving pods, configuring up to 20 GB of shared memory. @@ -234,7 +232,7 @@ Applied changes: ### Rkt support for sysctl -The only sysctl support in rkt is through a [CNI plugin](https://github.com/containernetworking/cni/blob/master/Documentation/tuning.md) plugin. The Kubernetes network plugin `kubenet` can easily be extended to call this with a given list of sysctls during pod launch. +The only sysctl support in rkt is through a [CNI plugin](https://github.com/containernetworking/plugins/blob/master/plugins/meta/tuning/README.md) plugin. The Kubernetes network plugin `kubenet` can easily be extended to call this with a given list of sysctls during pod launch. The default network plugin for rkt is `no-op` though. This mode leaves all network initialization to rkt itself. Rkt in turn uses the static CNI plugin configuration in `/etc/rkt/net.d`. This does not allow to customize the sysctls for a pod. Hence, in order to implement this proposal in `no-op` mode additional changes in rkt are necessary. @@ -253,7 +251,7 @@ Issues: ## Design Alternatives and Considerations - Each pod has its own network stack that is shared among its containers. - A privileged side-kick or init container (compare https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/examples/sysctl/change-proc-values-rc.yaml#L80) + A privileged side-kick or init container (compare https://git.k8s.io/contrib/ingress/controllers/nginx/examples/sysctl/change-proc-values-rc.yaml#L80) is able to set `net.*` sysctls. Clearly, this is completely uncontrolled by the kubelet, but is a usable work-around if privileged @@ -561,7 +559,7 @@ During kubelet launch the given value is checked against the list of known names #### Alternative 1: by name -A list of permissible sysctls is to be added to `pkg/apis/extensions/types.go` (compare [security-context-constraints](security-context-constraints.md)): +A list of permissible sysctls is to be added to `pkg/apis/extensions/types.go` (compare [pod-security-policy](pod-security-policy.md)): ```go // PodSecurityPolicySpec defines the policy enforced. diff --git a/contributors/design-proposals/node/troubleshoot-running-pods.md b/contributors/design-proposals/node/troubleshoot-running-pods.md new file mode 100644 index 00000000..b72102f3 --- /dev/null +++ b/contributors/design-proposals/node/troubleshoot-running-pods.md @@ -0,0 +1,738 @@ +# Troubleshoot Running Pods + +* Status: Pending +* Version: Alpha +* Implementation Owner: @verb + +This proposal seeks to add first class support for troubleshooting by creating a +mechanism to execute a shell or other troubleshooting tools inside a running pod +without requiring that the associated container images include such tools. + +## Motivation + +### Development + +Many developers of native Kubernetes applications wish to treat Kubernetes as an +execution platform for custom binaries produced by a build system. These users +can forgo the scripted OS install of traditional Dockerfiles and instead `COPY` +the output of their build system into a container image built `FROM scratch` or +a [distroless container +image](https://github.com/GoogleCloudPlatform/distroless). This confers several +advantages: + +1. **Minimal images** lower operational burden and reduce attack vectors. +1. **Immutable images** improve correctness and reliability. +1. **Smaller image size** reduces resource usage and speeds deployments. + +The disadvantage of using containers built `FROM scratch` is the lack of system +binaries provided by an Operating System image makes it difficult to +troubleshoot running containers. Kubernetes should enable one to troubleshoot +pods regardless of the contents of the container images. + +### Operations and Support + +As Kubernetes gains in popularity, it's becoming the case that a person +troubleshooting an application is not necessarily the person who built it. +Operations staff and Support organizations want the ability to attach a "known +good" or automated debugging environment to a pod. + +## Requirements + +A solution to troubleshoot arbitrary container images MUST: + +* troubleshoot arbitrary running containers with minimal prior configuration +* allow access to namespaces and the file systems of individual containers +* fetch troubleshooting utilities at debug time rather than at the time of pod + creation +* be compatible with admission controllers and audit logging +* allow discovery of debugging status +* support arbitrary runtimes via the CRI (possibly with reduced feature set) +* require no administrative access to the node +* have an excellent user experience (i.e. should be a feature of the platform + rather than config-time trickery) +* have no *inherent* side effects to the running container image + +## Feature Summary + +Any new debugging functionality will require training users. We can ease the +transition by building on an existing usage pattern. We will create a new +command, `kubectl debug`, which parallels an existing command, `kubectl exec`. +Whereas `kubectl exec` runs a *process* in a *container*, `kubectl debug` will +be similar but run a *container* in a *pod*. + +A container created by `kubectl debug` is a *Debug Container*. Just like a +process run by `kubectl exec`, a Debug Container is not part of the pod spec and +has no resource stored in the API. Unlike `kubectl exec`, a Debug Container +*does* have status that is reported in `v1.PodStatus` and displayed by `kubectl +describe pod`. + +For example, the following command would attach to a newly created container in +a pod: + +``` +kubectl debug -c debug-shell --image=debian target-pod -- bash +``` + +It would be reasonable for Kubernetes to provide a default container name and +image, making the minimal possible debug command: + +``` +kubectl debug target-pod +``` + +This creates an interactive shell in a pod which can examine and signal other +processes in the pod. It has access to the same network and IPC as processes in +the pod. It can access the filesystem of other processes by `/proc/$PID/root`. +As is already the case with regular containers, Debug Containers can enter +arbitrary namespaces of another container via `nsenter` when run with +`CAP_SYS_ADMIN`. + +*Please see the User Stories section for additional examples and Alternatives +Considered for the considerable list of other solutions we considered.* + +## Implementation Details + +The implementation of `kubectl debug` closely mirrors the implementation of +`kubectl exec`, with most of the complexity implemented in the `kubelet`. How +functionality like this best fits into Kubernetes API has been contentious. In +order to make progress, we will start with the smallest possible API change, +extending `/exec` to support Debug Containers, and iterate. + +From the perspective of the user, there's a new command, `kubectl debug`, that +creates a Debug Container and attaches to its console. We believe a new command +will be less confusing for users than overloading `kubectl exec` with a new +concept. Users give Debug Containers a name (e.g. "debug" or "shell") which can +subsequently be used to reattach and is reported by `kubectl describe`. + +### Kubernetes API Changes + +#### Chosen Solution: "exec++" + +We will extend `v1.Pod`'s `/exec` subresource to support "executing" container +images. The current `/exec` endpoint must implement `GET` to support streaming +for all clients. We don't want to encode a (potentially large) `v1.Container` as +an HTTP parameter, so we must extend `v1.PodExecOptions` with the specific +fields required for creating a Debug Container: + +``` +// PodExecOptions is the query options to a Pod's remote exec call +type PodExecOptions struct { + ... + // EphemeralContainerName is the name of an ephemeral container in which the + // command ought to be run. Either both EphemeralContainerName and + // EphemeralContainerImage fields must be set, or neither. + EphemeralContainerName *string `json:"ephemeralContainerName,omitempty" ...` + + // EphemeralContainerImage is the image of an ephemeral container in which the command + // ought to be run. Either both EphemeralContainerName and EphemeralContainerImage + // fields must be set, or neither. + EphemeralContainerImage *string `json:"ephemeralContainerImage,omitempty" ...` +} +``` + +After creating the Debug Container, the kubelet will upgrade the connection to +streaming and perform an attach to the container's console. If disconnected, the +Debug Container can be reattached using the pod's `/attach` endpoint with +`EphemeralContainerName`. + +Debug Containers cannot be removed via the API and instead the process must +terminate. While not ideal, this parallels existing behavior of `kubectl exec`. +To kill a Debug Container one would `attach` and exit the process interactively +or create a new Debug Container to send a signal with `kill(1)` to the original +process. + +#### Alternative 1: Debug Subresource + +Rather than extending an existing subresource, we could create a new, +non-streaming `debug` subresource. We would create a new API Object: + +``` +// DebugContainer describes a container to attach to a running pod for troubleshooting. +type DebugContainer struct { + metav1.TypeMeta + metav1.ObjectMeta + + // Name is the name of the Debug Container. Its presence will cause + // exec to create a Debug Container rather than performing a runtime exec. + Name string `json:"name,omitempty" ...` + + // Image is an optional container image name that will be used to for the Debug + // Container in the specified Pod with Command as ENTRYPOINT. If omitted a + // default image will be used. + Image string `json:"image,omitempty" ...` +} +``` + +The pod would gain a new `/debug` subresource that allows the following: + +1. A `POST` of a `PodDebugContainer` to + `/api/v1/namespaces/$NS/pods/$POD_NAME/debug/$NAME` to create Debug + Container named `$NAME` running in pod `$POD_NAME`. +1. A `DELETE` of `/api/v1/namespaces/$NS/pods/$POD_NAME/debug/$NAME` will stop + the Debug Container `$NAME` in pod `$POD_NAME`. + +Once created, a client would attach to the console of a debug container using +the existing attach endpoint, `/api/v1/namespaces/$NS/pods/$POD_NAME/attach`. + +However, this pattern does not resemble any other current usage of the API, so +we prefer to start with exec++ and reevaluate if we discover a compelling +reason. + +#### Alternative 2: Declarative Configuration + +Using subresources is an imperative style API where the client instructs the +kubelet to perform an action, but in general Kubernetes prefers declarative APIs +where the client declares a state for Kubernetes to enact. + +We could implement this in a declarative manner by creating a new +`EphemeralContainer` type: + +``` +type EphemeralContainer struct { + metav1.TypeMeta + metav1.ObjectMeta + + Spec EphemeralContainerSpec + Status v1.ContainerStatus +} +``` + +`EphemeralContainerSpec` is similar to `v1.Container`, but contains only fields +relevant to Debug Containers: + +``` +type EphemeralContainerSpec struct { + // Target is the pod in which to run the EphemeralContainer + // Required. + Target v1.ObjectReference + + Name string + Image String + Command []string + Args []string + ImagePullPolicy PullPolicy + SecurityContext *SecurityContext +} +``` + +A new controller in the kubelet would watch for EphemeralContainers and +create/delete debug containers. `EphemeralContainer.Status` would be updated by +the kubelet at the same time it updates `ContainerStatus` for regular and init +containers. Clients would create a new `EphemeralContainer` object, wait for it +to be started and then attach using the pod's attach subresource and the name of +the `EphemeralContainer`. + +Debugging is inherently imperative, however, rather than a state for Kubernetes +to enforce. Once a Debug Container is started it should not be automatically +restarted, for example. This solution imposes additionally complexity and +dependencies on the kubelet, but it's not yet clear if the complexity is +justified. + +### Debug Container Status + +The status of a Debug Container is reported in a new field in `v1.PodStatus`: + +``` +type PodStatus struct { + ... + EphemeralContainerStatuses []v1.ContainerStatus +} +``` + +This status is only populated for Debug Containers, but there's interest in +tracking status for traditional exec in a similar manner. + +Note that `Command` and `Args` would have to be tracked in the status object +because there is no spec for Debug Containers or exec. These must either be made +available by the runtime or tracked by the kubelet. For Debug Containers this +could be stored as runtime labels, but the kubelet currently has no method of +storing state across restarts for exec. Solving this problem for exec is out of +scope for Debug Containers, but we will look for a solution as we implement this +feature. + +`EphemeralContainerStatuses` is populated by the kubelet in the same way as +regular and init container statuses. This is sent to the API server and +displayed by `kubectl describe pod`. + +### Creating Debug Containers + +1. `kubectl` invokes the exec API as described in the preceding section. +1. The API server checks for name collisions with existing containers, performs + admission control and proxies the connection to the kubelet's + `/exec/$NS/$POD_NAME/$CONTAINER_NAME` endpoint. +1. The kubelet instructs the Runtime Manager to create a Debug Container. +1. The runtime manager uses the existing `startContainer()` method to create a + container in an existing pod. `startContainer()` has one modification for + Debug Containers: it creates a new runtime label (e.g. a docker label) that + identifies this container as a Debug Container. +1. After creating the container, the kubelet schedules an asynchronous update + of `PodStatus`. The update publishes the debug container status to the API + server at which point the Debug Container becomes visible via `kubectl + describe pod`. +1. The kubelet will upgrade the connection to streaming and attach to the + container's console. + +Rather than performing the implicit attach the kubelet could return success to +the client and require the client to perform an explicit attach, but the +implicit attach maintains consistent semantics across `/exec` rather than +varying behavior based on parameters. + +The apiserver detects container name collisions with both containers in the pod +spec and other running Debug Containers by checking +`EphemeralContainerStatuses`. In a race to create two Debug Containers with the +same name, the API server will pass both requests and the kubelet must return an +error to all but one request. + +There are no limits on the number of Debug Containers that can be created in a +pod, but exceeding a pod's resource allocation may cause the pod to be evicted. + +### Restarting and Reattaching Debug Containers + +Debug Containers will never be restarted automatically. It is possible to +replace a Debug Container that has exited by re-using a Debug Container name. It +is an error to attempt to replace a Debug Container that is still running, which +is detected by both the API server and the kubelet. + +One can reattach to a Debug Container using `kubectl attach`. When supported by +a runtime, multiple clients can attach to a single debug container and share the +terminal. This is supported by Docker. + +### Killing Debug Containers + +Debug containers will not be killed automatically until the pod (specifically, +the pod sandbox) is destroyed. Debug Containers will stop when their command +exits, such as exiting a shell. Unlike `kubectl exec`, processes in Debug +Containers will not receive an EOF if their connection is interrupted. + +### Container Lifecycle Changes + +Implementing debug requires no changes to the Container Runtime Interface as +it's the same operation as creating a regular container. The following changes +are necessary in the kubelet: + +1. `SyncPod()` must not kill any Debug Container even though it is not part of + the pod spec. +1. As an exception to the above, `SyncPod()` will kill Debug Containers when + the pod sandbox changes since a lone Debug Container in an abandoned sandbox + is not useful. Debug Containers are not automatically started in the new + sandbox. +1. `convertStatusToAPIStatus()` must sort Debug Containers status into + `EphemeralContainerStatuses` similar to as it does for + `InitContainerStatuses` +1. The kubelet must preserve `ContainerStatus` on debug containers for + reporting. +1. Debug Containers must be excluded from calculation of pod phase and + condition + +It's worth noting some things that do not change: + +1. `KillPod()` already operates on all running containers returned by the + runtime. +1. Containers created prior to this feature being enabled will have a + `containerType` of `""`. Since this does not match `"EPHEMERAL"` the special + handling of Debug Containers is backwards compatible. + +### Security Considerations + +Debug Containers have no additional privileges above what is available to any +`v1.Container`. It's the equivalent of configuring an shell container in a pod +spec but created on demand. + +Admission plugins that guard `/exec` must be updated for the new parameters. In +particular, they should enforce the same container image policy on the `Image` +parameter as is enforced for regular containers. During the alpha phase we will +additionally support a container image whitelist as a kubelet flag to allow +cluster administrators to easily constraint debug container images. + +### Additional Consideration + +1. Debug Containers are intended for interactive use and always have TTY and + Stdin enabled. +1. There are no guaranteed resources for ad-hoc troubleshooting. If + troubleshooting causes a pod to exceed its resource limit it may be evicted. +1. There's an output stream race inherent to creating then attaching a + container which causes output generated between the start and attach to go + to the log rather than the client. This is not specific to Debug Containers + and exists because Kubernetes has no mechanism to attach a container prior + to starting it. This larger issue will not be addressed by Debug Containers, + but Debug Containers would benefit from future improvements or work arounds. +1. We do not want to describe Debug Containers using `v1.Container`. This is to + reinforce that Debug Containers are not general purpose containers by + limiting their configurability. Debug Containers should not be used to build + services. +1. Debug Containers are of limited usefulness without a shared PID namespace. + If a pod is configured with isolated PID namespaces, the Debug Container + will join the PID namespace of the target container. Debug Containers will + not be available with runtimes that do not implement PID namespace sharing + in some form. + +## Implementation Plan + +### Alpha Release + +#### Goals and Non-Goals for Alpha Release + +We're targeting an alpha release in Kubernetes 1.9 that includes the following +basic functionality: + +* Support in the kubelet for creating debug containers in a running pod +* A `kubectl debug` command to initiate a debug container +* `kubectl describe pod` will list status of debug containers running in a pod + +Functionality will be hidden behind an alpha feature flag and disabled by +default. The following are explicitly out of scope for the 1.9 alpha release: + +* Exited Debug Containers will be garbage collected as regular containers and + may disappear from the list of Debug Container Statuses. +* Security Context for the Debug Container is not configurable. It will always + be run with `CAP_SYS_PTRACE` and `CAP_SYS_ADMIN`. +* Image pull policy for the Debug Container is not configurable. It will + always be run with `PullAlways`. + +#### kubelet Implementation + +Debug Containers are implemented in the kubelet's generic runtime manager. +Performing this operation with a legacy (non-CRI) runtime will result in a not +implemented error. Implementation in the kubelet will be split into the +following steps: + +##### Step 1: Container Type + +The first step is to add a feature gate to ensure all changes are off by +default. This will be added in the `pkg/features` `DefaultFeatureGate`. + +The runtime manager stores metadata about containers in the runtime via labels +(e.g. docker labels). These labels are used to populate the fields of +`kubecontainer.ContainerStatus`. Since the runtime manager needs to handle Debug +Containers differently in a few situations, we must add a new piece of metadata +to distinguish Debug Containers from regular containers. + +`startContainer()` will be updated to write a new label +`io.kubernetes.container.type` to the runtime. Existing containers will be +started with a type of `REGULAR` or `INIT`. When added in a subsequent step, +Debug Containers will start with with the type `EPHEMERAL`. + +##### Step 2: Creation and Handling of Debug Containers + +This step adds methods for creating debug containers, but doesn't yet modify the +kubelet API. Since the runtime manager discards runtime (e.g. docker) labels +after populating `kubecontainer.ContainerStatus`, the label value will be stored +in a the new field `ContainerStatus.Type` so it can be used by `SyncPod()`. + +The kubelet gains a `RunDebugContainer()` method which accepts a `v1.Container` +and passes it on to the Runtime Manager's `RunDebugContainer()` if implemented. +Currently only the Generic Runtime Manager (i.e. the CRI) implements the +`DebugContainerRunner` interface. + +The Generic Runtime Manager's `RunDebugContainer()` calls `startContainer()` to +create the Debug Container. Additionally, `SyncPod()` is modified to skip Debug +Containers unless the sandbox is restarted. + +##### Step 3: kubelet API changes + +The kubelet exposes the new functionality in its existing `/exec/` endpoint. +`ServeExec()` constructs a `v1.Container` based on `PodExecOptions`, calls +`RunDebugContainer()`, and performs the attach. + +##### Step 4: Reporting EphemeralContainerStatus + +The last major change to the kubelet is to populate +v1.`PodStatus.EphemeralContainerStatuses` based on the +`kubecontainer.ContainerStatus` for the Debug Container. + +#### Kubernetes API Changes + +There are two changes to be made to the Kubernetes, which will be made +independently: + +1. `v1.PodExecOptions` must be extended with new fields. +1. `v1.PodStatus` gains a new field to hold Debug Container statuses. + +In all cases, new fields will be prepended with `Alpha` for the duration of this +feature's alpha status. + +#### kubectl changes + +In anticipation of this change, [#46151](https://pr.k8s.io/46151) added a +`kubectl alpha` command to contain alpha features. We will add `kubectl alpha +debug` to invoke Debug Containers. `kubectl` does not use feature gates, so +`kubectl alpha debug` will be visible by default in `kubectl` 1.9 and return an +error when used on a cluster with the feature disabled. + +`kubectl describe pod` will report the contents of `EphemeralContainerStatuses` +when not empty as it means the feature is enabled. The field will be hidden when +empty. + +## Appendices + +We've researched many options over the life of this proposal. These Appendices +are included as optional reference material. It's not necessary to read this +material in order to understand the proposal in its current form. + +### Appendix 1: User Stories + +These user stories are intended to give examples how this proposal addresses the +above requirements. + +#### Operations + +Jonas runs a service "neato" that consists of a statically compiled Go binary +running in a minimal container image. One of the its pods is suddenly having +trouble connecting to an internal service. Being in operations, Jonas wants to +be able to inspect the running pod without restarting it, but he doesn't +necessarily need to enter the container itself. He wants to: + +1. Inspect the filesystem of target container +1. Execute debugging utilities not included in the container image +1. Initiate network requests from the pod network namespace + +This is achieved by running a new "debug" container in the pod namespaces. His +troubleshooting session might resemble: + +``` +% kubectl debug -it -m debian neato-5thn0 -- bash +root@debug-image:~# ps x + PID TTY STAT TIME COMMAND + 1 ? Ss 0:00 /pause + 13 ? Ss 0:00 bash + 26 ? Ss+ 0:00 /neato + 107 ? R+ 0:00 ps x +root@debug-image:~# cat /proc/26/root/etc/resolv.conf +search default.svc.cluster.local svc.cluster.local cluster.local +nameserver 10.155.240.10 +options ndots:5 +root@debug-image:~# dig @10.155.240.10 neato.svc.cluster.local. + +; <<>> DiG 9.9.5-9+deb8u6-Debian <<>> @10.155.240.10 neato.svc.cluster.local. +; (1 server found) +;; global options: +cmd +;; connection timed out; no servers could be reached +``` + +Thus Jonas discovers that the cluster's DNS service isn't responding. + +#### Debugging + +Thurston is debugging a tricky issue that's difficult to reproduce. He can't +reproduce the issue with the debug build, so he attaches a debug container to +one of the pods exhibiting the problem: + +``` +% kubectl debug -it --image=gcr.io/neato/debugger neato-5x9k3 -- sh +Defaulting container name to debug. +/ # ps x +PID USER TIME COMMAND + 1 root 0:00 /pause + 13 root 0:00 /neato + 26 root 0:00 sh + 32 root 0:00 ps x +/ # gdb -p 13 +... +``` + +He discovers that he needs access to the actual container, which he can achieve +by installing busybox into the target container: + +``` +root@debug-image:~# cp /bin/busybox /proc/13/root +root@debug-image:~# nsenter -t 13 -m -u -p -n -r /busybox sh + + +BusyBox v1.22.1 (Debian 1:1.22.0-9+deb8u1) built-in shell (ash) +Enter 'help' for a list of built-in commands. + +/ # ls -l /neato +-rwxr-xr-x 2 0 0 746888 May 4 2016 /neato +``` + +Note that running the commands referenced above require `CAP_SYS_ADMIN` and +`CAP_SYS_PTRACE`. + +#### Automation + +Ginger is a security engineer tasked with running security audits across all of +her company's running containers. Even though his company has no standard base +image, she's able to audit all containers using: + +``` +% for pod in $(kubectl get -o name pod); do + kubectl debug -m gcr.io/neato/security-audit -p $pod /security-audit.sh + done +``` + +#### Technical Support + +Roy's team provides support for his company's multi-tenant cluster. He can +access the Kubernetes API (as a viewer) on behalf of the users he's supporting, +but he does not have administrative access to nodes or a say in how the +application image is constructed. When someone asks for help, Roy's first step +is to run his team's autodiagnose script: + +``` +% kubectl debug --image=k8s.gcr.io/autodiagnose nginx-pod-1234 +``` + +### Appendix 2: Requirements Analysis + +Many people have proposed alternate solutions to this problem. This section +discusses how the proposed solution meets all of the stated requirements and is +intended to contrast the alternatives listed below. + +**Troubleshoot arbitrary running containers with minimal prior configuration.** +This solution requires no prior configuration. + +**Access to namespaces and the file systems of individual containers.** This +solution runs a container in the shared pod namespaces (e.g. network) and will +attach to the PID namespace of a target container when not shared with the +entire pod. It relies on the behavior of `/proc/<pid>/root` to provide access to +filesystems of individual containers. + +**Fetch troubleshooting utilities at debug time**. This solution uses normal +container image distribution mechanisms to fetch images when the debug command +is run. + +**Respect admission restrictions.** Requests from kubectl are proxied through +the apiserver and so are available to existing [admission +controllers](https://kubernetes.io/docs/admin/admission-controllers/). Plugins +already exist to intercept `exec` and `attach` calls, but extending this to +support `debug` has not yet been scoped. + +**Allow introspection of pod state using existing tools**. The list of +`EphemeralContainerStatuses` is never truncated. If a debug container has run in +this pod it will appear here. + +**Support arbitrary runtimes via the CRI**. This proposal is implemented +entirely in the kubelet runtime manager and requires no changes in the +individual runtimes. + +**Have an excellent user experience**. This solution is conceptually +straightforward and surfaced in a single `kubectl` command that "runs a thing in +a pod". Debug tools are distributed by container image, which is already well +understood by users. There is no automatic copying of files or hidden paths. + +By using container images, users are empowered to create custom debug images. +Available images can be restricted by admission policy. Some examples of +possible debug images: + +* A script that automatically gathers a debugging snapshot and uploads it to a + cloud storage bucket before killing the pod. +* An image with a shell modified to log every statement to an audit API. + +**Require no direct access to the node.** This solution uses the standard +streaming API. + +**Have no inherent side effects to the running container image.** The target pod +is not modified by default, but resources used by the debug container will be +billed to the pod's cgroup, which means it could be evicted. A future +improvement could be to decrease the likelihood of eviction when there's an +active debug container. + +### Appendix 3: Alternatives Considered + +#### Mutable Pod Spec + +Rather than adding an operation to have Kubernetes attach a pod we could instead +make the pod spec mutable so the client can generate an update adding a +container. `SyncPod()` has no issues adding the container to the pod at that +point, but an immutable pod spec has been a basic assumption in Kubernetes thus +far and changing it carries risk. It's preferable to keep the pod spec immutable +as a best practice. + +#### Ephemeral container + +An earlier version of this proposal suggested running an ephemeral container in +the pod namespaces. The container would not be added to the pod spec and would +exist only as long as the process it ran. This has the advantage of behaving +similarly to the current kubectl exec, but it is opaque and likely violates +design assumptions. We could add constructs to track and report on both +traditional exec process and exec containers, but this would probably be more +work than adding to the pod spec. Both are generally useful, and neither +precludes the other in the future, so we chose mutating the pod spec for +expedience. + +#### Attaching Container Type Volume + +Combining container volumes ([#831](https://issues.k8s.io/831)) with the ability +to add volumes to the pod spec would get us most of the way there. One could +mount a volume of debug utilities at debug time. Docker does not allow adding a +volume to a running container, however, so this would require a container +restart. A restart doesn't meet our requirements for troubleshooting. + +Rather than attaching the container at debug time, kubernetes could always +attach a volume at a random path at run time, just in case it's needed. Though +this simplifies the solution by working within the existing constraints of +`kubectl exec`, it has a sufficient list of minor limitations (detailed in +[#10834](https://issues.k8s.io/10834)) to result in a poor user experience. + +#### Inactive container + +If Kubernetes supported the concept of an "inactive" container, we could +configure it as part of a pod and activate it at debug time. In order to avoid +coupling the debug tool versions with those of the running containers, we would +need to ensure the debug image was pulled at debug time. The container could +then be run with a TTY and attached using kubectl. We would need to figure out a +solution that allows access the filesystem of other containers. + +The downside of this approach is that it requires prior configuration. In +addition to requiring prior consideration, it would increase boilerplate config. +A requirement for prior configuration makes it feel like a workaround rather +than a feature of the platform. + +#### Implicit Empty Volume + +Kubernetes could implicitly create an EmptyDir volume for every pod which would +then be available as target for either the kubelet or a sidecar to extract a +package of binaries. + +Users would have to be responsible for hosting a package build and distribution +infrastructure or rely on a public one. The complexity of this solution makes it +undesirable. + +#### Standalone Pod in Shared Namespace + +Rather than inserting a new container into a pod namespace, Kubernetes could +instead support creating a new pod with container namespaces shared with +another, target pod. This would be a simpler change to the Kubernetes API, which +would only need a new field in the pod spec to specify the target pod. To be +useful, the containers in this "Debug Pod" should be run inside the namespaces +(network, pid, etc) of the target pod but remain in a separate resource group +(e.g. cgroup for container-based runtimes). + +This would be a rather fundamental change to pod, which is currently treated as +an atomic unit. The Container Runtime Interface has no provisions for sharing +outside of a pod sandbox and would need a refactor. This could be a complicated +change for non-container runtimes (e.g. hypervisor runtimes) which have more +rigid boundaries between pods. + +Effectively, Debug Pod must be implemented by the runtimes while Debug +Containers are implemented by the kubelet. Minimizing change to the Kubernetes +API is not worth the increased complexity for the kubelet and runtimes. + +It could also be possible to implement a Debug Pod as a privileged pod that runs +in the host namespace and interacts with the runtime directly to run a new +container in the appropriate namespace. This solution would be runtime-specific +and effectively pushes the complexity of debugging to the user. Additionally, +requiring node-level access to debug a pod does not meet our requirements. + +#### Exec from Node + +The kubelet could support executing a troubleshooting binary from the node in +the namespaces of the container. Once executed this binary would lose access to +other binaries from the node, making it of limited utility and a confusing user +experience. + +This couples the debug tools with the lifecycle of the node, which is worse than +coupling it with container images. + +## Reference + +* [Pod Troubleshooting Tracking Issue](https://issues.k8s.io/27140) +* [CRI Tracking Issue](https://issues.k8s.io/28789) +* [CRI: expose optional runtime features](https://issues.k8s.io/32803) +* [Resource QoS in + Kubernetes](https://git.k8s.io/kubernetes/docs/design/resource-qos.md) +* Related Features + * [#1615](https://issues.k8s.io/1615) - Shared PID Namespace across + containers in a pod + * [#26751](https://issues.k8s.io/26751) - Pod-Level cgroup + * [#10782](https://issues.k8s.io/10782) - Vertical pod autoscaling diff --git a/contributors/design-proposals/propagation.md b/contributors/design-proposals/propagation.md deleted file mode 100644 index 03d4124c..00000000 --- a/contributors/design-proposals/propagation.md +++ /dev/null @@ -1,187 +0,0 @@ -# HostPath Volume Propagation - -## Abstract - -A proposal to add support for propagation mode in HostPath volume, which allows -mounts within containers to visible outside the container and mounts after pods -creation visible to containers. Propagation [modes] (https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt) contains "shared", "slave", "private", -"unbindable". Out of them, docker supports "shared" / "slave" / "private". - -Several existing issues and PRs were already created regarding that particular -subject: -* Capability to specify mount propagation mode of per volume with docker [#20698] (https://github.com/kubernetes/kubernetes/pull/20698) -* Set propagation to "shared" for hostPath volume [#31504] (https://github.com/kubernetes/kubernetes/pull/31504) - -## Use Cases - -1. (From @Kaffa-MY) Our team attempts to containerize flocker with zfs as back-end -storage, and launch them in DaemonSet. Containers in the same flocker node need -to read/write and share the same mounted volume. Currently the volume mount -propagation mode cannot be specified between the host and the container, and then -the volume mount of each container would be isolated from each other. -This use case is also referenced by Containerized Volume Client Drivers - Design -Proposal [#22216] (https://github.com/kubernetes/kubernetes/pull/22216) - -1. (From @majewsky) I'm currently putting the [OpenStack Swift object storage] (https://github.com/openstack/swift) into -k8s on CoreOS. Swift's storage services expect storage drives to be mounted at -/srv/node/{drive-id} (where {drive-id} is defined by the cluster's ring, the topology -description data structure which is shared between all cluster members). Because -there are several such services on each node (about a dozen, actually), I assemble -/srv/node in the host mount namespace, and pass it into the containers as a hostPath -volume. -Swift is designed such that drives can be mounted and unmounted at any time (most -importantly to hot-swap failed drives) and the services can keep running, but if -the services run in a private mount namespace, they won't see the mounts/unmounts -performed on the host mount namespace until the containers are restarted. -The slave mount namespace is the correct solution for this AFAICS. Until this -becomes available in k8s, we will have to have operations restart containers manually -based on monitoring alerts. - -1. (From @victorgp) When using CoreOS Container Linux that does not provides external fuse systems -like, in our case, GlusterFS, and you need a container to do the mounts. The only -way to see those mounts in the host, hence also visible by other containers, is by -sharing the mount propagation. - -1. (From @YorikSar) For OpenStack project, Neutron, we need network namespaces -created by it to persist across reboot of pods with Neutron agents. Without it -we have unnecessary data plane downtime during rolling update of these agents. -Neutron L3 agent creates interfaces and iptables rules for each virtual router -in a separate network namespace. For managing them it uses ip netns command that -creates persistent network namespaces by calling unshare(CLONE_NEWNET) and then -bind-mounting new network namespace's inode from /proc/self/ns/net to file with -specified name in /run/netns dir. These bind mounts are the only references to -these namespaces that remain. -When we restart the pod, its mount namespace is destroyed with all these bind -mounts, so all network namespaces created by the agent are gone. For them to -survive we need to bind mount a dir from host mount namespace to container one -with shared flag, so that all bind mounts are propagated across mount namespaces -and references to network namespaces persist. - - -## Implementation Alternatives - -### Add an option in VolumeMount API - -The new `VolumeMount` will look like: - -```go -type VolumeMount struct { - // Required: This must match the Name of a Volume [above]. - Name string `json:"name"` - // Optional: Defaults to false (read-write). - ReadOnly bool `json:"readOnly,omitempty"` - // Required. - MountPath string `json:"mountPath"` - // Optional. - Propagation string `json:"propagation"` -} -``` - -Opinion against this: - -1. This will affect all volumes, while only HostPath need this. - -1. This need API change, which is discouraged. - -### Add an option in HostPathVolumeSource - -The new `HostPathVolumeSource` will look like: - -```go -const ( - PropagationShared PropagationMode = "Shared" - PropagationSlave PropagationMode = "Slave" - PropagationPrivate PropagationMode = "Private" -) - -type HostPathVolumeSource struct { - Path string `json:"path"` - // Mount the host path with propagation mode specified. Docker only. - Propagation PropagationMode `json:"propagation,omitempty"` -} -``` - -Opinion against this: - -1. This need API change, which is discouraged. - -1. All containers use this volume will share the same propagation mode. - -1. (From @jonboulle) May cause cross-runtime compatibility issue. - -### Make HostPath shared for privileged containers, slave for non-privileged. - -Given only HostPath needs this feature, and CAP_SYS_ADMIN access is needed when -making mounts inside container, we can bind propagation mode with existing option -privileged, or we can introduce a new option in SecurityContext to control this. - -The propagation mode could be determined by the following logic: - -```go -// Environment check to ensure "rshared" is supported. -if !dockerNewerThanV110 || !mountPathIsShared { - return "" -} -if container.SecurityContext.Privileged { - return "rshared" -} else { - return "rslave" -} -``` - -Opinion against this: - -1. This changes the behavior of existing config. - -1. (From @euank) "shared" is not correctly supported by some kernels, we need -runtime support matrix and when that will be addressed. - -1. This may cause silently fail and be a debuggability nightmare on many -distros. - -1. (From @euank) Changing those mountflags may make docker even less stable, -this may lock up kernel accidentally or potentially leak mounts. - - -## Decision - -We will take 'Make HostPath shared for privileged containers, slave for -non-privileged', an environment check and an WARNING log will be emitted about -whether propagation mode is supported. - - -## Extra Concerns - -@lucab and @euank has some extra concerns about pod isolation when propagation -modes are changed, listed below: - -1. how to clean such pod resources (as mounts are now crossing pod boundaries, -thus they can be kept busy indefinitely by processes outside of the pod) - -1. side-effects on restarts (possibly piling up layers of full-propagation mounts) - -1. how does this interacts with other mount features (nested volumeMounts may or -may not propagate back to the host, depending of ordering of mount operations) - -1. limitations this imposes on runtimes (RO-remounting may now affects the host, -is it on purpose or a dangerous side-effect?) - -1. A shared mount target imposes some costraints on its parent subtree (generally, -it has to be shared as well), which in turn prevents some mount operations when -preparing a pod (eg. MS_MOVE). - -1. The "on-by-default" nature means existing hostpath mounts, which used to be -harmless, could begin consuming kernel resources and cause a node to crash. Even -if a pod does not create any new mountpoints under its hostpath bindmount, it's -not hard to reach multiplicative explosions with shared bindmounts and so the -change in default + no cleanup could result in existing workloads knocking the -node over. - -These concerns are valid and we decide to limit the propagation mode to HostPath -volume only, in HostPath, we expect any runtime should NOT perform any additional -actions (such as clean up). This behavior is also consistent with current HostPath -logic: kube does not take care of the content in HostPath either. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/release/OWNERS b/contributors/design-proposals/release/OWNERS new file mode 100644 index 00000000..9d8e7403 --- /dev/null +++ b/contributors/design-proposals/release/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-release-leads +approvers: + - sig-release-leads +labels: + - sig/release diff --git a/contributors/design-proposals/release-notes.md b/contributors/design-proposals/release/release-notes.md index f602eead..42f5ff24 100644 --- a/contributors/design-proposals/release-notes.md +++ b/contributors/design-proposals/release/release-notes.md @@ -4,7 +4,6 @@ [djmm@google.com](mailto:djmm@google.com)<BR> Last Updated: 2016-04-06 -<!-- BEGIN MUNGE: GENERATED_TOC --> - [Kubernetes Release Notes](#kubernetes-release-notes) - [Objective](#objective) @@ -22,7 +21,6 @@ Last Updated: 2016-04-06 - [Work estimates](#work-estimates) - [Caveats / Considerations](#caveats--considerations) -<!-- END MUNGE: GENERATED_TOC --> ## Objective @@ -188,7 +186,3 @@ Other notable changes * For now contributors should simply use the first PR that enables a new feature by default. We'll revisit if this does not work well. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/release-test-signal.md b/contributors/design-proposals/release/release-test-signal.md index 6975629d..6975629d 100644 --- a/contributors/design-proposals/release-test-signal.md +++ b/contributors/design-proposals/release/release-test-signal.md diff --git a/contributors/design-proposals/versioning.md b/contributors/design-proposals/release/versioning.md index 5b8dd66d..05e7faed 100644 --- a/contributors/design-proposals/versioning.md +++ b/contributors/design-proposals/release/versioning.md @@ -126,7 +126,3 @@ to go from 1.x to 1.x+y before they go to 2.x. There is a separate question of how to track the capabilities of a kubelet to facilitate rolling upgrades. That is not addressed here. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/resource-management/OWNERS b/contributors/design-proposals/resource-management/OWNERS new file mode 100644 index 00000000..60221854 --- /dev/null +++ b/contributors/design-proposals/resource-management/OWNERS @@ -0,0 +1,4 @@ +reviewers: + - wg-resource-management-leads +approvers: + - wg-resource-management-leads diff --git a/contributors/design-proposals/admission_control_limit_range.md b/contributors/design-proposals/resource-management/admission_control_limit_range.md index 3684b636..7dd454c7 100644 --- a/contributors/design-proposals/admission_control_limit_range.md +++ b/contributors/design-proposals/resource-management/admission_control_limit_range.md @@ -228,6 +228,3 @@ the following would happen. 3. If the container is later resized, it's cpu would be constrained to between .1 and 1 and the ratio of limit to request could not exceed 4. -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/admission_control_resource_quota.md b/contributors/design-proposals/resource-management/admission_control_resource_quota.md index 575db9a8..02c3afb6 100644 --- a/contributors/design-proposals/admission_control_resource_quota.md +++ b/contributors/design-proposals/resource-management/admission_control_resource_quota.md @@ -206,10 +206,25 @@ secrets 1 10 services 0 5 ``` +Simple object count quotas supported on all standard resources using`count/<resource>.<group>` syntax. + +For example: +```console +$ kubectl create quota test --hard=count/deployments.extensions=2,count/replicasets.extensions=4,count/pods=3,count/secrets=4 +resourcequota "test" created +$ kubectl run nginx --image=nginx --replicas=2 +$ kubectl describe quota +Name: test +Namespace: default +Resource Used Hard +-------- ---- ---- +count/deployments.extensions 1 2 +count/pods 2 3 +count/replicasets.extensions 1 4 +count/secrets 1 4 +``` + ## More information -See [resource quota document](../admin/resource-quota.md) and the [example of Resource Quota](../admin/resourcequota/) for more information. +See [resource quota document](https://kubernetes.io/docs/concepts/policy/resource-quotas/) and the [example of Resource Quota](https://kubernetes.io/docs/tasks/administer-cluster/quota-api-object/) for more information. -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/device-plugin-overview.png b/contributors/design-proposals/resource-management/device-plugin-overview.png Binary files differindex 4d3f90f8..4d3f90f8 100644 --- a/contributors/design-proposals/device-plugin-overview.png +++ b/contributors/design-proposals/resource-management/device-plugin-overview.png diff --git a/contributors/design-proposals/device-plugin.md b/contributors/design-proposals/resource-management/device-plugin.md index 9871ec35..6c203369 100644 --- a/contributors/design-proposals/device-plugin.md +++ b/contributors/design-proposals/resource-management/device-plugin.md @@ -41,7 +41,7 @@ consume hardware devices across k8s clusters. This document describes a vendor independant solution to: * Discovering and representing external devices * Making these devices available to the containers, using these devices, - scrubbing and securly sharing these devices. + scrubbing and securely sharing these devices. * Health Check of these devices Because devices are vendor dependant and have their own sets of problems @@ -198,7 +198,7 @@ When starting the gRPC server, they are expected to create a unix socket at the following host path: `/var/lib/kubelet/device-plugins/`. For non bare metal device plugin this means they will have to mount the folder -as a volume in their pod spec ([see Installation](##installation)). +as a volume in their pod spec ([see Installation](#installation)). Device plugins can expect to find the socket to register themselves on the host at the following path: @@ -486,10 +486,10 @@ Currently we require exact version match between Kubelet and Device Plugin. API version is expected to be increased only upon incompatible API changes. Follow protobuf guidelines on versionning: - * Do not change ordering - * Do not remove fields or change types - * Add optional fields - * Introducing new fields with proper default values + * Do not change ordering + * Do not remove fields or change types + * Add optional fields + * Introducing new fields with proper default values * Freeze the package name to `apis/device-plugin/v1alpha1` * Have kubelet and the Device Plugin negotiate versions if we do break the API diff --git a/contributors/design-proposals/device-plugin.png b/contributors/design-proposals/resource-management/device-plugin.png Binary files differindex 61ae6167..61ae6167 100644 --- a/contributors/design-proposals/device-plugin.png +++ b/contributors/design-proposals/resource-management/device-plugin.png diff --git a/contributors/design-proposals/gpu-support.md b/contributors/design-proposals/resource-management/gpu-support.md index 19aa1dac..ad4383ac 100644 --- a/contributors/design-proposals/gpu-support.md +++ b/contributors/design-proposals/resource-management/gpu-support.md @@ -1,5 +1,3 @@ -<!-- BEGIN MUNGE: GENERATED_TOC --> - - [GPU support](#gpu-support) - [Objective](#objective) - [Background](#background) @@ -22,7 +20,6 @@ - [Undetermined](#undetermined) - [Security considerations](#security-considerations) -<!-- END MUNGE: GENERATED_TOC --> # GPU support @@ -274,6 +271,3 @@ or roles to schedule GPU workloads. Overcommitting or sharing the same device across different pods is not considered safe. It should be possible to segregate such GPU-sharing pods by user, namespace or a combination thereof. -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/hugepages.md b/contributors/design-proposals/resource-management/hugepages.md index 27e5c5af..9901a4bb 100644 --- a/contributors/design-proposals/hugepages.md +++ b/contributors/design-proposals/resource-management/hugepages.md @@ -34,7 +34,7 @@ page size. A huge page is a memory page that is larger than 4Ki. On x86_64 architectures, there are two common huge page sizes: 2Mi and 1Gi. Sizes vary on other architectures, but the idea is the same. In order to use huge pages, -application must write code that is aware of them. Transparent huge pages (THP) +application must write code that is aware of them. Transparent Huge Pages (THP) attempts to automate the management of huge pages without application knowledge, but they have limitations. In particular, they are limited to 2Mi page sizes. THP might lead to performance degradation on nodes with high memory utilization @@ -88,14 +88,14 @@ The proposal introduces huge pages as an Alpha feature. It must be enabled via the `--feature-gates=HugePages=true` flag on pertinent components pending graduation to Beta. -## Node Specfication +## Node Specification Huge pages cannot be overcommitted on a node. A system may support multiple huge page sizes. It is assumed that most nodes will be configured to primarily use the default huge page size as returned via `grep Hugepagesize /proc/meminfo`. This defaults to 2Mi on most Linux systems -unless overriden by `default_hugepagesz=1g` in kernel boot parameters. +unless overridden by `default_hugepagesz=1g` in kernel boot parameters. For each supported huge page size, the node will advertise a resource of the form `hugepages-<hugepagesize>`. On Linux, supported huge page sizes are diff --git a/contributors/design-proposals/resource-quota-scoping.md b/contributors/design-proposals/resource-management/resource-quota-scoping.md index 5cffb998..edc6ed32 100644 --- a/contributors/design-proposals/resource-quota-scoping.md +++ b/contributors/design-proposals/resource-management/resource-quota-scoping.md @@ -326,8 +326,3 @@ Appropriate unit and e2e testing will be authored. Existing resource quota documentation and examples will be updated. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/rules-review-api.md b/contributors/design-proposals/rules-review-api.md new file mode 100644 index 00000000..1bc3d9d6 --- /dev/null +++ b/contributors/design-proposals/rules-review-api.md @@ -0,0 +1,128 @@ +# "What can I do?" API + +Author: Eric Chiang (eric.chiang@coreos.com) + +## Overview + +Currently, to determine if a user is authorized to perform a set of actions, that user has to query each action individually through a `SelfSubjectAccessReview`. + +Beyond making the authorization layer hard to reason about, it means web interfaces such as the OpenShift Web Console, Tectonic Console, and Kubernetes Dashboard, have to perform individual calls for _every resource_ a page displays. There's no way for a user, or an application acting on behalf of a user, to ask for all the permissions a user can make in bulk. This makes its hard to build pages that are proactive about what's displayed or grayed out based on the end user's permissions. UIs can only handle 403 responses after a user has already performed a forbidden action. + +This is a proposal to add authorization APIs that allow a client to determine what actions they can make within a namespace. We expect this API to be used by UIs to show/hide actions, or to quickly let an end user reason about their permissions. This API should NOT be used by external systems to drive their own authorization decisions, as this raises confused deputy, cache lifetime/revocation, and correctness concerns. The `*AccessReview` APIs remain the correct way to defer authorization decisions to the API server. + +OpenShift adopted a [`RulesReview` API][openshift-rules-review] to accomplish this same goal, and this proposal is largely a port of that implementation. + +[kubernetes/kubernetes#48051](https://github.com/kubernetes/kubernetes/pull/48051) implements most of this proposal. + +## API additions + +Add a top level type to the `authorization.k8s.io` API group called `SelfSubjectRulesReview`. This mirrors the existing `SelfSubjectAccessReview`. + +``` +type SelfSubjectRulesReview struct { + metav1.TypeMeta + + Spec SelfSubjectRulesReviewSpec + + // Status is filled in by the server and represents the set of actions a user can perform. + Status SubjectRulesReviewStatus +} + +type SelfSubjectRulesReviewSpec struct { + // Namespace to evaluate rules for. Required. + Namespace string +} + +type SubjectRulesReviewStatus struct { + // ResourceRules is the list of actions the subject is allowed to perform on resources. + // The list ordering isn't significant, may contain duplicates, and possibly be incomplete. + ResourceRules []ResourceRule + // NonResourceRules is the list of actions the subject is allowed to perform on non-resources. + // The list ordering isn't significant, may contain duplicates, and possibly be incomplete. + NonResourceRules []NonResourceRule + // EvaluationError can appear in combination with Rules. It indicates an error occurred during + // rule evaluation, such as an authorizer that doesn't support rule evaluation, and that + // ResourceRules and/or NonResourceRules may be incomplete. + EvaluationError string + // Incomplete indicates that the returned list is known to be incomplete. + Incomplete bool +} +``` + +The `ResourceRules` and `NonResourceRules` rules are similar to the types use by RBAC and the internal authorization system. + +``` +# docstrings omitted for brevity. +type ResourceRule struct { + Verbs []string + APIGroups []string + Resources []string + ResourceNames []string +} + +type NonResourceRule struct { + Verbs []string + NonResourceURLs []string +} +``` + +All of these fields can include the string `*` to indicate all values are allowed. + +### Differences from OpenShift: user extras vs. scopes + +OpenShift `SelfSubjectRulesReviewSpec` takes a set of [`Scopes`][openshift-scopes]. This lets OpenShift clients use the API for queries such as _"what could I do if I provide this scope to limit my credentials?"_ + + In core kube, scopes are replaced by "user extras" field, a map of opaque strings that can be used for implementation specific user data. Unlike OpenShift, where scopes are always used to restrict credential powers, user extras are commonly used to expand powers. For example, the proposed [Keystone authentiator][keystone-authn] used them to include additional roles and project fields. + +Since user extras can be used to expand the power of users, instead of only restricting, this proposal argues that `SelfSubjectRulesReview` shouldn't let a client specify them like `Scopes`. It wouldn't be within the spirit of a `SelfSubject` resource to let a user determine information about other projects or roles. + +This could hopefully be solved by introducing a `SubjectRulesReview` API to query the rules for any user. An aggregated API server could use the `SubjectRulesReview` to back an API resource that let a user provide restrictive user extras, such as scopes. + +## Webhook authorizers + +Some authorizers live external to Kubernetes through an API server webhook and wouldn't immediately support a rules review query. + +To communicate with external authorizers, the following types will be defined to query the rules for an arbitrary user. This proposal does NOT propose adding these types to the API immediately, since clients can use user impersonation and a `SelfSubjectRulesReview` to accomplish something similar. + +``` +type SubjectRulesReview struct { + metav1.TypeMeta + + Spec SubjectRulesReviewSpec + + // Status is filled in by the server and indicates the set of actions a user can perform. + Status SubjectRulesReviewStatus +} + +type SubjectRulesReviewSpec struct { + // Namespace to evalue rules for. Required. + Namespace string + + // User to be evaluated for. + UID string + User string + Groups []string + Extras map[string][]string +} +``` + +Currently, external authorizers are configured through the following API server flag and which POSTs a `SubjectAccessReview` to determine a user's access: + +``` +--authorization-webhook-config-file +``` + +The config file uses the kubeconfig format. + +There are a few options to support a second kind of query. + +* Add another webhook flag with a second config file. +* Introduce a [kubeconfig extension][kubeconfig-extension] that indicates the server can handle either a `SubjectRulesReview` or a `SubjectAccessReview` +* Introduce a second context in the kubeconfig for the `SubjectRulesReview`. Have some way of indicating which context for `SubjectRulesReview` and which is for `SubjectAccessReview`, for example by well-known context names for each. + +The doc proposed adding a second webhook config for `RulesReview`, and not overloading the existing config passed to `--authorization-webhook-config-file`. + +[openshift-rules-review]: https://github.com/openshift/origin/blob/v3.6.0/pkg/authorization/apis/authorization/types.go#L152 +[openshift-scopes]: https://github.com/openshift/origin/blob/v3.6.0/pkg/authorization/apis/authorization/types.go#L164-L168 +[keystone-authn]: https://github.com/kubernetes/kubernetes/pull/25624/files#diff-897f0cab87e784d9fc6813f04f128f62R40 +[kubeconfig-extension]: https://github.com/kubernetes/client-go/blob/v3.0.0/tools/clientcmd/api/v1/types.go#L51 diff --git a/contributors/design-proposals/Kubemark_architecture.png b/contributors/design-proposals/scalability/Kubemark_architecture.png Binary files differindex 479ad8b1..479ad8b1 100644 --- a/contributors/design-proposals/Kubemark_architecture.png +++ b/contributors/design-proposals/scalability/Kubemark_architecture.png diff --git a/contributors/design-proposals/scalability/OWNERS b/contributors/design-proposals/scalability/OWNERS new file mode 100644 index 00000000..2b68b875 --- /dev/null +++ b/contributors/design-proposals/scalability/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-scalability-leads +approvers: + - sig-scalability-leads +labels: + - sig/scalability diff --git a/contributors/design-proposals/kubemark.md b/contributors/design-proposals/scalability/kubemark.md index 95888c94..533295e0 100644 --- a/contributors/design-proposals/kubemark.md +++ b/contributors/design-proposals/scalability/kubemark.md @@ -45,8 +45,8 @@ components on non-default ports, and in the same time will allow to run multiple generate credentials for cluster communication and pass them to HollowKubelet/HollowProxy to use. Master will treat all HollowNodes as normal ones. - -*Kubmark architecture diagram for option 1* + +*Kubemark architecture diagram for option 1* ### Option 2 @@ -151,7 +151,3 @@ In the future we want to add following capabilities to the Kubemark system: - replaying real traffic reconstructed from the recorded Events stream, - simulating scraping things running on Nodes through Master proxy. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/scalability-testing.md b/contributors/design-proposals/scalability/scalability-testing.md index d0fcd1be..0195d7b1 100644 --- a/contributors/design-proposals/scalability-testing.md +++ b/contributors/design-proposals/scalability/scalability-testing.md @@ -19,7 +19,7 @@ Once we have a better understanding of their consequences, we may want to decide to drop one of them, but we are not yet in that position. -## Proposal 1 - Kubmark +## Proposal 1 - Kubemark In this proposal we are focusing on scalability testing of master components. We do NOT focus on node-scalability - this issue should be handled separately. @@ -66,7 +66,3 @@ as not to collide. Complications may currently exist around container tracking and ownership in docker. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/scheduling/OWNERS b/contributors/design-proposals/scheduling/OWNERS new file mode 100644 index 00000000..b3248766 --- /dev/null +++ b/contributors/design-proposals/scheduling/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-scheduling-leads +approvers: + - sig-scheduling-leads +labels: + - sig/scheduling diff --git a/contributors/design-proposals/images/.gitignore b/contributors/design-proposals/scheduling/images/.gitignore index e69de29b..e69de29b 100644 --- a/contributors/design-proposals/images/.gitignore +++ b/contributors/design-proposals/scheduling/images/.gitignore diff --git a/contributors/design-proposals/images/OWNERS b/contributors/design-proposals/scheduling/images/OWNERS index fe173c27..fe173c27 100644 --- a/contributors/design-proposals/images/OWNERS +++ b/contributors/design-proposals/scheduling/images/OWNERS diff --git a/contributors/design-proposals/images/preemption_1.png b/contributors/design-proposals/scheduling/images/preemption_1.png Binary files differindex 6d1660b6..6d1660b6 100644 --- a/contributors/design-proposals/images/preemption_1.png +++ b/contributors/design-proposals/scheduling/images/preemption_1.png diff --git a/contributors/design-proposals/images/preemption_2.png b/contributors/design-proposals/scheduling/images/preemption_2.png Binary files differindex 38fc9088..38fc9088 100644 --- a/contributors/design-proposals/images/preemption_2.png +++ b/contributors/design-proposals/scheduling/images/preemption_2.png diff --git a/contributors/design-proposals/images/preemption_3.png b/contributors/design-proposals/scheduling/images/preemption_3.png Binary files differindex 0f750edb..0f750edb 100644 --- a/contributors/design-proposals/images/preemption_3.png +++ b/contributors/design-proposals/scheduling/images/preemption_3.png diff --git a/contributors/design-proposals/images/preemption_4.png b/contributors/design-proposals/scheduling/images/preemption_4.png Binary files differindex f8343f3b..f8343f3b 100644 --- a/contributors/design-proposals/images/preemption_4.png +++ b/contributors/design-proposals/scheduling/images/preemption_4.png diff --git a/contributors/design-proposals/multiple-schedulers.md b/contributors/design-proposals/scheduling/multiple-schedulers.md index 1ea1b45c..1581a0d9 100644 --- a/contributors/design-proposals/multiple-schedulers.md +++ b/contributors/design-proposals/scheduling/multiple-schedulers.md @@ -133,6 +133,3 @@ randomly among the top N nodes instead of the one with the highest score. - [#16845](https://github.com/kubernetes/kubernetes/issues/16845): scheduling groups of pods - [#17208](https://github.com/kubernetes/kubernetes/issues/17208): guide to writing a new scheduler -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/nodeaffinity.md b/contributors/design-proposals/scheduling/nodeaffinity.md index 61e04169..ae167ce5 100644 --- a/contributors/design-proposals/nodeaffinity.md +++ b/contributors/design-proposals/scheduling/nodeaffinity.md @@ -144,7 +144,7 @@ Hopefully this won't cause too much confusion. ## Examples -** TODO: fill in this section ** +**TODO: fill in this section** * Run this pod on a node with an Intel or AMD CPU @@ -240,7 +240,3 @@ The main related issue is [#341](https://github.com/kubernetes/kubernetes/issues Issue [#367](https://github.com/kubernetes/kubernetes/issues/367) is also related. Those issues reference other related issues. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/pod-preemption.md b/contributors/design-proposals/scheduling/pod-preemption.md index 4fba2bf1..46843fce 100644 --- a/contributors/design-proposals/pod-preemption.md +++ b/contributors/design-proposals/scheduling/pod-preemption.md @@ -75,7 +75,7 @@ When scheduling a pending pod, scheduler tries to place the pod on a node that d #### Important notes -- When ordering the pods from lowest to highest priority for considering which pod(s) to preempt, among pods with equal priority the pods are ordered by their [QoS class](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-qos.md#qos-classes): Best Effort, Burstable, Guaranteed. +- When ordering the pods from lowest to highest priority for considering which pod(s) to preempt, among pods with equal priority the pods are ordered by their [QoS class](/contributors/design-proposals/node/resource-qos.md#qos-classes): Best Effort, Burstable, Guaranteed. - Scheduler respects pods' disruption budget when considering them for preemption. - Scheduler will try to minimize the number of preempted pods. As a result, it may preempt a pod while leaving lower priority pods running if preemption of those lower priority pods is not enough to schedule the pending pod while preemption of the higher priority pod(s) is enough to schedule the pending pod. For example, if node capacity is 10, and pending pod is priority 10 and requires 5 units of resource, and the running pods are {priority 0 request 3, priority 1 request 1, priority 2 request 5, priority 3 request 1}, scheduler will preempt the priority 2 pod only and leaves priority 1 and priority 0 running. - Scheduler does not have the knowledge of resource usage of pods. It makes scheduling decisions based on the requested resources ("requests") of the pods and when it considers a pod for preemption, it assumes the "requests" to be freed on the node. @@ -183,6 +183,6 @@ To solve the problem, the user might try running his web server as Guaranteed, b # References -- [Controlled Rescheduling in Kubernetes](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/rescheduling.md) +- [Controlled Rescheduling in Kubernetes](/contributors/design-proposals/scheduling/rescheduling.md) - [Resource sharing architecture for batch and serving workloads in Kubernetes](https://docs.google.com/document/d/1-H2hnZap7gQivcSU-9j4ZrJ8wE_WwcfOkTeAGjzUyLA) - [Design proposal for adding priority to Kubernetes API](https://github.com/kubernetes/community/pull/604/files)
\ No newline at end of file diff --git a/contributors/design-proposals/pod-priority-api.md b/contributors/design-proposals/scheduling/pod-priority-api.md index 914d229c..8b5d7219 100644 --- a/contributors/design-proposals/pod-priority-api.md +++ b/contributors/design-proposals/scheduling/pod-priority-api.md @@ -233,7 +233,7 @@ absolutely needed. Changing priority classes has the following disadvantages: ### Priority and QoS classes Kubernetes has [three QoS -classes](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-qos.md#qos-classes) +classes](/contributors/design-proposals/node/resource-qos.md#qos-classes) which are derived from request and limit of pods. Priority is introduced as an independent concept; meaning that any QoS class may have any valid priority. When a node is out of resources and pods needs to be preempted, we give diff --git a/contributors/design-proposals/podaffinity.md b/contributors/design-proposals/scheduling/podaffinity.md index 9291b8b9..89752150 100644 --- a/contributors/design-proposals/podaffinity.md +++ b/contributors/design-proposals/scheduling/podaffinity.md @@ -313,7 +313,7 @@ scheduler to not put more than one pod from S in the same zone, and thus by definition it will not put more than one pod from S on the same node, assuming each node is in one zone. This rule is more useful as PreferredDuringScheduling anti-affinity, e.g. one might expect it to be common in -[Cluster Federation](../../docs/proposals/federation.md) clusters.) +[Cluster Federation](/contributors/design-proposals/multicluster/federation.md) clusters.) * **Don't co-locate pods of this service with pods from service "evilService"**: `{LabelSelector: selector that matches evilService's pods, TopologyKey: "node"}` @@ -345,8 +345,7 @@ as shorthand for "RequiredDuringSchedulingScheduling pod affinity" and "SoftPodAffinity" as shorthand for "PreferredDuringScheduling pod affinity." Analogously for "HardPodAntiAffinity" and "SoftPodAntiAffinity." -** TODO: Update this algorithm to take weight for SoftPod{Affinity,AntiAffinity} -into account; currently it assumes all terms have weight 1. ** +**TODO: Update this algorithm to take weight for SoftPod{Affinity,AntiAffinity} into account; currently it assumes all terms have weight 1.** ``` Z = the pod you are scheduling @@ -664,10 +663,5 @@ This proposal is to satisfy [#14816](https://github.com/kubernetes/kubernetes/is ## Related work -** TODO: cite references ** +**TODO: cite references** - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/scheduling/predicates-ordering.md b/contributors/design-proposals/scheduling/predicates-ordering.md new file mode 100644 index 00000000..a02f70a6 --- /dev/null +++ b/contributors/design-proposals/scheduling/predicates-ordering.md @@ -0,0 +1,93 @@ +# predicates ordering + + + +Status: proposal + +Author: yastij +Approvers: +* gmarek +* bsalamat +* k82cn + + + + +## Abstract + +This document describes how and why reordering predicates helps to achieve performance for the kubernetes scheduler. +We will expose the motivations behind this proposal, The two steps/solution we see to tackle this problem and the timeline decided to implement these. + + +## Motivation + +While working on a [Pull request](https://github.com/kubernetes/kubernetes/pull/50185) related to a proposal, we saw that the order of running predicates isn’t defined. + +This makes the scheduler perform extra-computation that isn’t needed, As an example we [outlined](https://github.com/kubernetes/kubernetes/pull/50185) that the kubernetes scheduler runs predicates against nodes even if marked “unschedulable”. + +Reordering predicates allows us to avoid this problem, by computing the most restrictive predicates first. To do so, we propose two reordering types. + + + +## Static ordering + +This ordering will be the default ordering. If a policy config is provided with a subset of predicates, only those predicates will be invoked using the static ordering. + + + + +|Position | Predicate | comments (note, justification...) | + ----------------- | ---------------------------- | ------------------ +| 1 | `CheckNodeConditionPredicate` | we really don’t want to check predicates against unschedulable nodes. | +| 2 | `PodFitsHost` | we check the pod.spec.nodeName. | +| 3 | `PodFitsHostPorts` | we check ports asked on the spec. | +| 4 | `PodMatchNodeSelector` | check node label after narrowing search. | +| 5 | `PodFitsResources ` | this one comes here since it’s not restrictive enough as we do not try to match values but ranges. | +| 6 | `NoDiskConflict` | Following the resource predicate, we check disk | +| 7 | `PodToleratesNodeTaints '` | check toleration here, as node might have toleration | +| 8 | `PodToleratesNodeNoExecuteTaints` | check toleration here, as node might have toleration | +| 9 | `CheckNodeLabelPresence ` | labels are easy to check, so this one goes before | +| 10 | `checkServiceAffinity ` | - | +| 11 | `MaxPDVolumeCountPredicate ` | - | +| 12 | `VolumeNodePredicate ` | - | +| 13 | `VolumeZonePredicate ` | - | +| 14 | `CheckNodeMemoryPressurePredicate` | doesn’t happen often | +| 15 | `CheckNodeDiskPressurePredicate` | doesn’t happen often | +| 16 | `InterPodAffinityMatches` | Most expensive predicate to compute | + + +## End-user ordering + +Using scheduling policy file, the cluster admin can override the default static ordering. This gives administrator the maximum flexibility regarding scheduler behaviour and enables scheduler to adapt to cluster usage. +Please note that the order must be a positive integer, also, when providing equal ordering for many predicates, scheduler will determine the order and won't guarantee that the order will remain the same between them. +Finally updating the scheduling policy file will require a scheduler restart. + +as an example the following is scheduler policy file using an end-user ordering: + +``` json +{ +"kind" : "Policy", +"apiVersion" : "v1", +"predicates" : [ + {"name" : "PodFitsHostPorts", "order": 2}, + {"name" : "PodFitsResources", "order": 3}, + {"name" : "NoDiskConflict", "order": 5}, + {"name" : "PodToleratesNodeTaints", "order": 4}, + {"name" : "MatchNodeSelector", "order": 6}, + {"name" : "PodFitsHost", "order": 1} + ], +"priorities" : [ + {"name" : "LeastRequestedPriority", "weight" : 1}, + {"name" : "BalancedResourceAllocation", "weight" : 1}, + {"name" : "ServiceSpreadingPriority", "weight" : 1}, + {"name" : "EqualPriority", "weight" : 1} + ], +"hardPodAffinitySymmetricWeight" : 10 +} +``` + + +## Timeline + +* static ordering: GA in 1.9 +* dynamic ordering: TBD based on customer feedback diff --git a/contributors/design-proposals/rescheduler.md b/contributors/design-proposals/scheduling/rescheduler.md index faf53564..df36464a 100644 --- a/contributors/design-proposals/rescheduler.md +++ b/contributors/design-proposals/scheduling/rescheduler.md @@ -118,6 +118,3 @@ For scaling up the cluster, a reasonable workflow might be: 1. rescheduler triggers cluster auto-scaler to add a node of the appropriate type for the PENDING Pod 1. the PENDING Pod schedules onto the new node (and possibly the rescheduler also moves other Pods onto that node) -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/rescheduling-for-critical-pods.md b/contributors/design-proposals/scheduling/rescheduling-for-critical-pods.md index 525e0805..835088fc 100644 --- a/contributors/design-proposals/rescheduling-for-critical-pods.md +++ b/contributors/design-proposals/scheduling/rescheduling-for-critical-pods.md @@ -82,7 +82,3 @@ The rescheduler control loop will be as follow: * wait until scheduler will schedule the critical addon * if there is no more critical addons for which we can help, ensure there is no node with the taint - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/rescheduling.md b/contributors/design-proposals/scheduling/rescheduling.md index 1966de79..b812599c 100644 --- a/contributors/design-proposals/rescheduling.md +++ b/contributors/design-proposals/scheduling/rescheduling.md @@ -45,7 +45,7 @@ Example use cases for rescheduling are * moving a pod onto an under-utilized node * moving a pod onto a node that meets more of the pod's affinity/anti-affinity preferences * moving a running pod off of a node in anticipation of a known or speculated future event - * draining a node in preparation for maintenance, decomissioning, auto-scale-down, etc. + * draining a node in preparation for maintenance, decommissioning, auto-scale-down, etc. * "preempting" a running pod to make room for a pending pod to schedule * proactively/speculatively make room for large and/or exclusive pods to facilitate fast scheduling in the future (often called "defragmentation") @@ -145,12 +145,12 @@ it allows the API server to do validation (e.g. to catch mis-spelling). In the future, which priorities are usable for a given namespace and pods with certain attributes may be configurable, similar to ResourceQuota, LimitRange, or security policy. -Priority and resource QoS are indepedent. +Priority and resource QoS are independent. The priority we have described here might be used to prioritize the scheduling queue (i.e. the order in which a scheduler examines pods in its scheduling loop), but the two priority concepts do not have to be connected. It is somewhat logical to tie them -together, since a higher priority genreally indicates that a pod is more urgent to get +together, since a higher priority generally indicates that a pod is more urgent to get running. Also, scheduling low-priority pods before high-priority pods might lead to avoidable preemptions if the high-priority pods end up preempting the low-priority pods that were just scheduled. @@ -487,7 +487,3 @@ TODO. TODO: Add reference to this doc from docs/proposals/rescheduler.md - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/resources.md b/contributors/design-proposals/scheduling/resources.md index 6b01dbeb..356f57e7 100644 --- a/contributors/design-proposals/resources.md +++ b/contributors/design-proposals/scheduling/resources.md @@ -1,6 +1,6 @@ **Note: this is a design doc, which describes features that have not been completely implemented. User documentation of the current state is -[here](../user-guide/compute-resources.md). The tracking issue for +[here](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/). The tracking issue for implementation of this model is [#168](http://issue.k8s.io/168). Currently, both limits and requests of memory and cpu on containers (not pods) are supported. "memory" is in bytes and "cpu" is in milli-cores.** @@ -281,7 +281,7 @@ to encourage comments. Because resource usage and related metrics change continuously, need to be tracked over time (i.e., historically), can be characterized in a variety of ways, and are fairly voluminous, we will not include usage in core API objects, -such as [Pods](../user-guide/pods.md) and Nodes, but will provide separate APIs +such as [Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod/) and Nodes, but will provide separate APIs for accessing and managing that data. See the Appendix for possible representations of usage data, but the representation we'll use is TBD. @@ -364,7 +364,3 @@ second. * Units: operations per second * Compressible? yes - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/scheduling/scheduler-equivalence-class.md b/contributors/design-proposals/scheduling/scheduler-equivalence-class.md new file mode 100644 index 00000000..d47fabca --- /dev/null +++ b/contributors/design-proposals/scheduling/scheduler-equivalence-class.md @@ -0,0 +1,326 @@ +# Equivalence class based scheduling in Kubernetes + +**Authors**: + +@resouer @wojtek-t @davidopp + + +# Guideline + +- [Objectives](#objectives) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Background](#background) + - [Terminology](#terminology) +- [Overview](#overview) +- [Detailed Design](#detailed-design) + - [Define equivalence class](#define-equivalence-class) + - [Equivalence class in predicate phase](#equivalence-class-in-predicate) + - [Keep equivalence class cache up-to-date](#keep-equivalence--class-cache-up-to-date) +- [Notes for scheduler developers](#notes-for-scheduler-developer) +- [References](#references) + +# Objectives + +## Goals + +- Define the equivalence class for pods during predicate phase in Kubernetes. +- Define how to use equivalence class to speed up predicate process. +- Define how to ensure information cached in equivalence class is up-to-date. + +## Non-Goals + +- Apply equivalence class to priorities. We have refactored priorities to a Map-Reduce style process, we need to re-evaluate whether equivalence design can or can not apply to this new model. + +# Background + +Pods in Kubernetes cluster usually have identical requirements and constraints, just think about a Deployment with a number of replications. So rather than determining feasibility for every pending pod on every node, we can only do predicates one pod per equivalence class – a group of tasks with identical requirements, and reuse the predicate results for other equivalent pods. + +We hope to use this mechanism to help to improve scheduler's scalability, especially in cases like Replication Controller with huge number of instances, or eliminate pressure caused by complex predicate functions. + +The concept of equivalence class in scheduling is a proven feature used originally in [Google Borg] [1]. + +## Terminology + +Equivalence class: a group of pods which has identical requirements and constraints. + +Equivalence class based scheduling: the scheduler will do predicate for only one pod per equivalence class, and reuse this result for all other equivalent pods. + +# Overview + +This document describes what is equivalence class, and how to do equivalence based scheduling in Kubernetes. The basic idea is when you apply the predicate functions to a pod, cache the results (namely, for each machine, whether the pod is feasible on that machine). + +Scheduler watches for API objects change like bindings and unbindings and node changes, and marks a cached value as invalid whenever there is a change that invalidates a cached value. (For example, if the labels on a node change, or a new pod gets bound to a machine, then all cached values related to that machine are invalidated.) In the future when we have in-place updates, some updates to pods running on the machine would also cause the node to be marked invalid. This is how we keep equivalence class cache up-to-date. + +When scheduling a new pod, check to see if the predicate result for an equivalent pod is already cached. If so, re-evaluate the predicate functions just for the "invalid" values (i.e. not for all nodes and predicates), and update the cache. + + +# Detailed Design + +## 1. Define equivalence class + +There are two options were proposed. + +Option 1: use the attributes of Pod API object to decide if given pods are equivalent, the attributes include labels, some annotations, affinity, resource limit etc. + +Option 2: use controller reference, i.e. simply consider pods belonging to same controller reference +to be equivalent. + +Regarding first option - The biggest concern in this approach is that if someone will add dependency on some new field at some point, we don't have good way to test it and ensure that equivalence pod will be updated at that point too. + +Regarding second option - In detail, using the "ControllerRef" which is defined as "OwnerReference (from ObjectMeta) with the "Controller" field set to true as the "equivalence class". In this approach, we would have all RC, RS, Job etc handled by exactly the same mechanism. Also, this would be faster to compute it. + +For example, two pods created by the same `ReplicaSets` will be considered as equivalent since they will have exactly the same resource requirements from one pod template. On the other hand, two pods created by two `ReplicaSets` will not be considered as equivalent regardless of whether they have same resource requirements or not. + +**Conclusion:** + +Choose option 2. And we will calculate a unique `uint64` hash for pods belonging to same equivalence class which known as `equivalenceHash`. + +## 2. Equivalence class in predicate phase + +Predicate is the first phase in scheduler to filter out nodes which are feasible to run the workload. In detail: + +1. Predicates functions are registered in scheduler +2. The predicates will be checked by `scheduler.findNodesThatFit(pod, nodes, predicateFuncs ...)`. +3. The check process `scheduler.podFitsOnNode(pod, node, predicateFuncs ...)` is executed in parallel for every node. + +### 2.1 Design an equivalence class cache + +The step 3 is where registered predicate functions will be called against given pod and node. This step includes: + +1. Check if given pod has equivalence class. +2. If yes, use equivalence class cache to do predicate. + +In detail, we need to have an equivalence class cache to store all predicates results per node. The data structure is a 3 level map with keys of the levels being: `nodeName`, `predicateKey` and `equivalenceHash`. + +```go +predicateMap := algorithmCache[nodeName].predicatesCache.Get(predicateKey) +hostPredicate := predicateMap[equivalenceHash] +``` +For example: the cached `GeneralPredicates` result for equivalence class `1000392826` on node `node_1` is: + +```go +algorithmCache["node_1"].predicatesCache.Get("GeneralPredicates")[1000392826] +``` + +This will return a `HostPredicate` struct: + +```go +type HostPredicate struct { + Fit bool + FailReasons []algorithm.PredicateFailureReason +} + +``` + +Please note we use predicate name as key in `predicatesCache`, so the number of entries in the cache is less or equal to the total number of registered predicates in scheduler. The cache size is limited. + +### 2.2 Use cached predicate result to do predicate + +The pseudo code of predicate process with equivalence class will be like: + +```go +func (ec *EquivalenceCache) PredicateWithECache( + podName, nodeName, predicateKey string, + equivalenceHash uint64, +) (bool, []algorithm.PredicateFailureReason, bool) { + if algorithmCache, exist := ec.algorithmCache[nodeName]; exist { + if predicateMap, exist := algorithmCache.predicatesCache.Get(predicateKey); exist { + if hostPredicate, ok := predicateMap[equivalenceHash]; ok { + // fit + if hostPredicate.Fit { + return true, []algorithm.PredicateFailureReason{}, false + } else { + // unfit + return false, hostPredicate.FailReasons, false + } + } else { + // cached result is invalid + return false, []algorithm.PredicateFailureReason{}, true + } + } + } + return false, []algorithm.PredicateFailureReason{}, true +} +``` + +One thing to note is, if the `hostPredicate` is not present in the logic above, it will be considered as `invalid`. That means although this pod has equivalence class, it does not have cached predicate result yet, or the cached data is not valid. It needs to go through normal predicate process and write the result into equivalence clas cache. + +### 2.3 What if no equivalence class is found for pod? + +If no equivalence class is found for given pod, normal predicate process will be executed. + +## 3. Keep equivalence class cache up-to-date + +The key of this equivalence class based scheduling is how to keep the equivalence cache up-to-date. Since even one single pod been scheduled to a node will make the cached result not stand as the available resource on this node has changed. + +One approach is that we can invalidate the cached predicate result for this node. But in a heavy load cluster state change happens frequently and makes the design less meaningful. + +So in this design, we proposed the ability to invalidate cached result for specific predicate. For example, when a new pod is scheduled to a node, the cached result for `PodFitsResources` should be invalidated on this node while others can still be re-used. That's also another reason we use predicate name as key for the cached value. + +During the implementation, we need to consider all the cases which may affect the effectiveness of cached predicate result. The logic includes three dimensions: + +- **Operation**: + - what operation will cause this cache invalid. +- **Invalid predicates**: + - what predicate should be invalidated. +- **Scope**: + - the cache of which node should be invalidated, or all nodes. + +Please note with the change of predicates in subsequent development, this doc will become out-of-date, while you can always check the latest e-class cache update process in `plugin/pkg/scheduler/factory/factory.go`. + +### 3.1 Persistent Volume + +- **Operation:** + - ADD, DELETE + +- **Invalid predicates**: + + - `MaxEBSVolumeCount`, `MaxGCEPDVolumeCount`, `MaxAzureDiskVolumeCount` (only if the added/deleted PV is one of them) + +- **Scope**: + + - All nodes (we don't know which node this PV will be attached to) + + +### 3.2 Persistent Volume Claim + +- **Operation:** + - ADD, DELETE + +- **Invalid predicates:** + + - `MaxPDVolumeCountPredicate` (only if the added/deleted PVC as a binded volume so it drops to the PV change case, otherwise it should not affect scheduler). + +- **Scope:** + - All nodes (we don't know which node this PV will be attached to). + + +### 3.3 Service + +- **Operation:** + - ADD, DELETE + +- **Invalid predicates:** + + - `ServiceAffinity` + +- **Scope:** + - All nodes (`serviceAffinity` is a cluster scope predicate). + + + +- **Operation:** + - UPDATE + +- **Invalid predicates:** + + - `ServiceAffinity` (only if the `spec.Selector` filed is updated) + +- **Scope:** + - All nodes (`serviceAffinity` is a cluster scope predicate),. + + +### 3.4 Pod + +- **Operation:** + - ADD + +- **Invalid predicates:** + - `GeneralPredicates`. This invalidate should be done during `scheduler.assume(...)` because binding can be asynchronous. So we just optimistically invalidate predicate cached result there, and if later this pod failed to bind, the following pods will go through normal predicate functions and nothing breaks. + + - No `MatchInterPodAffinity`: the scheduler will make sure newly binded pod will not break the existing inter pod affinity. So we does not need to invalidate MatchInterPodAffinity when pod added. But when a pod is deleted, existing inter pod affinity may become invalid. (e.g. this pod was preferred by some else, or vice versa). + + - NOTE: assumptions above **will not** stand when we implemented features like `RequiredDuringSchedulingRequiredDuringExecution`. + + - No `NoDiskConflict`: the newly scheduled pod fits to existing pods on this node, it will also fits to equivalence class of existing pods. + +- **Scope:** + - The node which the pod was binded with. + + + +- **Operation:** + - UPDATE + +- **Invalid predicates:** + + - Only if `pod.NodeName` did not change (otherwise it drops to add/delete case) + + - `GeneralPredicates` if the pod's resource requests are updated. + + - `MatchInterPodAffinity` if the pod's labels are updated. + +- **Scope:** + - The node which the pod was binded with + + + +- **Operation:** + - DELETE + +- **Invalid predicates:** + - `MatchInterPodAffinity` if the pod's labels are updated. + +- **Scope:** + - All nodes in the same failure domain + +- **Invalid predicates:** + + - `NoDiskConflict` if the pod has special volume like `RBD`, `ISCSI`, `GCEPersistentDisk` etc. + +- **Scope:** + - The node which the pod was binded with. + + +### 3.5 Node + + +- **Operation:** + - UPDATE + +- **Invalid predicates:** + + - `GeneralPredicates`, if `node.Status.Allocatable` or node labels changed. + + - `ServiceAffinity`, if node labels changed, since selector result may change. + + - `MatchInterPodAffinity`, if value of label changed, since any node label can be topology key of pod. + + - `NoVolumeZoneConflict`, if zone related label change. + + - `PodToleratesNodeTaints`, if node taints changed. + + - `CheckNodeMemoryPressure`, `CheckNodeDiskPressure`, `CheckNodeCondition`, if related node condition changed. + +- **Scope:** + - The updated node. + +- **Operation:** + - DELETE + +- **Invalid predicates:** + - All predicates + +- **Scope:** + - The deleted node + + +# Notes for scheduler developers + +1. When implementing a new predicate, developers are expected to check how related API object changes (add/delete/update) affect the result of their new predicate function and invalidate cached results of the predicate function if necessary, in scheduler/factory/factory.go. + +2. When updating an existing predicate, developers should consider whether their changes introduce new dependency on attributes of any API objects like Pod, Node, Service, etc. If so, developer should consider invalidating caches results of this predicate in scheduler/factory/factory.go. + + +# References + +Main implementation PRs: + +- https://github.com/kubernetes/kubernetes/pull/31605 +- https://github.com/kubernetes/kubernetes/pull/34685 +- https://github.com/kubernetes/kubernetes/pull/36238 +- https://github.com/kubernetes/kubernetes/pull/41541 + + +[1]: http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43438.pdf "Google Borg paper"
\ No newline at end of file diff --git a/contributors/design-proposals/scheduler_extender.md b/contributors/design-proposals/scheduling/scheduler_extender.md index 1903535a..de7a6259 100644 --- a/contributors/design-proposals/scheduler_extender.md +++ b/contributors/design-proposals/scheduling/scheduler_extender.md @@ -2,7 +2,7 @@ There are three ways to add new scheduling rules (predicates and priority functions) to Kubernetes: (1) by adding these rules to the scheduler and -recompiling, [described here](https://github.com/kubernetes/community/blob/master/contributors/devel/scheduler.md), +recompiling, [described here](/contributors/devel/scheduler.md), (2) implementing your own scheduler process that runs instead of, or alongside of, the standard Kubernetes scheduler, (3) implementing a "scheduler extender" process that the standard Kubernetes scheduler calls out to as a final pass when @@ -120,6 +120,3 @@ type ExtenderBindingArgs struct { Node string } ``` -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/taint-node-by-condition.md b/contributors/design-proposals/scheduling/taint-node-by-condition.md index 35a5ee2d..550e9cd9 100644 --- a/contributors/design-proposals/taint-node-by-condition.md +++ b/contributors/design-proposals/scheduling/taint-node-by-condition.md @@ -18,20 +18,20 @@ In addition to this, with taint-based-eviction, the Node Controller already tain | ConditionType | Condition Status |Effect | Key | | ------------------ | ------------------ | ------------ | -------- | |Ready |True | - | | -| |False | NoExecute | node.kubernetes.io/notReady | +| |False | NoExecute | node.kubernetes.io/not-ready | | |Unknown | NoExecute | node.kubernetes.io/unreachable | -|OutOfDisk |True | NoSchedule | node.kubernetes.io/outOfDisk | +|OutOfDisk |True | NoSchedule | node.kubernetes.io/out-of-disk | | |False | - | | | |Unknown | - | | -|MemoryPressure |True | NoSchedule | node.kubernetes.io/memoryPressure | +|MemoryPressure |True | NoSchedule | node.kubernetes.io/memory-pressure | | |False | - | | | |Unknown | - | | -|DiskPressure |True | NoSchedule | node.kubernetes.io/diskPressure | +|DiskPressure |True | NoSchedule | node.kubernetes.io/disk-pressure | | |False | - | | | |Unknown | - | | -|NetworkUnavailable |True | NoSchedule | node.kubernetes.io/networkUnavailable | +|NetworkUnavailable |True | NoSchedule | node.kubernetes.io/network-unavailable | | |False | - | | | |Unknown | - | | -For example, if a CNI network is not detected on the node (e.g. a network is unavailable), the Node Controller will taint the node with `node.kubernetes.io/networkUnavailable=:NoSchedule`. This will then allow users to add a toleration to their `PodSpec`, ensuring that the pod can be scheduled to this node if necessary. If the kubelet did not update the node’s status after a grace period, the Node Controller will only taint the node with `node.kubernetes.io/unreachable`; it will not taint the node with any unknown condition. +For example, if a CNI network is not detected on the node (e.g. a network is unavailable), the Node Controller will taint the node with `node.kubernetes.io/network-unavailable=:NoSchedule`. This will then allow users to add a toleration to their `PodSpec`, ensuring that the pod can be scheduled to this node if necessary. If the kubelet did not update the node’s status after a grace period, the Node Controller will only taint the node with `node.kubernetes.io/unreachable`; it will not taint the node with any unknown condition. diff --git a/contributors/design-proposals/taint-toleration-dedicated.md b/contributors/design-proposals/scheduling/taint-toleration-dedicated.md index c523319f..e451f395 100644 --- a/contributors/design-proposals/taint-toleration-dedicated.md +++ b/contributors/design-proposals/scheduling/taint-toleration-dedicated.md @@ -283,9 +283,3 @@ The relationship between taints and node drains is discussed in [#1574](https:// The concepts of taints and tolerations were originally developed as part of the Omega project at Google. - - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/service-catalog/OWNERS b/contributors/design-proposals/service-catalog/OWNERS new file mode 100644 index 00000000..5c6b18ed --- /dev/null +++ b/contributors/design-proposals/service-catalog/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-service-catalog-leads +approvers: + - sig-service-catalog-leads +labels: + - sig/service-catalog diff --git a/contributors/design-proposals/pod-preset.md b/contributors/design-proposals/service-catalog/pod-preset.md index 53da4dd9..28f8d8a8 100644 --- a/contributors/design-proposals/pod-preset.md +++ b/contributors/design-proposals/service-catalog/pod-preset.md @@ -15,7 +15,7 @@ * [PodPreset Exclude Annotation](#podpreset-exclude-annotation) * [Examples](#examples) * [Simple Pod Spec Example](#simple-pod-spec-example) - * [Pod Spec with `ConfigMap` Example](#pod-spec-with-`configmap`-example) + * [Pod Spec with `ConfigMap` Example](#pod-spec-with-configmap-example) * [ReplicaSet with Pod Spec Example](#replicaset-with-pod-spec-example) * [Multiple PodPreset Example](#multiple-podpreset-example) * [Conflict Example](#conflict-example) @@ -69,7 +69,7 @@ information into every pod spec where it is needed. 2. Database Administrator creates secrets for the cluster containing the database name, username, and password. 3. Database Administrator creates a `PodPreset` defining the database - port as an enviornment variable, as well as the secrets. See + port as an environment variable, as well as the secrets. See [Examples](#examples) below for various examples. 4. Developer of an application can now label their pod with the specified `Selector` the Database Administrator tells them, and consume the MySQL diff --git a/contributors/design-proposals/sig-cli/OWNERS b/contributors/design-proposals/sig-cli/OWNERS deleted file mode 100644 index 47f590b6..00000000 --- a/contributors/design-proposals/sig-cli/OWNERS +++ /dev/null @@ -1,4 +0,0 @@ -reviewers: - - pwittrock -approvers: - - pwittrock diff --git a/contributors/design-proposals/storage/OWNERS b/contributors/design-proposals/storage/OWNERS new file mode 100644 index 00000000..fb58418f --- /dev/null +++ b/contributors/design-proposals/storage/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-storage-leads +approvers: + - sig-storage-leads +labels: + - sig/storage diff --git a/contributors/design-proposals/storage/container-storage-interface.md b/contributors/design-proposals/storage/container-storage-interface.md new file mode 100644 index 00000000..c72d1528 --- /dev/null +++ b/contributors/design-proposals/storage/container-storage-interface.md @@ -0,0 +1,488 @@ +# CSI Volume Plugins in Kubernetes Design Doc + +***Status:*** Pending + +***Version:*** Alpha + +***Author:*** Saad Ali ([@saad-ali](https://github.com/saad-ali), saadali@google.com) + +*This document was drafted [here](https://docs.google.com/document/d/10GDyPWbFE5tQunKMlTXbcWysUttMFhBFJRX8ntaS_4Y/edit?usp=sharing).* + +## Terminology + +Term | Definition +---|--- +Container Storage Interface (CSI) | A specification attempting to establish an industry standard interface that Container Orchestration Systems (COs) can use to expose arbitrary storage systems to their containerized workloads. +in-tree | Code that exists in the core Kubernetes repository. +out-of-tree | Code that exists somewhere outside the core Kubernetes repository. +CSI Volume Plugin | A new, in-tree volume plugin that acts as an adapter and enables out-of-tree, third-party CSI volume drivers to be used in Kubernetes. +CSI Volume Driver | An out-of-tree CSI compatible implementation of a volume plugin that can be used in Kubernetes through the Kubernetes CSI Volume Plugin. + + +## Background & Motivations + +Kubernetes volume plugins are currently “in-tree” meaning they are linked, compiled, built, and shipped with the core kubernetes binaries. Adding a new storage system to Kubernetes (a volume plugin) requires checking code into the core Kubernetes code repository. This is undesirable for many reasons including: + +1. Volume plugin development is tightly coupled and dependent on Kubernetes releases. +2. Kubernetes developers/community are responsible for testing and maintaining all volume plugins, instead of just testing and maintaining a stable plugin API. +3. Bugs in volume plugins can crash critical Kubernetes components, instead of just the plugin. +4. Volume plugins get full privileges of kubernetes components (kubelet and kube-controller-manager). +5. Plugin developers are forced to make plugin source code available, and can not choose to release just a binary. + +The existing [Flex Volume](/contributors/devel/flexvolume.md) plugin attempted to address this by exposing an exec based API for mount/unmount/attach/detach. Although it enables third party storage vendors to write drivers out-of-tree, it requires access to the root filesystem of node and master machines in order to deploy the third party driver files. + +Additionally, it doesn’t address another pain of in-tree volumes plugins: dependencies. Volume plugins tend to have many external requirements: dependencies on mount and filesystem tools, for example. These dependencies are assumed to be available on the underlying host OS, which often is not the case, and installing them requires direct machine access. There are efforts underway, for example https://github.com/kubernetes/community/pull/589, that are hoping to address this for in-tree volume plugins. But, enabling volume plugins to be completely containerized will make dependency management much easier. + +While Kubernetes has been dealing with these issues, the broader storage community has also been dealing with a fragmented story for how to make their storage system available in different Container Orchestration Systems (COs). Storage vendors have to either write and support multiple volume drivers for different COs or choose to not support some COs. + +The Container Storage Interface (CSI) is a specification that resulted from cooperation between community members from various COs--including Kubernetes, Mesos, Cloud Foundry, and Docker. The goal of this interface is to establish a standardized mechanism for COs to expose arbitrary storage systems to their containerized workloads. + +The primary motivation for Storage vendors to adopt the interface is a desire to make their system available to as many users as possible with as little work as possible. The primary motivation for COs to adopt the interface is to invest in a mechanism that will enable their users to use as many different storage systems as possible. In addition, for Kubernetes, adopting CSI will have the added benefit of moving volume plugins out of tree, and enabling volume plugins to be containerized. + +### Links + +* [Container Storage Interface (CSI) Spec](https://github.com/container-storage-interface/spec/blob/master/spec.md) + +## Objective + +The objective of this document is to document all the requirements for enabling a CSI compliant volume plugin (a CSI volume driver) in Kubernetes. + +## Goals + +* Define Kubernetes API for interacting with an arbitrary, third-party CSI volume drivers. +* Define mechanism by which Kubernetes master and node components will securely communicate with an arbitrary, third-party CSI volume drivers. +* Define mechanism by which Kubernetes master and node components will discover and register an arbitrary, third-party CSI volume driver deployed on Kubernetes. +* Recommend packaging requirements for Kubernetes compatible, third-party CSI Volume drivers. +* Recommend deployment process for Kubernetes compatible, third-party CSI Volume drivers on a Kubernetes cluster. + +## Non-Goals +* Replace [Flex Volume plugin](/contributors/devel/flexvolume.md) + * The Flex volume plugin exists as an exec based mechanism to create “out-of-tree” volume plugins. + * Because Flex drivers exist and depend on the Flex interface, it will continue to be supported with a stable API. + * The CSI Volume plugin will co-exist with Flex volume plugin. + +## Design Overview + +To support CSI Compliant Volume plugins, a new in-tree CSI Volume plugin will be introduced in Kubernetes. This new volume plugin will be the mechanism by which Kubernetes users (application developers and cluster admins) interact with external CSI volume drivers. + +The `SetUp`/`TearDown` calls for the new in-tree CSI volume plugin will directly invoke `NodePublishVolume` and `NodeUnpublishVolume` CSI RPCs through a unix domain socket on the node machine. + +Provision/delete and attach/detach must be handled by some external component that monitors the Kubernetes API on behalf of a CSI volume driver and invokes the appropriate CSI RPCs against it. + +To simplify integration, the Kubernetes team will offer a containers that captures all the Kubernetes specific logic and act as adapters between third-party containerized CSI volume drivers and Kubernetes (each deployment of a CSI driver would have it’s own instance of the adapter). + +## Design Details + +### Third-Party CSI Volume Drivers + +Kubernetes is as minimally prescriptive on the packaging and deployment of a CSI Volume Driver as possible. Use of the *Communication Channels* (documented below) is the only requirement for enabling an arbitrary external CSI compatible storage driver in Kubernetes. + +This document recommends a standard mechanism for deploying an arbitrary containerized CSI driver on Kubernetes. This can be used by a Storage Provider to simplify deployment of containerized CSI compatible volume drivers on Kubernetes (see the “Recommended Mechanism for Deploying CSI Drivers on Kubernetes” section below). This mechanism, however, is strictly optional. + +### Communication Channels + +#### Kubelet to CSI Driver Communication + +Kubelet (responsible for mount and unmount) will communicate with an external “CSI volume driver” running on the same host machine (whether containerized or not) via a Unix Domain Socket. + +CSI volume drivers should create a socket at the following path on the node machine: `/var/lib/kubelet/plugins/[SanitizedCSIDriverName]/csi.sock`. For alpha, kubelet will assume this is the location for the Unix Domain Socket to talk to the CSI volume driver. For the beta implementation, we can consider using the [Device Plugin Unix Domain Socket Registration](/contributors/design-proposals/resource-management/device-plugin.md#unix-socket) mechanism to register the Unix Domain Socket with kubelet. This mechanism would need to be extended to support registration of both CSI volume drivers and device plugins independently. + +`Sanitized CSIDriverName` is CSI driver name that does not contain dangerous character and can be used as annotation name. It can follow the same pattern that we use for [volume plugins](https://git.k8s.io/kubernetes/pkg/util/strings/escape.go#L27). Too long or too ugly driver names can be rejected, i.e. all components described in this document will report an error and won't talk to this CSI driver. Exact sanitization method is implementation detail (SHA in the worst case). + +Upon initialization of the external “CSI volume driver”, some external component must call the CSI method `GetNodeId` to get the mapping from Kubernetes Node names to CSI driver NodeID. It must then add the CSI driver NodeID to the `csi.volume.kubernetes.io/nodeid` annotation on the Kubernetes Node API object. The key of the annotation must be `csi.volume.kubernetes.io/nodeid`. The value of the annotation is a JSON blob, containing key/value pairs for each CSI driver. + +For example: +``` +csi.volume.kubernetes.io/nodeid: "{ \"driver1\": \"name1\", \"driver2\": \"name2\" } +``` + +This will enable the component that will issue `ControllerPublishVolume` calls to use the annotation as a mapping from cluster node ID to storage node ID. + +To enable easy deployment of an external containerized CSI volume driver, the Kubernetes team will provide a sidecar "Kubernetes CSI Helper" container that can manage the unix domain socket registration and NodeId initialization. This is detailed in the “Suggested Mechanism for Deploying CSI Drivers on Kubernetes” section below. + +#### Master to CSI Driver Communication + +Because CSI volume driver code is considered untrusted, it might not be allowed to run on the master. Therefore, the Kube controller manager (responsible for create, delete, attach, and detach) can not communicate via a Unix Domain Socket with the “CSI volume driver” container. Instead, the Kube controller manager will communicate with the external “CSI volume driver” through the Kubernetes API. + +More specifically, some external component must watch the Kubernetes API on behalf of the external CSI volume driver and trigger the appropriate operations against it. This eliminates the problems of discovery and securing a channel between the kube-controller-manager and the CSI volume driver. + +To enable easy deployment of an external containerized CSI volume driver on Kubernetes, without making the driver Kubernetes aware, Kubernetes will provide a sidecar “Kubernetes to CSI” proxy container that will watch the Kubernetes API and trigger the appropriate operations against the “CSI volume driver” container. This is detailed in the “Suggested Mechanism for Deploying CSI Drivers on Kubernetes” section below. + +The external component watching the Kubernetes API on behalf of the external CSI volume driver must handle provisioning, deleting, attaching, and detaching. + +##### Provisioning and Deleting + +Provisioning and deletion operations are handled using the existing [external provisioner mechanism](https://github.com/kubernetes-incubator/external-storage/tree/master/docs), where the external component watching the Kubernetes API on behalf of the external CSI volume driver will act as an external provisioner. + +In short, to dynamically provision a new CSI volume, a cluster admin would create a `StorageClass` with the provisioner corresponding to the name of the external provisioner handling provisioning requests on behalf of the CSI volume driver. + +To provision a new CSI volume, an end user would create a `PersistentVolumeClaim` object referencing this `StorageClass`. The external provisioner will react to the creation of the PVC and issue the `CreateVolume` call against the CSI volume driver to provision the volume. The `CreateVolume` name will be auto-generated as it is for other dynamically provisioned volumes. The `CreateVolume` capacity will be take from the `PersistentVolumeClaim` object. The `CreateVolume` parameters will be passed through from the `StorageClass` parameters (opaque to Kubernetes). Once the operation completes successfully, the external provisioner creates a `PersistentVolume` object to represent the volume using the information returned in the `CreateVolume` response. The `PersistentVolume` object is bound to the `PersistentVolumeClaim` and available for use. + +To delete a CSI volume, an end user would delete the corresponding `PersistentVolumeClaim` object. The external provisioner will react to the deletion of the PVC and based on its reclamation policy it will issue the `DeleteVolume` call against the CSI volume driver commands to delete the volume. It will then delete the `PersistentVolume` object. + +##### Attaching and Detaching + +Attach/detach operations must also be handled by an external component (an “attacher”). The attacher watches the Kubernetes API on behalf of the external CSI volume driver for new `VolumeAttachment` objects (defined below), and triggers the appropriate calls against the CSI volume driver to attach the volume. The attacher must watch for `VolumeAttachment` object and mark it as attached even if the underlying CSI driver does not support `ControllerPublishVolume` call, as Kubernetes has no knowledge about it. + +More specifically, an external “attacher” must watch the Kubernetes API on behalf of the external CSI volume driver to handle attach/detach requests. + +Once the following conditions are true, the external-attacher should call `ControllerPublishVolume` against the CSI volume driver to attach the volume to the specified node: + +1. A new `VolumeAttachment` Kubernetes API objects is created by Kubernetes attach/detach controller. +2. The `VolumeAttachment.Spec.Attacher` value in that object corresponds to the name of the external attacher. +3. The `VolumeAttachment.Status.Attached` value is not yet set to true. +4. A Kubernetes Node API object exists with the name matching `VolumeAttachment.Spec.NodeName` and that object contains a `csi.volume.kubernetes.io/nodeid` annotation. This annotation contains a JSON blob, a list of key/value pairs, where one of they keys corresponds with the CSI volume driver name, and the value is the NodeID for that driver. This NodeId mapping can be retrieved and used in the `ControllerPublishVolume` calls. +5. The `VolumeAttachment.Metadata.DeletionTimestamp` is not set. + +Before starting the `ControllerPublishVolume` operation, the external-attacher should add these finalizers to these Kubernetes API objects: + +* To the `VolumeAttachment` so that when the object is deleted, the external-attacher has an opportunity to detach the volume first. External attacher removes this finalized once the volume is fully detached from the node. +* To the `PersistentVolume` referenced by `VolumeAttachment` so the the PV cannot be deleted while the volume is attached. External attacher needs information from the PV to perform detach operation. The attacher will remove the finalizer once all `VolumeAttachment` objects that refer to the PV are deleted, i.e. the volume is detached from all nodes. + +If the operation completes successfully, the external-attacher will: + +1. Set `VolumeAttachment.Status.Attached` field to true to indicate the volume is attached. +2. Update the `VolumeAttachment.Status.AttachmentMetadata` field with the contents of the returned `PublishVolumeInfo`. +3. Clear the `VolumeAttachment.Status.AttachError` field. + +If the operation fails, the external-attacher will: + +1. Ensure the `VolumeAttachment.Status.Attached` field to still false to indicate the volume is not attached. +2. Set the `VolumeAttachment.Status.AttachError` field detailing the error. +3. Create an event against the Kubernetes API associated with the `VolumeAttachment` object to inform users what went wrong. + +The external-attacher may implement it’s own error recovery strategy, and retry as long as conditions specified for attachment above are valid. It is strongly recommended that the external-attacher implement an exponential backoff strategy for retries. + +The detach operation will be triggered by the deletion of the `VolumeAttachment` Kubernetes API objects. Since the `VolumeAttachment` Kubernetes API object will have a finalizer added by the external-attacher, it will wait for confirmation from the external-attacher before deleting the object. + +Once all the following conditions are true, the external-attacher should call `ControllerUnpublishVolume` against the CSI volume driver to detach the volume from the specified node: +1. A `VolumeAttachment` Kubernetes API object is marked for deletion: the value for the `VolumeAttachment.metadata.deletionTimestamp` field is set. + +If the operation completes successfully, the external-attacher will: +1. Remove its finalizer from the list of finalizers on the `VolumeAttachment` object permitting the delete operation to continue. + +If the operation fails, the external-attacher will: + +1. Ensure the `VolumeAttachment.Status.Attached` field remains true to indicate the volume is not yet detached. +2. Set the `VolumeAttachment.Status.DetachError` field detailing the error. +3. Create an event against the Kubernetes API associated with the `VolumeAttachment` object to inform users what went wrong. + +The new API object called `VolumeAttachment` will be defined as follows: + +```GO + +// VolumeAttachment captures the intent to attach or detach the specified volume +// to/from the specified node. +// +// VolumeAttachment objects are non-namespaced. +type VolumeAttachment struct { + metav1.TypeMeta `json:",inline"` + + // Standard object metadata. + // More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata + // +optional + metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"` + + // Specification of the desired attach/detach volume behavior. + // Populated by the Kubernetes system. + Spec VolumeAttachmentSpec `json:"spec" protobuf:"bytes,2,opt,name=spec"` + + // Status of the VolumeAttachment request. + // Populated by the entity completing the attach or detach + // operation, i.e. the external-attacher. + // +optional + Status VolumeAttachmentStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"` +} + +// The specification of a VolumeAttachment request. +type VolumeAttachmentSpec struct { + // Attacher indicates the name of the volume driver that MUST handle this + // request. This is the name returned by GetPluginName() and must be the + // same as StorageClass.Provisioner. + Attacher string `json:"attacher" protobuf:"bytes,1,opt,name=attacher"` + + // AttachedVolumeSource represents the volume that should be attached. + VolumeSource AttachedVolumeSource `json:"volumeSource" protobuf:"bytes,2,opt,name=volumeSource"` + + // Kubernetes node name that the volume should be attached to. + NodeName string `json:"nodeName" protobuf:"bytes,3,opt,name=nodeName"` +} + +// VolumeAttachmentSource represents a volume that should be attached. +// Right now only PersistenVolumes can be attached via external attacher, +// in future we may allow also inline volumes in pods. +// Exactly one member can be set. +type AttachedVolumeSource struct { + // Name of the persistent volume to attach. + // +optional + PersistentVolumeName *string `json:"persistentVolumeName,omitempty" protobuf:"bytes,1,opt,name=persistentVolumeName"` + + // Placeholder for *VolumeSource to accommodate inline volumes in pods. +} + +// The status of a VolumeAttachment request. +type VolumeAttachmentStatus struct { + // Indicates the volume is successfully attached. + // This field must only be set by the entity completing the attach + // operation, i.e. the external-attacher. + Attached bool `json:"attached" protobuf:"varint,1,opt,name=attached"` + + // Upon successful attach, this field is populated with any + // information returned by the attach operation that must be passed + // into subsequent WaitForAttach or Mount calls. + // This field must only be set by the entity completing the attach + // operation, i.e. the external-attacher. + // +optional + AttachmentMetadata map[string]string `json:"attachmentMetadata,omitempty" protobuf:"bytes,2,rep,name=attachmentMetadata"` + + // The most recent error encountered during attach operation, if any. + // This field must only be set by the entity completing the attach + // operation, i.e. the external-attacher. + // +optional + AttachError *VolumeError `json:"attachError,omitempty" protobuf:"bytes,3,opt,name=attachError,casttype=VolumeError"` + + // The most recent error encountered during detach operation, if any. + // This field must only be set by the entity completing the detach + // operation, i.e. the external-attacher. + // +optional + DetachError *VolumeError `json:"detachError,omitempty" protobuf:"bytes,4,opt,name=detachError,casttype=VolumeError"` +} + +// Captures an error encountered during a volume operation. +type VolumeError struct { + // Time the error was encountered. + // +optional + Time metav1.Time `json:"time,omitempty" protobuf:"bytes,1,opt,name=time"` + + // String detailing the error encountered during Attach or Detach operation. + // This string maybe logged, so it should not contain sensitive + // information. + // +optional + Message string `json:"message,omitempty" protobuf:"bytes,2,opt,name=message"` +} + +``` + +### Kubernetes In-Tree CSI Volume Plugin + +A new in-tree Kubernetes CSI Volume plugin will contain all the logic required for Kubernetes to communicate with an arbitrary, out-of-tree, third-party CSI compatible volume driver. + +The existing Kubernetes volume components (attach/detach controller, PVC/PV controller, Kubelet volume manager) will handle the lifecycle of the CSI volume plugin operations (everything from triggering volume provisioning/deleting, attaching/detaching, and mounting/unmounting) just as they do for existing in-tree volume plugins. + +#### Proposed API + +A new `CSIPersistentVolumeSource` object will be added to the Kubernetes API. It will be part of the existing `PersistentVolumeSource` object and thus can be used only via PersistentVolumes. CSI volumes will not be allow referencing directly from Pods without a `PersistentVolumeClaim`. + +```GO +type CSIPersistentVolumeSource struct { + // Driver is the name of the driver to use for this volume. + // Required. + Driver string `json:"driver" protobuf:"bytes,1,opt,name=driver"` + + // VolumeHandle is the unique volume name returned by the CSI volume + // plugin’s CreateVolume to refer to the volume on all subsequent calls. + VolumeHandle string `json:"volumeHandle" protobuf:"bytes,2,opt,name=volumeHandle"` + + // Optional: The value to pass to ControllerPublishVolumeRequest. + // Defaults to false (read/write). + // +optional + ReadOnly bool `json:"readOnly,omitempty" protobuf:"varint,5,opt,name=readOnly"` +} +``` + +#### Internal Interfaces + +The in-tree CSI volume plugin will implement the following internal Kubernetes volume interfaces: + +1. `VolumePlugin` + * Mounting/Unmounting of a volume to a specific path. +2. `AttachableVolumePlugin` + * Attach/detach of a volume to a given node. + +Notably, `ProvisionableVolumePlugin` and `DeletableVolumePlugin` are not implemented because provisioning and deleting for CSI volumes is handled by an external provisioner. + +#### Mount and Unmount + +The in-tree volume plugin’s SetUp and TearDown methods will trigger the `NodePublishVolume` and `NodeUnpublishVolume` CSI calls via Unix Domain Socket. Kubernetes will generate a unique `target_path` (unique per pod per volume) to pass via `NodePublishVolume` for the CSI plugin to mount the volume. Upon successful completion of the `NodeUnpublishVolume` call (once volume unmount has been verified), Kubernetes will delete the directory. + +The Kubernetes volume sub-system does not currently support block volumes (only file), so for alpha, the Kubernetes CSI volume plugin will only support file. + +#### Attaching and Detaching + +The attach/detach controller,running as part of the kube-controller-manager binary on the master, decides when a CSI volume must be attached or detached from a particular node. + +When the controller decides to attach a CSI volume, it will call the in-tree CSI volume plugin’s attach method. The in-tree CSI volume plugin’s attach method will do the following: + +1. Create a new `VolumeAttachment` object (defined in the “Communication Channels” section) to attach the volume. + * The name of the of the `VolumeAttachment` object will be `pv-<SHA256(PVName+NodeName)>`. + * `pv-` prefix is used to allow using other scheme(s) for inline volumes in the future, with their own prefix. + * SHA256 hash is to reduce length of `PVName` plus `NodeName` string, each of which could be max allowed name length (hexadecimal representation of SHA256 is 64 characters). + * `PVName` is `PV.name` of the attached PersistentVolume. + * `NodeName` is `Node.name` of the node where the volume should be attached to. + * If a `VolumeAttachment` object with the corresponding name already exists, the in-tree volume plugin will simply begin to poll it as defined below. The object is not modified; only the external-attacher should change the status fields; and the external-attacher is responsible for it’s own retry and error handling logic. +2. Poll the `VolumeAttachment` object waiting for one of the following conditions: + * The `VolumeAttachment.Status.Attached` field to become `true`. + * The operation completes successfully. + * An error to be set in the `VolumeAttachment.Status.AttachError` field. + * The operation terminates with the specified error. + * The operation to timeout. + * The operation terminates with timeout error. + * The `VolumeAttachment.DeletionTimestamp` is set. + * The operation terminates with an error indicating a detach operation is in progress. + * The `VolumeAttachment.Status.Attached` value must not be trusted. The attach/detach controller has to wait until the object is deleted by the external-attacher before creating a new instance of the object. + +When the controller decides to detach a CSI volume, it will call the in-tree CSI volume plugin’s detach method. The in-tree CSI volume plugin’s detach method will do the following: + +1. Delete the corresponding `VolumeAttachment` object (defined in the “Communication Channels” section) to indicate the volume should be detached. +2. Poll the `VolumeAttachment` object waiting for one of the following conditions: + * The `VolumeAttachment.Status.Attached` field to become false. + * The operation completes successfully. + * An error to be set in the `VolumeAttachment.Status.DetachError` field. + * The operation terminates with the specified error. + * The object to no longer exists. + * The operation completes successfully. + * The operation to timeout. + * The operation terminates with timeout error. + +### Recommended Mechanism for Deploying CSI Drivers on Kubernetes + +Although, Kubernetes does not dictate the packaging for a CSI volume driver, it offers the following recommendations to simplify deployment of a containerized CSI volume driver on Kubernetes. + + + +To deploy a containerized third-party CSI volume driver, it is recommended that storage vendors: + + * Create a “CSI volume driver” container that implements the volume plugin behavior and exposes a gRPC interface via a unix domain socket, as defined in the CSI spec (including Controller, Node, and Identity services). + * Bundle the “CSI volume driver” container with helper containers (external-attacher, external-provisioner, Kubernetes CSI Helper) that the Kubernetes team will provide (these helper containers will assist the “CSI volume driver” container in interacting with the Kubernetes system). More specifically, create the following Kubernetes objects: + * A `StatefulSet` (to facilitate communication with the Kubernetes controllers) that has: + * Replica size 1 + * Guarantees that no more than 1 instance of the pod will be running at once (so we don’t have to worry about multiple instances of the `external-provisioner` or `external-attacher` in the cluster). + * The following containers + * The “CSI volume driver” container created by the storage vendor. + * The `external-attacher` container provided by the Kubernetes team. + * The `external-provisioner` container provided by the Kubernetes team. + * The following volumes: + * `emptyDir` volume + * Mounted inside all containers at `/var/lib/csi/sockets/pluginproxy/` + * The “CSI volume driver” container should create its Unix Domain Socket in this directory to enable communication with the Kubernetes helper container(s) (`external-provisioner`, `external-attacher`). + * A `DaemonSet` (to facilitate communication with every instance of kubelet) that has: + * The following containers + * The “CSI volume driver” container created by the storage vendor. + * The “Kubernetes CSI Helper” container provided by the Kubernetes team + * Responsible for registering the unix domain socket with kubelet and initializing NodeId. + * The following volumes: + * `hostpath` volume + * Expose `/var/lib/kubelet/device-plugins/kubelet.sock` from the host. + * Mount only in “Kubernetes CSI Helper” container at `/var/lib/csi/sockets/kubelet.sock` + * The Kubernetes to CSI proxy container will use this unix domain socket to register the CSI driver’s unix domain socket with kubelet. + * `hostpath` volume + * Expose `/var/lib/kubelet/` from the host. + * Mount only in “CSI volume driver” container at `/var/lib/kubelet/` + * Ensure [bi-directional mount propagation](https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation) is enabled, so that any mounts setup inside this container are propagated back to the host machine. + * `hostpath` volume + * Expose `/var/lib/kubelet/plugins/[SanitizedCSIDriverName]/` from the host as `hostPath.type = "DirectoryOrCreate"`. + * Mount inside “CSI volume driver” container at the path the CSI gRPC socket will be created. + * This is the primary means of communication between Kubelet and the “CSI volume driver” container (gRPC over UDS). + * Have cluster admins deploy the above `StatefulSet` and `DaemonSet` to aded support for the storage system in their Kubernetes cluster. + +Alternatively, deployment could be simplified by having all components (including external-provisioner and external-attacher) in the same pod (DaemonSet). Doing so, however, would consume more resources, and require a leader election protocol (likely https://git.k8s.io/contrib/election) in the `external-provisioner` and `external-attacher` components. + +### Example Walkthrough + +#### Provisioning Volumes + +1. A cluster admin creates a `StorageClass` pointing to the CSI driver’s external-provisioner and specifying any parameters required by the driver. +2. A user creates a `PersistentVolumeClaim` referring to the new `StorageClass`. +3. The persistent volume controller realizes that dynamic provisioning is needed, and marks the PVC with a `volume.beta.kubernetes.io/storage-provisioner` annotation. +4. The external-provisioner for the CSI driver sees the `PersistentVolumeClaim` with the `volume.beta.kubernetes.io/storage-provisioner` annotation so it starts dynamic volume provisioning: + 1. It dereferences the `StorageClass` to collect the opaque parameters to use for provisioning. + 2. It calls `CreateVolume` against the CSI driver container with parameters from the `StorageClass` and `PersistentVolumeClaim` objects. +5. Once the volume is successfully created, the external-provisioner creates a `PersistentVolume` object to represent the newly created volume and binds it to the `PersistentVolumeClaim`. + +#### Deleting Volumes + +1. A user deletes a `PersistentVolumeClaim` object bound to a CSI volume. +2. The external-provisioner for the CSI driver sees the the `PersistentVolumeClaim` was deleted and triggers the retention policy: + 1. If the retention policy is `delete` + 1. The external-provisioner triggers volume deletion by issuing a `DeleteVolume` call against the CSI volume plugin container. + 2. Once the volume is successfully deleted, the external-provisioner deletes the corresponding `PersistentVolume` object. + 2. If the retention policy is `retain` + 1. The external-provisioner does not delete the `PersistentVolume` object. + +#### Attaching Volumes + +1. The Kubernetes attach/detach controller, running as part of the `kube-controller-manager` binary on the master, sees that a pod referencing a CSI volume plugin is scheduled to a node, so it calls the in-tree CSI volume plugin’s attach method. +2. The in-tree volume plugin creates a new `VolumeAttachment` object in the kubernetes API and waits for its status to change to completed or error. +3. The external-attacher sees the `VolumeAttachment` object and triggers a `ControllerPublish` against the CSI volume driver container to fulfil it (meaning the external-attacher container issues a gRPC call via underlying UNIX domain socket to the CSI driver container). +4. Upon successful completion of the `ControllerPublish` call the external-attacher updates the status of the `VolumeAttachment` object to indicate the volume is successful attached. +5. The in-tree volume plugin watching the status of the `VolumeAttachment` object in the kubernetes API, sees the `Attached` field set to true indicating the volume is attached, so It updates the attach/detach controller’s internal state to indicate the volume is attached. + +#### Detaching Volumes + +1. The Kubernetes attach/detach controller, running as part of the `kube-controller-manager` binary on the master, sees that a pod referencing an attached CSI volume plugin is terminated or deleted, so it calls the in-tree CSI volume plugin’s detach method. +2. The in-tree volume plugin deletes the corresponding `VolumeAttachment` object. +3. The external-attacher sees a `deletionTimestamp` set on the `VolumeAttachment` object and triggers a `ControllerUnpublish` against the CSI volume driver container to detach it. +4. Upon successful completion of the `ControllerUnpublish` call, the external-attacher removes the finalizer from the `VolumeAttachment` object to indicate successful completion of the detach operation allowing the `VolumeAttachment` object to be deleted. +5. The in-tree volume plugin waiting for the `VolumeAttachment` object sees it deleted and assumes the volume was successfully detached, so It updates the attach/detach controller’s internal state to indicate the volume is detached. + +#### Mounting Volumes + +1. The volume manager component of kubelet notices a new volume, referencing a CSI volume, has been scheduled to the node, so it calls the in-tree CSI volume plugin’s `WaitForAttach` method. +2. The in-tree volume plugin’s `WaitForAttach` method watches the `Attached` field of the `VolumeAttachment` object in the kubernetes API to become `true`, it then returns without error. +3. Kubelet then calls the in-tree CSI volume plugin’s `MountDevice` method which is a no-op and returns immediately. +4. Finally kubelet calls the in-tree CSI volume plugin’s mount (setup) method, which causes the in-tree volume plugin to issue a `NodePublishVolume` call via the registered unix domain socket to the local CSI driver. +5. Upon successful completion of the `NodePublishVolume` call the specified path is mounted into the pod container. + +#### Unmounting Volumes +1. The volume manager component of kubelet, notices a mounted CSI volume, referenced by a pod that has been deleted or terminated, so it calls the in-tree CSI volume plugin’s `UnmountDevice` method which is a no-op and returns immediately. +2. Next kubelet calls the in-tree CSI volume plugin’s unmount (teardown) method, which causes the in-tree volume plugin to issue a `NodeUnpublishVolume` call via the registered unix domain socket to the local CSI driver. If this call fails from any reason, kubelet re-tries the call periodically. +3. Upon successful completion of the `NodeUnpublishVolume` call the specified path is unmounted from the pod container. + + +### CSI Credentials + +This part of proposal is not going to be implemented in alpha release. + +#### End user credentials +CSI allows specifying *end user credentials* in all operations. Kubernetes does not have facility to configure a Secret per *user*, we usually track objects per *namespace*. Therefore we decided to postpone implementation of these credentials and wait until CSI is clarified. + +#### Volume specific credentials +Some storage technologies (e.g. iSCSI with CHAP) require credentials tied to the volume (iSCSI LUN) that must be used during `NodePublish` request. It is expected that these credentials will be provided during dynamic provisioning of the volume, however CSI `CreateVolume` response does not provide any. In case it gets fixed soon external provisioner can save the secrets in a dedicated namespace and make them available to external attacher and internal CSI volume plugin using these `CSIPersistentVolumeSource` fields: + +// ... +```go +type CSIPersistentVolumeSource struct { + + // Optional: MountSecretRef is a reference to the secret object containing + // sensitive information to pass to the CSI driver during NodePublish. + // This may be empty if no secret is required. If the secret object contains + // more than one secret, all secrets are passed. + // +optional + MountSecretRef *SecretReference `json:"mountSecretRef,omitempty" protobuf:"bytes,3,opt,name=mountSecretRef"` + + // Optional: AttachSecretRef is a reference to the secret object containing + // sensitive information to pass to the CSI driver during ControllerPublish. + // This may be empty if no secret is required. If the secret object contains + // more than one secret, all secrets are passed. + // +optional + AttachSecretRef *SecretReference `json:"attachSecretRef,omitempty" protobuf:"bytes,4,opt,name=attachSecretRef"` +} +``` + +Note that a malicious provisioner could obtain an arbitrary secret by setting the mount secret in PV object to whatever secret it wants. It is assumed that cluster admins will only run trusted provisioners. + +Because the kubelet would be responsible for fetching and passing the mount secret to the CSI driver,the Kubernetes NodeAuthorizer must be updated to allow kubelet read access to mount secrets. + +## Alternatives Considered + +### Extending PersistentVolume Object + +Instead of creating a new `VolumeAttachment` object, another option we considered was extending the exiting `PersistentVolume` object. + +`PersistentVolumeSpec` would be extended to include: +* List of nodes to attach the volume to (initially empty). + +`PersistentVolumeStatus` would be extended to include: +* List of nodes the volume was successfully attached to. + +We dismissed this approach because having attach/detach triggered by the creation/deletion of an object is much easier to manage (for both external-attacher and Kubernetes) and more robust (fewer corner cases to worry about). diff --git a/contributors/design-proposals/storage/container-storage-interface_diagram1.png b/contributors/design-proposals/storage/container-storage-interface_diagram1.png Binary files differnew file mode 100644 index 00000000..eb42add7 --- /dev/null +++ b/contributors/design-proposals/storage/container-storage-interface_diagram1.png diff --git a/contributors/design-proposals/storage/containerized-mounter-pod.md b/contributors/design-proposals/storage/containerized-mounter-pod.md new file mode 100644 index 00000000..00eb3884 --- /dev/null +++ b/contributors/design-proposals/storage/containerized-mounter-pod.md @@ -0,0 +1,149 @@ +# Containerized mounter using volume utilities in pods + +## Goal +Kubernetes should be able to run all utilities that are needed to provision/attach/mount/unmount/detach/delete volumes in *pods* instead of running them on *the host*. The host can be a minimal Linux distribution without tools to create e.g. Ceph RBD or mount GlusterFS volumes. + +## Secondary objectives +These are not requirements per se, just things to consider before drawing the final design. +* CNCF designs Container Storage Interface (CSI). So far, this CSI expects that "volume plugins" on each host are long-running processes with a fixed gRPC API. We should aim the same direction, hoping to switch to CSI when it's ready. In other words, there should be one long-running container for a volume plugin that serves all volumes of given type on a host. +* We should try to avoid complicated configuration. The system should work out of the box or with very limited configuration. + +## Terminology + +**Mount utilities** for a volume plugin are all tools that are necessary to use a volume plugin. This includes not only utilities needed to *mount* the filesystem (e.g. `mount.glusterfs` for Gluster), but also utilities needed to attach, detach, provision or delete the volume, such as `/usr/bin/rbd` for Ceph RBD. + +## User story +Admin wants to run Kubernetes on a distro that does not ship `mount.glusterfs` that's needed for GlusterFS volumes. +1. Admin installs and runs Kubernetes in any way. +1. Admin deploys a DaemonSet that runs a pod with `mount.glusterfs` on each node. In future, this could be done by installer. +1. User creates a pod that uses a GlusterFS volume. Kubelet finds a pod with mount utilities on the node and uses it to mount the volume instead of expecting that `mount.glusterfs` is available on the host. + +- User does not need to configure anything and sees the pod Running as usual. +- Admin just needs to deploy the DaemonSet. +- It's quite hard to update the DaemonSet, see below. + +## Alternatives +### Sidecar containers +We considered this user story: +* Admin installs Kubernetes. +* Admin configures Kubernetes to use sidecar container with template XXX for glusterfs mount/unmount operations and pod with template YYY for glusterfs provision/attach/detach/delete operations. These templates would be yaml files stored somewhere. +* User creates a pod that uses a GlusterFS volume. Kubelet find a sidecar template for gluster, injects it into the pod and runs it before any mount operation. It then uses `docker exec mount <what> <where>` to mount Gluster volumes for the pod. After that, it starts init containers and the "real" pod containers. +* User deletes the pod. Kubelet kills all "real" containers in the pod and uses the sidecar container to unmount gluster volumes. Finally, it kills the sidecar container. + +-> User does not need to configure anything and sees the pod Running as usual. +-> Admin needs to set up the templates. + +Similarly, when attaching/detaching a volume, attach/detach controller would spawn a pod on a random node and the controller would then use `kubectl exec <the pod> <any attach/detach utility>` to attach/detach the volume. E.g. Ceph RBD volume plugin needs to execute things during attach/detach. After the volume is attached, the controller would kill the pod. + +Advantages: +* It's probably easier to update the templates than update the DaemonSet. + +Drawbacks: +* Admin needs to store the templates somewhere. Where? +* Short-living processes instead of long-running ones that would mimic CSI (so we could catch bugs early or even redesign CSI). +* Needs some refactoring in kubelet - now kubelet mounts everything and then starts containers. We would need kubelet to start some container(s) first, then mount, then run the rest. This is probably possible, but needs better analysis (and I got lost in kubelet...) + +### Infrastructure containers + +Mount utilities could be also part of infrastructure container that holds network namespace (when using Docker). Now it's typically simple `pause` container that does not do anything, it could hold mount utilities too. + +Advantages: +* Easy to set up +* No extra container running + +Disadvantages: +* One container for all mount utilities. Admin needs to make a single container that holds utilities for e.g. both gluster and nfs and whatnot. +* Needs some refactoring in kubelet - now kubelet mounts everything and then starts containers. We would need kubelet to start some container(s) first, then mount, then run the rest. This is probably possible, but needs better analysis (and I got lost in kubelet...) +* Short-living processes instead of long-running ones that would mimic CSI (so we could catch bugs early or even redesign CSI). +* Infrastructure container is implementation detail and CRI does not even allow executing binaries in it. + +**We've decided to go with long running DaemonSet pod as described below.** + +## Design + +* Pod with mount utilities puts a registration JSON file into `/var/lib/kubelet/plugin-containers/<plugin name>.json` on the host with name of the container where mount utilities should be executed: + ```json + { + "podNamespace": "kubernetes-storage", + "podName": "gluster-daemon-set-xtzwv", + "podUID": "5d1942bd-7358-40e8-9547-a04345c85be9", + "containerName": "gluster" + } + ``` + * Pod UID is used to avoid situation when a pod with mount utilities is terminated and leaves its registration file on the host. Kubelet should not assume that newly started pod with the same namespace+name has the same mount utilities. + * All slashes in `<plugin name>` must be replaced with tilde, e.g. `kubernetes.io~glusterfs.json`. + * Creating the file must be atomic so kubelet cannot accidentally read partly written file. + + * All volume plugins use `VolumeHost.GetExec` to get the right exec interface when running their utilities. + + * Kubelet's implementation of `VolumeHost.GetExec` looks at `/var/lib/kubelet/plugin-containers/<plugin name>.json` if it has a container for given volume plugin. + * If the file exists and referred container is running, it returns `Exec` interface implementation that leads to CRI's `ExecSync` into the container (i.e. `docker exec <container> ...`) + * If the file does not exist or referred container is not running, it returns `Exec` interface implementation that leads to `os.Exec`. This way, pods do not need to remove the registration file when they're terminated. + * Kubelet does not cache content of `plugin-containers/`, one extra `open()`/`read()` with each exec won't harm and it makes Kubelet more robust to changes in the directory. + +* In future, this registration of volume plugin pods should be replaced by a gRPC interface based on Device Plugin registration. + +## Requirements on DaemonSets with mount utilities +These are rules that need to be followed by DaemonSet authors: +* One DaemonSet can serve mount utilities for one or more volume plugins. We expect that one volume plugin per DaemonSet will be the most popular choice. +* One DaemonSet must provide *all* utilities that are needed to provision, attach, mount, unmount, detach and delete a volume for a volume plugin, including `mkfs` and `fsck` utilities if they're needed. + * E.g. `mkfs.ext4` is likely to be available on all hosts, but a pod with mount utilities should not depend on that nor use it. + * Kernel modules should be available in the pod with mount utilities too. "Available" does not imply that they need to be shipped in a container, we expect that binding `/lib/modules` from host to `/lib/modules` in the pod will be enough for all modules that are needed by Kubernetes internal volume plugins (all distros I checked incl. the "minimal" ones ship scsi.ko, rbd.ko, nfs.ko and fuse). This will allow future flex volumes ship vendor-specific kernel modules. It's up to the vendor to ensure that any kernel module matches the kernel on the host. + * The only exception is udev (or similar device manager). Only one udev can run on a system, therefore it should run on the host. If a volume plugin needs to talk to udev (e.g. by calling `udevadm trigger`), they must do it on the host and not in a container with mount utilities. +* It is expected that these daemon sets will run privileged pods that will see host's `/proc`, `/dev`, `/sys`, `/var/lib/kubelet` and such. Especially `/var/lib/kubelet` must be mounted with shared mount propagation so kubelet can see mounts created by the pods. +* The pods with mount utilities should run some simple init as PID 1 that reaps zombies of potential fuse daemons. +* The pods with mount utilities must put a file into `/var/lib/kubelet/plugin-containers/<plugin name>.json` for each volume plugin it supports. It should overwrite any existing file - it's probably leftover from older pod. + * Admin is responsible to run only one pod with utilities for one volume plugin on a single host. When two pods for say GlusterFS are scheduled on the same node they will overwrite the registration file of each other. + * Downward API can be used to get pod's name and namespace. + * Root privileges (or CAP_DAC_OVERRIDE) are needed to write to `/var/lib/kubelet/plugin-containers/`. + +To sum it up, it's just a daemon set that spawns privileged pods, running a simple init and registering itself into Kubernetes by placing a file into well-known location. + +**Note**: It may be quite difficult to create a pod that see's host's `/dev` and `/sys`, contains necessary kernel modules, does the initialization right and reaps zombies. We're going to provide a template with all this. + +### Upgrade +Upgrade of DaemonSets with pods with fuse-based mount utilities needs to be done node by node and with extra care. Killing a pod with fuse daemon(s) inside will un-mount all volumes that are used by other pods on the host and may result in data loss. + +In order to update the fuse-based DaemonSet (=GlusterFS or CephFS), admin must do for every node: +* Mark the node as tainted. Only the pod with mount utilities can tolerate the taint, all other pods are evicted. As result, all volumes are unmounted and detached. +* Update the pod. +* Remove the taint. + +Is there a way how to make it with DaemonSet rolling update? Is there any other way how to do this upgrade better? + +### Containerized kubelet + +Kubelet should behave the same when it runs inside a container: +* Use `os.Exec` to run mount utilities inside its own container when no pod with mount utilities is registered. This is current behavior, `mkfs.ext4`, `lsblk`, `rbd` and such are executed in context of the kubelet's container now. +* Use `nsenter <host> mount` to mount things when no pod with mount utilities is registered. Again, this is current behavior. +* Use CRI's `ExecSync` to execute both utilities and the final `mount` when a pod with mount utilities is registered so everything is executed in this pod. + +## Open items + +* How will controller-manager talk to pods with mount utilities? + + 1. Mount pods expose a gRPC service. + * controller-manager must be configured with the service namespace + name. + * Some authentication must be implemented (=additional configuration of certificates and whatnot). + * -> seems to be complicated. + + 2. Mount pods run in a dedicated namespace and have labels that tell which volume plugins they can handle. + * controller manager scans a namespace with a labelselector and does `kubectl exec <pod>` to execute anything in the pod. + * Needs configuration of the namespace. + * Admin must make sure that nothing else can run in the namespace (e.g. rogue pods that would steal volumes). + * Admin/installer must configure access to the namespace so only pv-controller and attach-detach-controller can do `exec` there. + + 3. We allow pods to run on hosts that run controller-manager. + + * Usual socket in `/var/lib/kubelet/plugin-sockets` will work. + * Can it work on GKE? + +We do not implement any of these approaches, as we expect that most volume plugins are going to be moved to CSI soon-ish. The only affected volume plugins are: + +* Ceph dynamic provisioning - we can use external provisioner during tests. +* Flex - it has its own dynamic registration of flex drivers. + +## Implementation notes +As we expect that most volume plugins are going to be moved to CSI soon, all implementation of this proposal will be guarded by alpha feature gate "MountContainers" which is never going leave alpha. Whole implementation of this proposal is going to be removed when the plugins are fully moved to CSI. + +Corresponding e2e tests for internal volume plugins will initially run only when with the feature gate is enabled and they will continue running when we move the volume plugins to CSI to ensure we won't introduce regressions. diff --git a/contributors/design-proposals/containerized-mounter.md b/contributors/design-proposals/storage/containerized-mounter.md index b1c8f298..b1c8f298 100644 --- a/contributors/design-proposals/containerized-mounter.md +++ b/contributors/design-proposals/storage/containerized-mounter.md diff --git a/contributors/design-proposals/default-storage-class.md b/contributors/design-proposals/storage/default-storage-class.md index c454ce62..c454ce62 100644 --- a/contributors/design-proposals/default-storage-class.md +++ b/contributors/design-proposals/storage/default-storage-class.md diff --git a/contributors/design-proposals/storage/flexvolume-deployment.md b/contributors/design-proposals/storage/flexvolume-deployment.md new file mode 100644 index 00000000..0b40748b --- /dev/null +++ b/contributors/design-proposals/storage/flexvolume-deployment.md @@ -0,0 +1,166 @@ +# **Dynamic Flexvolume Plugin Discovery** + +## **Objective** + +Kubelet and controller-manager do not need to be restarted manually in order for new Flexvolume plugins to be recognized. + +## **Background** + +Beginning in version 1.8, the Kubernetes Storage SIG is putting a stop to accepting in-tree volume plugins and advises all storage providers to implement out-of-tree plugins. Currently, there are two recommended implementations: Container Storage Interface (CSI) and Flexvolume. + +[CSI](https://github.com/container-storage-interface/spec/blob/master/spec.md) provides a single interface that storage vendors can implement in order for their storage solutions to work across many different container orchestrators, and volume plugins are out-of-tree by design. This is a large effort, the full implementation of CSI is several quarters away, and there is a need for an immediate solution for storage vendors to continue adding volume plugins. + +[Flexvolume](/contributors/devel/flexvolume.md) is an in-tree plugin that has the ability to run any storage solution by executing volume commands against a user-provided driver on the Kubernetes host, and this currently exists today. However, the process of setting up Flexvolume is very manual, pushing it out of consideration for many users. Problems include having to copy the driver to a specific location in each node, manually restarting kubelet, and user's limited access to machines. + +An automated deployment technique is discussed in [Recommended Driver Deployment Method](#recommended-driver-deployment-method). The crucial change required to enable this method is allowing kubelet and controller manager to dynamically discover plugin changes. + + +## **Overview** + +When there is a modification of the driver directory, a notification is sent to the filesystem watch from kubelet or controller manager. When kubelet or controller-manager searches for plugins (such as when a volume needs to be mounted), if there is a signal from the watch, it probes the driver directory and loads currently installed drivers as volume plugins. + +The modification can be a driver install (addition), upgrade/downgrade (update), or uninstall (deletion). If a volume depends on an existing driver, it can be *updated* but not *deleted*. + +## **Detailed Design** + +In the volume plugin code, introduce a `PluginStub` interface containing a single method `Init()`, and have `VolumePlugin` extend it. Create a `PluginProber` type which extends `PluginStub` and includes methods `Init()` and `Probe()`. Change the type of plugins inside the volume plugin manager's plugin list to `PluginStub`. + +`Init()` initializes fsnotify, creates a watch on the driver directory as well as its subdirectories (if any), and spawn a goroutine listening to the signal. When the goroutine receives signal that a new directory is created, create a watch for the directory so that driver changes can be seen. + +`Probe()` scans the driver directory only when the goroutine sets a flag. If the flag is set, return true (indicating that new plugins are available) and the list of plugins. Otherwise, return false and nil. After the scan, the watch is refreshed to include the new list of subdirectories. The goroutine should only record a signal if there has been a 1-second delay since the last signal (see [Security Considerations](#security-considerations)). Because inotify (used by fsnotify) can only be used to watch an existing directory, the goroutine needs to maintain the invariant that the driver directory always exists. + +Iterating through the list of plugins inside `InitPlugins()` from `volume/plugins.go`, if the plugin is an instance of `PluginProber`, only call its `Init()` and nothing else. Add an additional field, `flexVolumePluginList`, in `VolumePluginMgr` as a cache. For every iteration of the plugin list, call `Probe()` and update `flexVolumePluginList` if true is returned, and iterate through the new plugin list. If the return value is false, iterate through the existing `flexVolumePluginList`. If `Probe()` fails, use the cached plugin instead. However, if the plugin fails to initialize, log the error but do not use the cached version. The user needs to be aware that their driver implementation has a problem initializing, so the system should not silently use an older version. + +Because Flexvolume has two separate plugin instantiations (attachable and non-attachable), it's worth considering the case when a driver that implements attach/detach is replaced with a driver that does not, or vice versa. This does not cause an issue because plugins are recreated every time the driver directory is changed. + +There is a possibility that a Flexvolume command execution occurs at the same time as the driver is updated, which leads to a bad execution. This cannot be solved within the Kubernetes system without an overhaul. Instead, this is discussed in [Atomic Driver Installation](#atomic-driver-installation) as part of the deployment mechanism. As part of the solution, the Prober will **ignore all files that begins with "."** in the driver directory. + +Word of caution about symlinks in the Flexvolume plugin directory: as a result of the recursive filesystem watch implementation, if a symlink links to a directory, unless the directory is visible to the prober (i.e. it's inside the Flexvolume plugin directory and does not start with '.'), the directory's files and subdirectories are not added to filesystem watch, thus their change will not trigger a probe. + + +## **Alternative Designs** + +1) Make `PluginProber` a separate component, and pass it around as a dependency. + +Pros: Avoids the common `PluginStub` interface. There isn't much shared functionality between `VolumePlugin` and `PluginProber`. The only purpose this shared abstraction serves is for `PluginProber` to reuse the existing machinery of plugins list. + +Cons: Would have to increase dependency surface area, notably `KubeletDeps`. + +I'm currently undecided whether to use this design or the `PluginStub` design. + +2) Use a polling model instead of a watch for probing for driver changes. + +Pros: Simpler to implement. + +Cons: Kubelet or controller manager iterates through the plugin list many times, so Probe() is called very frequently. Using this model would increase unnecessary disk usage. This issue is mitigated if we guarantee that `PluginProber` is the last `PluginStub` in the iteration, and only `Probe()` if no other plugin is matched, but this logic adds additional complexity. + +3) Use a polling model + cache. Poll every x seconds/minutes. + +Pros: Mostly mitigates issues with the previous approach. + +Cons: Depending on the polling period, either it's needlessly frequent, or it's too infrequent to pick up driver updates quickly. + +4) Have the `flexVolumePluginList` cache live in `PluginProber` instead of `VolumePluginMgr`. + +Pros: `VolumePluginMgr` doesn't need to treat Flexvolume plugins any differently from other plugins. + +Cons: `PluginProber` doesn't have the function to validate a plugin. This function lives in `VolumePluginMgr`. Alternatively, the function can be passed into `PluginProber`. + + +## **Security Considerations** + +The Flexvolume driver directory can be continuously modified (accidentally or maliciously), making every` Probe()` call trigger a disk read, and `Probe()` calls could happen every couple of milliseconds and in bursts (i.e. lots of calls at first and then silence for some time). This may decrease kubelet's or controller manager's disk IO usage, impacting the performance of other system operations. + +As a safety measure, add a 1-second minimum delay between the processing of filesystem watch signals. + + +## **Testing Plan** + +Add new unit tests in `plugin_tests.go` to cover new probing functionality and the heterogeneous plugin types in the plugins list. + +Add e2e tests that follow the user story. Write one for initial driver installation, one for an update for the same driver, one for adding another driver, and one for removing a driver. + +## **Recommended Driver Deployment Method** + +This section describes one possible method to automatically deploy Flexvolume drivers. The goal is that drivers must be deployed on nodes (and master when attach is required) without having to manually access any machine instance. + +Driver Installation: + +* Alice is a storage plugin author and would like to deploy a Flexvolume driver on all node instances. She creates an image by copying her driver and the [deployment script](#driver-deployment-script) to a busybox base image, and makes her image available Bob, a cluster admin. +* Bob modifies the existing deployment DaemonSet spec with the name of the given image, and creates the DaemonSet. +* Charlie, an end user, creates volumes using the installed plugin. + +The user story for driver update is similar: Alice creates a new image with her new drivers, and Bob deploys it using the DaemonSet spec. + +### Driver Deployment Script + +This script assumes that only a *single driver file* is necessary, and is located at `/$DRIVER` on the deployment image. + +``` bash +#!/bin/sh + +set -o errexit +set -o pipefail + +VENDOR=k8s.io +DRIVER=nfs + +# Assuming the single driver file is located at /$DRIVER inside the DaemonSet image. + +driver_dir=$VENDOR${VENDOR:+"~"}${DRIVER} +if [ ! -d "/flexmnt/$driver_dir" ]; then + mkdir "/flexmnt/$driver_dir" +fi + +cp "/$DRIVER" "/flexmnt/$driver_dir/.$DRIVER" +mv -f "/flexmnt/$driver_dir/.$DRIVER" "/flexmnt/$driver_dir/$DRIVER" + +while : ; do + sleep 3600 +done +``` + +### Deployment DaemonSet +``` yaml +apiVersion: extensions/v1beta1 +kind: DaemonSet +metadata: + name: flex-set +spec: + template: + metadata: + name: flex-deploy + labels: + app: flex-deploy + spec: + containers: + - image: <deployment_image> + name: flex-deploy + securityContext: + privileged: true + volumeMounts: + - mountPath: /flexmnt + name: flexvolume-mount + volumes: + - name: flexvolume-mount + hostPath: + path: <host_driver_directory> +``` + +### Atomic Driver Installation +Regular file copy is not an atomic file operation, so if it were used to install the driver, it's possible that kubelet or controller manager executes the driver when it's partially installed, or the driver gets modified while it's being executed. Care must be taken to ensure the installation operation is atomic. + +The deployment script provided above uses renaming, which is atomic, to ensure that from the perspective of kubelet or controller manager, the driver file is completely written to disk in a single operation. The file is first installed with a name prefixed with '.', which the prober ignores. + +### Alternatives + +* Using Jobs instead of DaemonSets to deploy. + +Pros: Designed for containers that eventually terminate. No need to have the container go into an infinite loop. + +Cons: Does not guarantee every node has a pod running. Pod anti-affinity can be used to ensure no more than one pod runs on the same node, but since the Job spec requests a constant number of pods to run to completion, Jobs cannot ensure that pods are scheduled on new nodes. + +## **Open Questions** + +* How does this system work with containerized kubelet? +* Are there any SELinux implications? diff --git a/contributors/design-proposals/storage/grow-volume-size.md b/contributors/design-proposals/storage/grow-volume-size.md new file mode 100644 index 00000000..4ef2ab2c --- /dev/null +++ b/contributors/design-proposals/storage/grow-volume-size.md @@ -0,0 +1,310 @@ +# Growing Persistent Volume size + +## Goals + +Enable users to increase size of PVs that their pods are using. The user will update PVC for requesting a new size. Underneath we expect that - a controller will apply the change to PV which is bound to the PVC. + +## Non Goals + +* Reducing size of Persistent Volumes: We realize that, reducing size of PV is way riskier than increasing it. Reducing size of a PV could be a destructive operation and it requires support from underlying file system and volume type. In most cases it also requires that file system being resized is unmounted. + +* Rebinding PV and PVC: Kubernetes will only attempt to resize the currently bound PV and PVC and will not attempt to relocate data from a PV to a new PV and rebind the PVC to newly created PV. + +## Use Cases + +* As a user I am running Mysql on a 100GB volume - but I am running out of space, I should be able to increase size of volume mysql is using without losing all my data. (*online and with data*) +* As a user I created a PVC requesting 2GB space. I am yet to start a pod with this PVC but I realize that I probably need more space. Without having to create a new PVC, I should be able to request more size with same PVC. (*offline and no data on disk*) +* As a user I was running a rails application with 5GB of assets PVC. I have taken my application offline for maintenance but I would like to grow asset PVC to 10GB in size. (*offline but with data*) +* As a user I am running an application on glusterfs. I should be able to resize the gluster volume without losing data or mount point. (*online and with data and without taking pod offline*) +* In the logging project we run on dedicated clusters, we start out with 187Gi PVs for each of the elastic search pods. However, the amount of logs being produced can vary greatly from one cluster to another and its not uncommon that these volumes fill and we need to grow them. + +## Volume Plugin Matrix + + +| Volume Plugin | Supports Resize | Requires File system Resize | Supported in 1.8 Release | +| ----------------| :---------------: | :--------------------------:| :----------------------: | +| EBS | Yes | Yes | Yes | +| GCE PD | Yes | Yes | Yes | +| GlusterFS | Yes | No | Yes | +| Cinder | Yes | Yes | Yes | +| Vsphere | Yes | Yes | No | +| Ceph RBD | Yes | Yes | No | +| Host Path | No | No | No | +| Azure Disk | Yes | Yes | No | +| Azure File | No | No | No | +| Cephfs | No | No | No | +| NFS | No | No | No | +| Flex | Yes | Maybe | No | +| LocalStorage | Yes | Yes | No | + + +## Implementation Design + +For volume type that requires both file system expansion and a volume plugin based modification, growing persistent volumes will be two +step process. + + +For volume types that only require volume plugin based api call, this will be one step process. + +### Prerequisite + +* `pvc.spec.resources.requests.storage` field of pvc object will become mutable after this change. +* #sig-api-machinery has agreed to allow pvc's status update from kubelet as long as pvc and node relationship + can be validated by node authorizer. +* This feature will be protected by an alpha feature gate, so as API changes needed for it. + + +### Admission Control and Validations + +* Resource quota code has to be updated to take into account PVC expand feature. +* In case volume plugin doesn’t support resize feature. The resize API request will be rejected and PVC object will not be saved. This check will be performed via an admission controller plugin. +* In case requested size is smaller than current size of PVC. A validation will be used to reject the API request. (This could be moved to admission controller plugin too.) +* Not all PVCs will be resizable even if underlying volume plugin allows that. Only dynamically provisioned volumes +which are explicitly enabled by an admin will be allowed to be resized. A plugin in admission controller will forbid +size update for PVCs for which resizing is not enabled by the admin. +* The design proposal for raw block devices should make sure that, users aren't able to resize raw block devices. + + +### Controller Manager resize + +A new controller called `volume_expand_controller` will listen for pvc size expansion requests and take action as needed. The steps performed in this +new controller will be: + +* Watch for pvc update requests and add pvc to controller's work queue if a increase in volume size was requested. Once PVC is added to + controller's work queue - `pvc.Status.Conditions` will be updated with `ResizeStarted: True`. +* For unbound or pending PVCs - resize will trigger no action in `volume_expand_controller`. +* If `pv.Spec.Capacity` already is of size greater or equal than requested size, similarly no action will be performed by the controller. +* A separate goroutine will read work queue and perform corresponding volume resize operation. If there is a resize operation in progress + for same volume then resize request will be pending and retried once previous resize request has completed. +* Controller resize in effect will be level based rather than edge based. If there are more than one pending resize request for same PVC then + new resize requests for same PVC will replace older pending request. +* Resize will be performed via volume plugin interface, executed inside a goroutine spawned by `operation_exectutor`. +* A new plugin interface called `volume.Expander` will be added to volume plugin interface. The `Expander` interface + will also define if volume requires a file system resize: + + ```go + type Expander interface { + // ExpandVolume expands the volume + ExpandVolumeDevice(spec *Spec, newSize resource.Quantity, oldSize resource.Quantity) error + RequiresFSResize() bool + } + ``` + +* The controller call to expand the PVC will look like: + +```go +func (og *operationGenerator) GenerateExpandVolumeFunc( + pvcWithResizeRequest *expandcache.PvcWithResizeRequest, + resizeMap expandcache.VolumeResizeMap) (func() error, error) { + + volumePlugin, err := og.volumePluginMgr.FindExpandablePluginBySpec(pvcWithResizeRequest.VolumeSpec) + expanderPlugin, err := volumePlugin.NewExpander(pvcWithResizeRequest.VolumeSpec) + + + expandFunc := func() error { + expandErr := expanderPlugin.ExpandVolumeDevice(pvcWithResizeRequest.ExpectedSize, pvcWithResizeRequest.CurrentSize) + + if expandErr != nil { + og.recorder.Eventf(pvcWithResizeRequest.PVC, v1.EventTypeWarning, kevents.VolumeResizeFailed, expandErr.Error()) + resizeMap.MarkResizeFailed(pvcWithResizeRequest, expandErr.Error()) + return expandErr + } + + // CloudProvider resize succeeded - lets mark api objects as resized + if expanderPlugin.RequiresFSResize() { + err := resizeMap.MarkForFileSystemResize(pvcWithResizeRequest) + if err != nil { + og.recorder.Eventf(pvcWithResizeRequest.PVC, v1.EventTypeWarning, kevents.VolumeResizeFailed, err.Error()) + return err + } + } else { + err := resizeMap.MarkAsResized(pvcWithResizeRequest) + + if err != nil { + og.recorder.Eventf(pvcWithResizeRequest.PVC, v1.EventTypeWarning, kevents.VolumeResizeFailed, err.Error()) + return err + } + } + return nil + + } + return expandFunc, nil +} +``` + +* Once volume expand is successful, the volume will be marked as expanded and new size will be updated in `pv.spec.capacity`. Any errors will be reported as *events* on PVC object. +* If resize failed in above step, in addition to events - `pvc.Status.Conditions` will be updated with `ResizeFailed: True`. Corresponding error will be added to condition field as well. +* Depending on volume type next steps would be: + + * If volume is of type that does not require file system resize, then `pvc.status.capacity` will be immediately updated to reflect new size. This would conclude the volume expand operation. Also `pvc.Status.Conditions` will be updated with `Ready: True`. + * If volume if of type that requires file system resize then a file system resize will be performed on kubelet. Read below for steps that will be performed for file system resize. + +* If volume plugin is of type that can not do resizing of attached volumes (such as `Cinder`) then `ExpandVolumeDevice` can return error by checking for + volume status with its own API (such as by making Openstack Cinder API call in this case). Controller will keep trying to resize the volume until it is + successful. + +* To consider cases of missed PVC update events, an additional loop will reconcile bound PVCs with PVs. This additional loop will loop through all PVCs + and match `pvc.spec.resources.requests` with `pv.spec.capacity` and add PVC in `volume_expand_controller`'s work queue if `pv.spec.capacity` is less + than `pvc.spec.resources.requests`. + +* There will be additional checks in controller that grows PV size - to ensure that we do not make volume plugin API calls that can reduce size of PV. + +### File system resize on kubelet + +A File system resize will be pending on PVC until a new pod that uses this volume is scheduled somewhere. While theoretically we *can* perform +online file system resize if volume type and file system supports it - we are leaving it for next iteration of this feature. + +#### Prerequisite of File system resize + +* `pv.spec.capacity` must be greater than `pvc.status.spec.capacity`. +* A fix in pv_controller has to made to fix `claim.Status.Capacity` only during binding. See comment by jan here - https://github.com/kubernetes/community/pull/657#discussion_r128008128 +* A fix in attach_detach controller has to be made to prevent fore detaching of volumes that are undergoing resize. +This can be done by checking `pvc.Status.Conditions` during force detach. `AttachedVolume` struct doesn't hold a reference to PVC - so PVC info can either be directly cached in `AttachedVolume` along with PV spec or it can be fetched from PersistentVolume's ClaimRef binding info. + +#### Steps for resizing file system available on Volume + +* When calling `MountDevice` or `Setup` call of volume plugin, volume manager will in addition compare `pv.spec.capacity` and `pvc.status.capacity` and if `pv.spec.capacity` is greater + than `pvc.status.spec.capacity` then volume manager will additionally resize the file system of volume. +* The call to resize file system will be performed inside `operation_generator.GenerateMountVolumeFunc`. `VolumeToMount` struct will be enhanced to store PVC as well. +* The flow of file system resize will be as follow: + * Perform a resize based on file system used inside block device. + * If resize succeeds, proceed with mounting the device as usual. + * If resize failed with an error that shows no file system exists on the device, then log a warning and proceed with format and mount. + * If resize failed with any other error then fail the mount operation. +* Any errors during file system resize will be added as *events* to Pod object and mount operation will be failed. +* If there are any errors during file system resize `pvc.Status.Conditions` will be updated with `ResizeFailed: True`. Any errors will be added to + `Conditions` field. +* File System resize will not be performed on kubelet where volume being attached is ReadOnly. This is similar to pattern being used for performing formatting. +* After file system resize is successful, `pvc.status.capacity` will be updated to match `pv.spec.capacity` and volume expand operation will be considered complete. Also `pvc.Status.Conditions` will be updated with `Ready: True`. + +#### Reduce coupling between resize operation and file system type + +A file system resize in general requires presence of tools such as `resize2fs` or `xfs_growfs` on the host where kubelet is running. There is a concern +that open coding call to different resize tools directly in Kubernetes will result in coupling between file system and resize operation. To solve this problem +we have considered following options: + +1. Write a library that abstracts away various file system operations, such as - resizing, formatting etc. + + Pros: + * Relatively well known pattern + + Cons: + * Depending on version with which Kubernetes is compiled with, we are still tied to which file systems are supported in which version + of kubernetes. +2. Ship a wrapper shell script that encapsulates various file system operations and as long as the shell script supports particular file system + the resize operation is supported. + Pros: + * Kubernetes Admin can easily replace default shell script with her own version and thereby adding support for more file system types. + + Cons: + * I don't know if there is a pattern that exists in kube today for shipping shell scripts that are called out from code in Kubernetes. Flex is + different because, none of the flex scripts are shipped with Kuberntes. +3. Ship resizing tools in a container. + + +Of all options - #3 is our best bet but we are not quite there yet. Hence, I would like to propose that we ship with support for +most common file systems in current release and we revisit this coupling and solve it in next release. + +## API and UI Design + +Given a PVC definition: + +```yaml +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: volume-claim + annotations: + volume.beta.kubernetes.io/storage-class: "generalssd" +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi +``` + +Users can request new size of underlying PV by simply editing the PVC and requesting new size: + +``` +~> kubectl edit pvc volume-claim +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: volume-claim + annotations: + volume.beta.kubernetes.io/storage-class: "generalssd" +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi +``` + +## API Changes + +### PVC API Change + +`pvc.spec.resources.requests.storage` field of pvc object will become mutable after this change. + +In addition to that PVC's status will have a `Conditions []PvcCondition` - which will be used +to communicate the status of PVC to the user. + +The API change will be protected by Alpha feature gate and api-server will not allow PVCs with +`Status.Conditions` field if feature is not enabled. `omitempty` in serialization format will +prevent presence of field if not set. + +So the `PersistentVolumeClaimStatus` will become: + +```go +type PersistentVolumeClaimStatus struct { + Phase PersistentVolumeClaimPhase + AccessModes []PersistentVolumeAccessMode + Capacity ResourceList + // New Field added as part of this Change + Conditions []PVCCondition +} + +// new API type added +type PVCCondition struct { + Type PVCConditionType + Status ConditionStatus + LastProbeTime metav1.Time + LastTransitionTime metav1.Time + Reason string + Message string +} + +// new API type +type PVCConditionType string + +// new Constants +const ( + PVCReady PVCConditionType = "Ready" + PVCResizeStarted PVCConditionType = "ResizeStarted" + PVCResizeFailed PVCResizeFailed = "ResizeFailed" +) +``` + +### StorageClass API change + +A new field called `AllowVolumeExpand` will be added to StorageClass. The default of this value +will be `false` and only if it is true - PVC expansion will be allowed. + +```go +type StorageClass struct { + metav1.TypeMeta + metav1.ObjectMeta + Provisioner string + Parameters map[string]string + // New Field added + // +optional + AllowVolumeExpand bool +} +``` + +### Other API changes + +This proposal relies on ability to update PVC status from kubelet. While updating PVC's status +a PATCH request must be made from kubelet to update the status. diff --git a/contributors/design-proposals/local-storage-overview.md b/contributors/design-proposals/storage/local-storage-overview.md index 8b69d4cb..de1298a0 100644 --- a/contributors/design-proposals/local-storage-overview.md +++ b/contributors/design-proposals/storage/local-storage-overview.md @@ -35,7 +35,7 @@ Today, ephemeral local storage is exposed to pods via the container’s writable Distributed filesystems and databases are the primary use cases for persistent local storage due to the following factors: * Performance: On cloud providers, local SSDs give better performance than remote disks. -* Cost: On baremetal, in addition to performance, local storage is typically cheaper and using it is a necessity to provision distributed filesystems. +* Cost: On bare metal, in addition to performance, local storage is typically cheaper and using it is a necessity to provision distributed filesystems. Distributed systems often use replication to provide fault tolerance, and can therefore tolerate node failures. However, data gravity is preferred for reducing replication traffic and cold startup latencies. @@ -50,7 +50,7 @@ Primary partitions are shared partitions that can provide ephemeral local storag This partition holds the kubelet’s root directory (`/var/lib/kubelet` by default) and `/var/log` directory. This partition may be shared between user pods, OS and Kubernetes system daemons. This partition can be consumed by pods via EmptyDir volumes, container logs, image layers and container writable layers. Kubelet will manage shared access and isolation of this partition. This partition is “ephemeral” and applications cannot expect any performance SLAs (Disk IOPS for example) from this partition. ### Runtime -This is an optional partition which runtimes can use for overlay filesystems. Kubelet will attempt to identify and provide shared access along with isolation to this partition. Container image layers and writable later is stored here. If the runtime partition exists, `root` parition will not hold any image layer or writable layers. +This is an optional partition which runtimes can use for overlay filesystems. Kubelet will attempt to identify and provide shared access along with isolation to this partition. Container image layers and writable later is stored here. If the runtime partition exists, `root` partition will not hold any image layer or writable layers. ## Secondary Partitions All other partitions are exposed as local persistent volumes. The PV interface allows for varying storage configurations to be supported, while hiding specific configuration details to the pod. All the local PVs can be queried and viewed from a cluster level using the existing PV object. Applications can continue to use their existing PVC specifications with minimal changes to request local storage. @@ -59,7 +59,7 @@ The local PVs can be precreated by an addon DaemonSet that discovers all the sec Local PVs can only provide semi-persistence, and are only suitable for specific use cases that need performance, data gravity and can tolerate data loss. If the node or PV fails, then either the pod cannot run, or the pod has to give up on the local PV and find a new one. Failure scenarios can be handled by unbinding the PVC from the local PV, and forcing the pod to reschedule and find a new PV. -Since local PVs are only accessible from specific nodes, the scheduler needs to take into account a PV's node constraint when placing pods. This can be generalized to a storage toplogy constraint, which can also work with zones, and in the future: racks, clusters, etc. +Since local PVs are only accessible from specific nodes, the scheduler needs to take into account a PV's node constraint when placing pods. This can be generalized to a storage topology constraint, which can also work with zones, and in the future: racks, clusters, etc. The term `Partitions` are used here to describe the main use cases for local storage. However, the proposal doesn't require a local volume to be an entire disk or a partition - it supports arbitrary directory. This implies that cluster administrator can create multiple local volumes in one partition, each has the capacity of the partition, or even create local volume under primary partitions. Unless strictly required, e.g. if you have only one partition in your host, this is strongly discouraged. For this reason, following description will use `partition` or `mount point` exclusively. @@ -87,7 +87,7 @@ The term `Partitions` are used here to describe the main use cases for local sto ```yaml apiVersion: v1 - kind: pod + kind: Pod metadata: name: foo spec: @@ -122,7 +122,7 @@ The term `Partitions` are used here to describe the main use cases for local sto ```yaml apiVersion: v1 - kind: pod + kind: Pod metadata: name: foo namespace: myns @@ -158,7 +158,7 @@ The term `Partitions` are used here to describe the main use cases for local sto ```yaml apiVersion: v1 - kind: pod + kind: Pod metadata: name: foo spec: @@ -182,7 +182,7 @@ The term `Partitions` are used here to describe the main use cases for local sto ```yaml apiVersion: v1 - kind: pod + kind: Pod metadata: name: foo spec: @@ -266,7 +266,7 @@ The term `Partitions` are used here to describe the main use cases for local sto terminationGracePeriodSeconds: 10 containers: - name: nginx - image: gcr.io/google_containers/nginx-slim:0.8 + image: k8s.gcr.io/nginx-slim:0.8 ports: - containerPort: 80 name: web @@ -332,7 +332,7 @@ The term `Partitions` are used here to describe the main use cases for local sto spec: <snip> nodeTolerations: - - key: node.alpha.kubernetes.io/notReady + - key: node.alpha.kubernetes.io/not-ready operator: TolerationOpExists tolerationSeconds: 600 - key: node.alpha.kubernetes.io/unreachable @@ -374,7 +374,7 @@ The term `Partitions` are used here to describe the main use cases for local sto 3. Whenever a pod experiences I/O issues with it's EmptyDir volume, Phippy reconfigures those pods to use an inline Persistent Volume, whose lifetime is tied to the pod. ```yaml apiVersion: v1 - kind: pod + kind: Pod metadata: name: foo spec: @@ -402,7 +402,7 @@ The term `Partitions` are used here to describe the main use cases for local sto ```yaml apiVersion: v1 - kind: pod + kind: Pod metadata: name: foo spec: @@ -444,7 +444,7 @@ The term `Partitions` are used here to describe the main use cases for local sto ```yaml apiVersion: v1 - kind: pod + kind: Pod metadata: name: foo spec: @@ -520,7 +520,7 @@ Note: Block access will be considered as a separate feature because it can work storage: 80Gi ``` -4. It is also possible for a PVC that requests `volumeType: block` to also use file-based bolume. In this situation, the block device would get formatted with the filesystem type specified in the PVC spec. And when the PVC gets destroyed, then the filesystem also gets destroyed to return back to the original block state. +4. It is also possible for a PVC that requests `volumeType: block` to also use file-based volume. In this situation, the block device would get formatted with the filesystem type specified in the PVC spec. And when the PVC gets destroyed, then the filesystem also gets destroyed to return back to the original block state. ```yaml kind: PersistentVolumeClaim @@ -575,7 +575,7 @@ Note: Block access will be considered as a separate feature because it can work ### Why is the kubelet managing logs? -Kubelet is managing access to shared storage on the node. Container logs outputted via it's stdout and stderr ends up on the shared storage that kubelet is managing. So, kubelet needs direct control over the log data to keep the containers running (by rotating logs), store them long enough for break glass situations and apply different storage policies in a multi-tenent cluster. All of these features are not easily expressible through external logging agents like journald for example. +Kubelet is managing access to shared storage on the node. Container logs outputted via it's stdout and stderr ends up on the shared storage that kubelet is managing. So, kubelet needs direct control over the log data to keep the containers running (by rotating logs), store them long enough for break glass situations and apply different storage policies in a multi-tenant cluster. All of these features are not easily expressible through external logging agents like journald for example. ### Master are upgraded prior to nodes. How should storage as a new compute resource be rolled out on to existing clusters? @@ -591,7 +591,7 @@ Kubelet will attempt to enforce capacity limits on a best effort basis. If the u ### Are LocalStorage PVs required to be a whole partition? -No, but it is the recommended way to ensure capacity and performance isolation. For HDDs, a whole disk is recommended for performance isolation. In some environments, multiple storage partitions are not available, so the only option is to share the same filesystem. In that case, directories in the same filesystem can be specified, and the adminstrator could configure group quota to provide capacity isolation. +No, but it is the recommended way to ensure capacity and performance isolation. For HDDs, a whole disk is recommended for performance isolation. In some environments, multiple storage partitions are not available, so the only option is to share the same filesystem. In that case, directories in the same filesystem can be specified, and the administrator could configure group quota to provide capacity isolation. # Features & Milestones diff --git a/contributors/design-proposals/storage/local-storage-pv.md b/contributors/design-proposals/storage/local-storage-pv.md new file mode 100644 index 00000000..d397c681 --- /dev/null +++ b/contributors/design-proposals/storage/local-storage-pv.md @@ -0,0 +1,763 @@ +# Local Storage Persistent Volumes + +Authors: @msau42, @vishh, @dhirajh, @ianchakeres + +This document presents a detailed design for supporting persistent local storage, +as outlined in [Local Storage Overview](local-storage-overview.md). +Supporting all the use cases for persistent local storage will take many releases, +so this document will be extended for each new release as we add more features. + +## Goals + +* Allow pods to mount any local block or filesystem based volume. +* Allow pods to mount dedicated local disks, or channeled partitions as volumes for +IOPS isolation. +* Allow pods do access local volumes without root privileges. +* Allow pods to access local volumes without needing to understand the storage +layout on every node. +* Persist local volumes and provide data gravity for pods. Any pod +using the local volume will be scheduled to the same node that the local volume +is on. +* Allow pods to release their local volume bindings and lose that volume's data +during failure conditions, such as node, storage or scheduling failures, where +the volume is not accessible for some user-configurable time. +* Allow pods to specify local storage as part of a Deployment or StatefulSet. +* Allow administrators to set up and configure local volumes with simple methods. +* Do not require administrators to manage the local volumes once provisioned +for a node. + +## Non-Goals + +* Provide data availability for a local volume beyond its local node. +* Support the use of HostPath volumes and Local PVs on the same volume. + +## Background + +In Kubernetes, there are two main types of storage: remote and local. + +Remote storage is typically used with persistent volumes where the data can +persist beyond the lifetime of the pod. + +Local storage is typically used with ephemeral volumes where the data only +persists during the lifetime of the pod. + +There is increasing demand for using local storage as persistent volumes, +especially for distributed filesystems and databases such as GlusterFS and +Cassandra. The main motivations for using persistent local storage, instead +of persistent remote storage include: + +* Performance: Local SSDs achieve higher IOPS and throughput than many +remote storage solutions. + +* Cost: Operational costs may be reduced by leveraging existing local storage, +especially in bare metal environments. Network storage can be expensive to +setup and maintain, and it may not be necessary for certain applications. + +## Use Cases + +### Distributed filesystems and databases + +Many distributed filesystem and database implementations, such as Cassandra and +GlusterFS, utilize the local storage on each node to form a storage cluster. +These systems typically have a replication feature that sends copies of the data +to other nodes in the cluster in order to provide fault tolerance in case of +node failures. Non-distributed, but replicated databases, like MySQL, can also +utilize local storage to store replicas. + +The main motivations for using local persistent storage are performance and +cost. Since the application handles data replication and fault tolerance, these +application pods do not need networked storage to provide shared access to data. +In addition, installing a high-performing NAS or SAN solution can be more +expensive, and more complex to configure and maintain than utilizing local +disks, especially if the node was already pre-installed with disks. Datacenter +infrastructure and operational costs can be reduced by increasing storage +utilization. + +These distributed systems are generally stateful, infrastructure applications +that provide data services to higher-level applications. They are expected to +run in a cluster with many other applications potentially sharing the same +nodes. Therefore, they expect to have high priority and node resource +guarantees. They typically are deployed using StatefulSets, custom +controllers, or operators. + +### Caching + +Caching is one of the recommended use cases for ephemeral local storage. The +cached data is backed by persistent storage, so local storage data durability is +not required. However, there is a use case for persistent local storage to +achieve data gravity for large caches. For large caches, if a pod restarts, +rebuilding the cache can take a long time. As an example, rebuilding a 100GB +cache from a hard disk with 150MB/s read throughput can take around 10 minutes. +If the service gets restarted and all the pods have to restart, then performance +and availability can be impacted while the pods are rebuilding. If the cache is +persisted, then cold startup latencies are reduced. + +Content-serving applications and producer/consumer workflows commonly utilize +caches for better performance. They are typically deployed using Deployments, +and could be isolated in its own cluster, or shared with other applications. + +## Environments + +### Baremetal + +In a baremetal environment, nodes may be configured with multiple local disks of +varying capacity, speeds and mediums. Mediums include spinning disks (HDDs) and +solid-state drives (SSDs), and capacities of each disk can range from hundreds +of GBs to tens of TB. Multiple disks may be arranged in JBOD or RAID configurations +to consume as persistent storage. + +Currently, the methods to use the additional disks are to: + +* Configure a distributed filesystem +* Configure a HostPath volume + +It is also possible to configure a NAS or SAN on a node as well. Speeds and +capacities will widely vary depending on the solution. + +### GCE/GKE + +GCE and GKE both have a local SSD feature that can create a VM instance with up +to 8 fixed-size 375GB local SSDs physically attached to the instance host and +appears as additional disks in the instance. The local SSDs have to be +configured at the VM creation time and cannot be dynamically attached to an +instance later. If the VM gets shutdown, terminated, pre-empted, or the host +encounters a non-recoverable error, then the SSD data will be lost. If the +guest OS reboots, or a live migration occurs, then the SSD data will be +preserved. + +### EC2 + +In EC2, the instance store feature attaches local HDDs or SSDs to a new instance +as additional disks. HDD capacities can go up to 24 2TB disks for the largest +configuration. SSD capacities can go up to 8 800GB disks or 2 2TB disks for the +largest configurations. Data on the instance store only persists across +instance reboot. + +## Limitations of current volumes + +The following is an overview of existing volume types in Kubernetes, and how +they cannot completely address the use cases for local persistent storage. + +* EmptyDir: A temporary directory for a pod that is created under the kubelet +root directory. The contents are deleted when a pod dies. Limitations: + + * Volume lifetime is bound to the pod lifetime. Pod failure is more likely +than node failure, so there can be increased network and storage activity to +recover data via replication and data backups when a replacement pod is started. + * Multiple disks are not supported unless the administrator aggregates them +into a spanned or RAID volume. In this case, all the storage is shared, and +IOPS guarantees cannot be provided. + * There is currently no method of distinguishing between HDDs and SDDs. The +“medium” field could be expanded, but it is not easily generalizable to +arbitrary types of mediums. + +* HostPath: A direct mapping to a specified directory on the node. The +directory is not managed by the cluster. Limitations: + + * Admin needs to manually setup directory permissions for the volume’s users. + * Admin has to manage the volume lifecycle manually and do cleanup of the data and +directories. + * All nodes have to have their local storage provisioned the same way in order to +use the same pod template. + * There can be path collision issues if multiple pods get scheduled to the same +node that want the same path + * If node affinity is specified, then the user has to do the pod scheduling +manually. + +* Provider’s block storage (GCE PD, AWS EBS, etc): A remote disk that can be +attached to a VM instance. The disk’s lifetime is independent of the pod’s +lifetime. Limitations: + + * Doesn’t meet performance requirements. +[Performance benchmarks on GCE](https://cloud.google.com/compute/docs/disks/performance) +show that local SSD can perform better than SSD persistent disks: + + * 16x read IOPS + * 11x write IOPS + * 6.5x read throughput + * 4.5x write throughput + +* Networked filesystems (NFS, GlusterFS, etc): A filesystem reachable over the +network that can provide shared access to data. Limitations: + + * Requires more configuration and setup, which adds operational burden and +cost. + * Requires a high performance network to achieve equivalent performance as +local disks, especially when compared to high-performance SSDs. + +Due to the current limitations in the existing volume types, a new method for +providing persistent local storage should be considered. + +## Feature Plan + +A detailed implementation plan can be found in the +[Storage SIG planning spreadsheet](https://docs.google.com/spreadsheets/d/1t4z5DYKjX2ZDlkTpCnp18icRAQqOE85C1T1r2gqJVck/view#gid=1566770776). +The following is a high level summary of the goals in each phase. + +### Phase 1 + +* Support Pod, Deployment, and StatefulSet requesting a single local volume +* Support pre-configured, statically partitioned, filesystem-based local volumes + +### Phase 2 + +* Block devices and raw partitions +* Smarter PV binding to consider local storage and pod scheduling constraints, +such as pod affinity/anti-affinity, and requesting multiple local volumes + +### Phase 3 + +* Support common partitioning patterns +* Volume taints and tolerations for unbinding volumes in error conditions + +### Phase 4 + +* Dynamic provisioning + +## Design + +A high level proposal with user workflows is available in the +[Local Storage Overview](local-storage-overview.md). + +This design section will focus on one phase at a time. Each new release will +extend this section. + +### Phase 1: 1.7 alpha + +#### Local Volume Plugin + +A new volume plugin will be introduced to represent logical block partitions and +filesystem mounts that are local to a node. Some examples include whole disks, +disk partitions, RAID volumes, LVM volumes, or even directories in a shared +partition. Multiple Local volumes can be created on a node, and is +accessed through a local mount point or path that is bind-mounted into the +container. It is only consumable as a PersistentVolumeSource because the PV +interface solves the pod spec portability problem and provides the following: + +* Abstracts volume implementation details for the pod and expresses volume +requirements in terms of general concepts, like capacity and class. This allows +for portable configuration, as the pod is not tied to specific volume instances. +* Allows volume management to be independent of the pod lifecycle. The volume can +survive container, pod and node restarts. +* Allows volume classification by StorageClass. +* Is uniquely identifiable within a cluster and is managed from a cluster-wide +view. + +There are major changes in PV and pod semantics when using Local volumes +compared to the typical remote storage volumes. + +* Since Local volumes are fixed to a node, a pod using that volume has to +always be scheduled on that node. +* Volume availability is tied to the node’s availability. If the node is +unavailable, then the volume is also unavailable, which impacts pod +availability. +* The volume’s data durability characteristics are determined by the underlying +storage system, and cannot be guaranteed by the plugin. A Local volume +in one environment can provide data durability, but in another environment may +only be ephemeral. As an example, in the GCE/GKE/AWS cloud environments, the +data in directly attached, physical SSDs is immediately deleted when the VM +instance terminates or becomes unavailable. + +Due to these differences in behaviors, Local volumes are not suitable for +general purpose use cases, and are only suitable for specific applications that +need storage performance and data gravity, and can tolerate data loss or +unavailability. Applications need to be aware of, and be able to handle these +differences in data durability and availability. + +Local volumes are similar to HostPath volumes in the following ways: + +* Partitions need to be configured by the storage administrator beforehand. +* Volume is referenced by the path to the partition. +* Provides the same underlying partition’s support for IOPS isolation. +* Volume is permanently attached to one node. +* Volume can be mounted by multiple pods on the same node. + +However, Local volumes will address these current issues with HostPath +volumes: + +* Security concerns allowing a pod to access any path in a node. Local +volumes cannot be consumed directly by a pod. They must be specified as a PV +source, so only users with storage provisioning privileges can determine which +paths on a node are available for consumption. +* Difficulty in permissions setup. Local volumes will support fsGroup so +that the admins do not need to setup the permissions beforehand, tying that +particular volume to a specific user/group. During the mount, the fsGroup +settings will be applied on the path. However, multiple pods +using the same volume should use the same fsGroup. +* Volume lifecycle is not clearly defined, and the volume has to be manually +cleaned up by users. For Local volumes, the PV has a clearly defined +lifecycle. Upon PVC deletion, the PV will be released (if it has the Delete +policy), and all the contents under the path will be deleted. In the future, +advanced cleanup options, like zeroing can also be specified for a more +comprehensive cleanup. + +##### API Changes + +All new changes are protected by a new feature gate, `PersistentLocalVolumes`. + +A new `LocalVolumeSource` type is added as a `PersistentVolumeSource`. For this +initial phase, the path can only be a mount point or a directory in a shared +filesystem. + +``` +type LocalVolumeSource struct { + // The full path to the volume on the node + // For alpha, this path must be a directory + // Once block as a source is supported, then this path can point to a block device + Path string +} + +type PersistentVolumeSource struct { + <snip> + // Local represents directly-attached storage with node affinity. + // +optional + Local *LocalVolumeSource +} +``` + +The relationship between a Local volume and its node will be expressed using +PersistentVolume node affinity, described in the following section. + +Users request Local volumes using PersistentVolumeClaims in the same manner as any +other volume type. The PVC will bind to a matching PV with the appropriate capacity, +AccessMode, and StorageClassName. Then the user specifies that PVC in their +Pod spec. There are no special annotations or fields that need to be set in the Pod +or PVC to distinguish between local and remote storage. It is abstracted by the +StorageClass. + +#### PersistentVolume Node Affinity + +PersistentVolume node affinity is a new concept and is similar to Pod node affinity, +except instead of specifying which nodes a Pod has to be scheduled to, it specifies which nodes +a PersistentVolume can be attached and mounted to, influencing scheduling of Pods that +use local volumes. + +For a Pod that uses a PV with node affinity, a new scheduler predicate +will evaluate that node affinity against the node's labels. For this initial phase, the +PV node affinity is only considered by the scheduler for already-bound PVs. It is not +considered during the initial PVC/PV binding, which will be addressed in a future release. + +Only the `requiredDuringSchedulingIgnoredDuringExecution` field will be supported. + +##### API Changes + +For the initial alpha phase, node affinity is expressed as an optional +annotation in the PersistentVolume object. + +``` +// AlphaStorageNodeAffinityAnnotation defines node affinity policies for a PersistentVolume. +// Value is a string of the json representation of type NodeAffinity +AlphaStorageNodeAffinityAnnotation = "volume.alpha.kubernetes.io/node-affinity" +``` + +#### Local volume initial configuration + +There are countless ways to configure local storage on a node, with different patterns to +follow depending on application requirements and use cases. Some use cases may require +dedicated disks; others may only need small partitions and are ok with sharing disks. +Instead of forcing a partitioning scheme on storage administrators, the Local volume +is represented by a path, and lets the administrators partition their storage however they +like, with a few minimum requirements: + +* The paths to the mount points are always consistent, even across reboots or when storage +is added or removed. +* The paths are backed by a filesystem (block devices or raw partitions are not supported for +the first phase) +* The directories have appropriate permissions for the provisioner to be able to set owners and +cleanup the volume. + +#### Local volume management + +Local PVs are statically created and not dynamically provisioned for the first phase. +To mitigate the amount of time an administrator has to spend managing Local volumes, +a Local static provisioner application will be provided to handle common scenarios. For +uncommon scenarios, a specialized provisioner can be written. + +The Local static provisioner will be developed in the +[kubernetes-incubator/external-storage](https://github.com/kubernetes-incubator) +repository, and will loosely follow the external provisioner design, with a few differences: + +* A provisioner instance needs to run on each node and only manage the local storage on its node. +* For phase 1, it does not handle dynamic provisioning. Instead, it performs static provisioning +by discovering available partitions mounted under configurable discovery directories. + +The basic design of the provisioner will have two separate handlers: one for PV deletion and +cleanup, and the other for static PV creation. A PersistentVolume informer will be created +and its cache will be used by both handlers. + +PV deletion will operate on the Update event. If the PV it provisioned changes to the “Released” +state, and if the reclaim policy is Delete, then it will cleanup the volume and then delete the PV, +removing it from the cache. + +PV creation does not operate on any informer events. Instead, it periodically monitors the discovery +directories, and will create a new PV for each path in the directory that is not in the PV cache. It +sets the "pv.kubernetes.io/provisioned-by" annotation so that it can distinguish which PVs it created. + +For phase 1, the allowed discovery file types are directories and mount points. The PV capacity +will be the capacity of the underlying filesystem. Therefore, PVs that are backed by shared +directories will report its capacity as the entire filesystem, potentially causing overcommittment. +Separate partitions are recommended for capacity isolation. + +The name of the PV needs to be unique across the cluster. The provisioner will hash the node name, +StorageClass name, and base file name in the volume path to generate a unique name. + +##### Packaging + +The provisioner is packaged as a container image and will run on each node in the cluster as part of +a DaemonSet. It needs to be run with a user or service account with the following permissions: + +* Create/delete/list/get PersistentVolumes - Can use the `system:persistentvolumeprovisioner` ClusterRoleBinding +* Get ConfigMaps - To access user configuration for the provisioner +* Get Nodes - To get the node's UID and labels + +These are broader permissions than necessary (a node's access to PVs should be restricted to only +those local to the node). A redesign will be considered in a future release to address this issue. + +In addition, it should run with high priority so that it can reliably handle all the local storage +partitions on each node, and with enough permissions to be able to cleanup volume contents upon +deletion. + +The provisioner DaemonSet requires the following configuration: + +* The node's name set as the MY_NODE_NAME environment variable +* ConfigMap with StorageClass -> discovery directory mappings +* Each mapping in the ConfigMap needs a hostPath volume +* User/service account with all the required permissions + +Here is an example ConfigMap: + +``` +kind: ConfigMap +metadata: + name: local-volume-config + namespace: kube-system +data: + storageClassMap: | + local-fast: + hostDir: "/mnt/ssds" + mountDir: "/local-ssds" + local-slow: + hostDir: "/mnt/hdds" + mountDir: "/local-hdds" +``` + +The `hostDir` is the discovery path on the host, and the `mountDir` is the path it is mounted to in +the provisioner container. The `hostDir` is required because the provisioner needs to create Local PVs +with the `Path` based off of `hostDir`, not `mountDir`. + +The DaemonSet for this example looks like: +``` + +apiVersion: extensions/v1beta1 +kind: DaemonSet +metadata: + name: local-storage-provisioner + namespace: kube-system +spec: + template: + metadata: + labels: + system: local-storage-provisioner + spec: + containers: + - name: provisioner + image: "k8s.gcr.io/local-storage-provisioner:v1.0" + imagePullPolicy: Always + volumeMounts: + - name: vol1 + mountPath: "/local-ssds" + - name: vol2 + mountPath: "/local-hdds" + env: + - name: MY_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + volumes: + - name: vol1 + hostPath: + path: "/mnt/ssds" + - name: vol2 + hostPath: + path: "/mnt/hdds" + serviceAccount: local-storage-admin +``` + +##### Provisioner Boostrapper + +Manually setting up this DaemonSet spec can be tedious and it requires duplicate specification +of the StorageClass -> directory mappings both in the ConfigMap and as hostPath volumes. To +make it simpler and less error prone, a boostrapper application will be provided to generate +and launch the provisioner DaemonSet based off of the ConfigMap. It can also create a service +account with all the required permissions. + +The boostrapper accepts the following optional arguments: + +* -image: Name of local volume provisioner image (default +"quay.io/external_storage/local-volume-provisioner:latest") +* -volume-config: Name of the local volume configuration configmap. The configmap must reside in the same +namespace as the bootstrapper. (default "local-volume-default-config") +* -serviceaccount: Name of the service accout for local volume provisioner (default "local-storage-admin") + +The boostrapper requires the following permissions: + +* Get/Create/Update ConfigMap +* Create ServiceAccount +* Create ClusterRoleBindings +* Create DaemonSet + +Since the boostrapper generates the DaemonSet spec, the ConfigMap can be simplifed to just specify the +host directories: + +``` +kind: ConfigMap +metadata: + name: local-volume-config + namespace: kube-system +data: + storageClassMap: | + local-fast: + hostDir: "/mnt/ssds" + local-slow: + hostDir: "/mnt/hdds" +``` + +The boostrapper will update the ConfigMap with the generated `mountDir`. It generates the `mountDir` +by stripping off the initial "/" in `hostDir`, replacing the remaining "/" with "~", and adding the +prefix path "/mnt/local-storage". + +In the above example, the generated `mountDir` is `/mnt/local-storage/mnt ~ssds` and +`/mnt/local-storage/mnt~hdds`, respectively. + +#### Use Case Deliverables + +This alpha phase for Local PV support will provide the following capabilities: + +* Local directories to be specified as Local PVs with node affinity +* Pod using a PVC that is bound to a Local PV will always be scheduled to that node +* External static provisioner DaemonSet that discovers local directories, creates, cleans up, +and deletes Local PVs + +#### Limitations + +However, some use cases will not work: + +* Specifying multiple Local PVCs in a pod. Most likely, the PVCs will be bound to Local PVs on +different nodes, +making the pod unschedulable. +* Specifying Pod affinity/anti-affinity with Local PVs. PVC binding does not look at Pod scheduling +constraints at all. +* Using Local PVs in a highly utilized cluster. PVC binding does not look at Pod resource requirements +and Node resource availability. + +These issues will be solved in a future release with advanced storage topology scheduling. + +As a workaround, PVCs can be manually prebound to Local PVs to essentially manually schedule Pods to +specific nodes. + +#### Test Cases + +##### API unit tests + +* LocalVolumeSource cannot be specified without the feature gate +* Non-empty PV node affinity is required for LocalVolumeSource +* Preferred node affinity is not allowed +* Path is required to be non-empty +* Invalid json representation of type NodeAffinity returns error + +##### PV node affinity unit tests + +* Nil or empty node affinity evaluates to true for any node +* Node affinity specifying existing node labels evaluates to true +* Node affinity specifying non-existing node label keys evaluates to false +* Node affinity specifying non-existing node label values evaluates to false + +##### Local volume plugin unit tests + +* Plugin can support PersistentVolumeSource +* Plugin cannot support VolumeSource +* Plugin supports ReadWriteOnce access mode +* Plugin does not support remaining access modes +* Plugin supports Mounter and Unmounter +* Plugin does not support Provisioner, Recycler, Deleter +* Plugin supports readonly +* Plugin GetVolumeName() returns PV name +* Plugin ConstructVolumeSpec() returns PV info +* Plugin disallows backsteps in the Path + +##### Local volume provisioner unit tests + +* Directory not in the cache and PV should be created +* Directory is in the cache and PV should not be created +* Directories created later are discovered and PV is created +* Unconfigured directories are ignored +* PVs are created with the configured StorageClass +* PV name generation hashed correctly using node name, storageclass and filename +* PV creation failure should not add directory to cache +* Non-directory type should not create a PV +* PV is released, PV should be deleted +* PV should not be deleted for any other PV phase +* PV deletion failure should not remove PV from cache +* PV cleanup failure should not delete PV or remove from cache + +##### E2E tests + +* Pod that is bound to a Local PV is scheduled to the correct node +and can mount, read, and write +* Two pods serially accessing the same Local PV can mount, read, and write +* Two pods simultaneously accessing the same Local PV can mount, read, and write +* Test both directory-based Local PV, and mount point-based Local PV +* Launch local volume provisioner, create some directories under the discovery path, +and verify that PVs are created and a Pod can mount, read, and write. +* After destroying a PVC managed by the local volume provisioner, it should cleanup +the volume and recreate a new PV. +* Pod using a Local PV with non-existent path fails to mount +* Pod that sets nodeName to a different node than the PV node affinity cannot schedule. + + +### Phase 2: 1.9 alpha + +#### Smarter PV binding + +The issue of PV binding not taking into account pod scheduling requirements affects any +type of volume that imposes topology constraints, such as local storage and zonal disks. + +Because this problem affects more than just local volumes, it will be treated as a +separate feature with a separate proposal. Once that feature is implemented, then the +limitations outlined above will be fixed. + +#### Block devices and raw partitions + +Pods accessing raw block storage is a new alpha feature in 1.9. Changes are required in +the Local volume plugin and provisioner to be able to support raw block devices. The local +volume provisioner will be enhanced to support discovery of block devices and creation of +PVs corresponding to those block devices. In addition, when a block device based PV is +released, the local volume provisioner will cleanup the block devices. The cleanup +mechanism will be configurable and also customizable as no single mechanism covers all use +cases. + + +##### Discovery + +Much like the current file based PVs, the local volume provisioner will look for block devices +under designated directories that have been mounted on the provisioner container. Currently, for +each storage class, the provisioner has a configmap entry that looks like this: + +``` +data: + storageClassMap: | + local-fast: + hostDir: "/mnt/disks" + mountDir: "/local-ssds" +``` + +With this current approach, filesystems that were meant to be exposed as PVs are supposed to be +mounted on sub-directories under hostDir and the provisioner running in a container would walk +through the corresponding "mountDir" to find all the PVs. + +For block discovery, we will extend the same approach to enable discovering block devices. The +admin can create symbolic links under hostDir for each block device that should be discovered +under that storage class. The provisioner would use the same configMap and its logic will be +enhanced to auto detect if the entry under the directory is a block device or a file system. If +it is a block device, then a block based PV is created, otherwise a file based PV is created. + +##### Cleanup after Release + +Cleanup of a block device can be a bit more involved for the following reasons: + +* With file based PVs, a quick deletion of all files (inode information) was sufficient, with +block devices one might want to wipe all current content. +* Overwriting SSDs is not guaranteed to securely cleanup all previous content as there is a +layer of indirection in SSDs called the FTL (flash translation layer) and also wear leveling +techniques in SSDs that prevent reliable overwrite of all previous content. +* SSDs can also suffer from wear if they are repeatedly subjected to zeroing out, so one would +need different tools and strategies for HDDs vs SSDs +* A cleanup process which favors overwriting every block in the disk can take several hours. + +For this reason, the cleanup process has been made configurable and extensible, so that admin +can use the most appropriate method for their environment. + +Block device cleanup logic will be encapsulated in separate scripts or binaries. There will be +several scripts that will be made available out of the box, for example: + + +| Cleanup Method | Description | Suitable for Device | +|:--------------:|-------------|:-------------------:| +|dd-zero| Used for zeroing the device repeatedly | HDD | +|blkdiscard| Discards sectors on the device. This cleanup method may not be supported by all devices.| SSD | +|fs-reset| A non-secure overwrite of any existing filesystem with mkfs, followed by wipefs to remove the signature of the file system | SSD/HDD | +|shred|Repeatedly writes random values to the block device. Less effective with wear levelling in SSDs.| HDD | +| hdparm| Issues [ATA secure erase](https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase) command to erase data on device. See ATA Secure Erase. Please note that the utility has to be supported by the device in question. | SSD/HDD | + +The fs-reset method is a quick and minimal approach as it does a reset of any file system, which +works for both SSD and HDD and will be the default choice for cleaning. For SSDs, admins could +opt for either blkdiscard which is also quite fast or hdparm. For HDDs they could opt for +dd-zeroing or shred, which can take some time to run. Finally, the user is free to create new +cleanup scripts of their own and have them specified in the configmap of the provisioner. + +The configmap from earlier section will be enhanced as follows +``` +data: + storageClassMap: | + local-fast: + hostDir: "/mnt/disks" + mountDir: "/local-ssds" + blockCleanerCommand: + - "/scripts/dd_zero.sh" + - "2" + ``` + +The block cleaner command will specify the script and any arguments that need to be passed to it. +The actual block device being cleaned will be supplied to the script as an environment variable +(LOCAL_PV_BLKDEVICE) as opposed to command line, so that the script command line has complete +freedom on its structure. The provisioner will validate that the block device path is actually +within the directory managed by the provisioner, to prevent destructive operations on arbitrary +paths. + +The provisioner logic currently does each volume’s cleanup as a synchronous serial activity. +However, with cleanup now potentially being a multi hour activity, the processes will have to +be asynchronous and capable of being executed in parallel. The provisioner will ensure that all +current asynchronous cleanup processes are tracked. Special care needs to be taken to ensure that +when a disk has only been partially cleaned. This scenario can happen if some impatient user +manually deletes a PV and the provisioner ends up re-creating pv ready for use (but only partially +cleaned). This issue will be addressed in the re-design of the provisioner (details will be provided +in the re-design section). The re-design will ensure that all disks being cleaned will be tracked +through custom resources, so no disk being cleaned will be re-created as a PV. + +The provisioner will also log events to let the user know that cleaning is in progress and it can +take some time to complete. + +##### Testing + +The unit tests in the provisioner will be enhanced to test all the new block discover, block cleaning +and asynchronous cleaning logic. The tests include +* Validating that a discovery directory containing both block and file system volumes are appropriately discovered and have PVs created. +* Validate that both success and failure of asynchronous cleanup processes are properly tracked by the provisioner +* Ensure a new PV is not created while cleaning of volume behind the PV is still in progress +* Ensure two simultaneous cleaning operations on the same PV do not occur + + In addition, end to end tests will be added to support block cleaning. The tests include: +* Validate block PV are discovered and created +* Validate cleaning of released block PV using each of the block cleaning scripts included. +* Validate that file and block volumes in the same discovery path have correct PVs created, and that they are appropriately cleaned up. +* Leverage block PV via PVC and validate that serially writes data in one pod, then reads and validates the data from a second pod. +* Restart of the provisioner during cleaning operations, and validate that the PV is not recreated by the provisioner until cleaning has occurred. + +#### Provisioner redesign for stricter K8s API access control + +In 1.7, each instance of the provisioner on each node has full permissions to create and +delete all PVs in the system. This is unnecessary and potentially a vulnerability if the +node gets compromised. + +To address this issue, the provisioner will be redesigned into two major components: + +1. A central manager pod that handles the creation and deletion of PV objects. +This central pod can run on a trusted node and be given PV create/delete permissions. +2. Worker pods on each node, run as a DaemonSet, that discovers and cleans up the local +volumes on that node. These workers do not interact with PV objects, however +they still require permissions to be able to read the `Node.Labels` on their node. + +The central manager will poll each worker for their discovered volumes and create PVs for +them. When a PV is released, then it will send the cleanup request to the worker. + +Detailed design TBD diff --git a/contributors/design-proposals/mount-options.md b/contributors/design-proposals/storage/mount-options.md index 099e8602..099e8602 100644 --- a/contributors/design-proposals/mount-options.md +++ b/contributors/design-proposals/storage/mount-options.md diff --git a/contributors/design-proposals/persistent-storage.md b/contributors/design-proposals/storage/persistent-storage.md index 7d413218..d91ee256 100644 --- a/contributors/design-proposals/persistent-storage.md +++ b/contributors/design-proposals/storage/persistent-storage.md @@ -286,7 +286,3 @@ reference from the PV and change the PVs status to 'Released'. Admins can script the recycling of released volumes. Future dynamic provisioners will understand how a volume should be recycled. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/pod-safety.md b/contributors/design-proposals/storage/pod-safety.md index 10f7589b..cca09292 100644 --- a/contributors/design-proposals/pod-safety.md +++ b/contributors/design-proposals/storage/pod-safety.md @@ -105,7 +105,7 @@ Deleting a pod with grace period 0 is called **force deletion** and will update the pod with a `deletionGracePeriodSeconds` of 0, and then immediately remove the pod from etcd. Because all communication is asynchronous, force deleting a pod means that the pod processes may continue -to run for an arbitary amount of time. If a higher level component like the +to run for an arbitrary amount of time. If a higher level component like the StatefulSet controller treats the existence of the pod API object as a strongly consistent entity, deleting the pod in this fashion will violate the at-most-one guarantee we wish to offer for pet sets. @@ -400,8 +400,3 @@ and unambiguous way to end users. force deleting them. * Decision: YES - captured above. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md b/contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md new file mode 100644 index 00000000..4885bcbf --- /dev/null +++ b/contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md @@ -0,0 +1,106 @@ +# Postpone Deletion of a Persistent Volume Claim in case It Is Used by a Pod + +Status: Proposal + +Version: GA + +Implementation Owner: @pospispa + +## Motivation + +User can delete a Persistent Volume Claim (PVC) that is being used by a pod. This may have negative impact on the pod and it may result in data loss. + +For more details see issue https://github.com/kubernetes/kubernetes/issues/45143 + +## Proposal + +Postpone the PVC deletion until the PVC is not used by any pod. + +## User Experience + +### Use Cases + +1. User deletes a PVC that is being used by a pod. This may have negative impact on the pod and may result in data loss. As a user, I want that any PVC deletion does not have any negative impact on any pod. As a user, I do not want to experience data loss. + +#### Scenarios for data loss +Depending on the storage type the data loss occurs in one of the below scenarios: +- in case the dynamic provisioning is used and reclaim policy is `Delete` the PVC deletion triggers deletion of the associated storage asset and PV. +- the same as above applies for the static provisioning and `Delete` reclaim policy. + +## Implementation + +### API Server, PVC Admission Controller, PVC Create +A new plugin for PVC admission controller will be created. The plugin will automatically add finalizer information into newly created PVC's metadata. + +### Scheduler +Scheduler will check if a pod uses a PVC and if any of the PVCs has `deletionTimestamp` set. In case this is true an error will be logged: "PVC (%pvcName) is in scheduled for deletion state" and scheduler will behave as if PVC was not found. + +### Kubelet +Kubelet does currently live lookup of PVC(s) that are used by a pod. + +In case any of the PVC(s) used by the pod has the `deletionTimestamp` set kubelet won't start the pod but will report and error: "can't start pod (%pod) because it's using PVC (%pvcName) that is being deleted". Kubelet will follow the same code path as if PVC(s) do not exist. + +### PVC Finalizing Controller +PVC finalizing controller is a new internal controller. + +PVC finalizing controller watches for both PVC and pod events that are processed as described below: +1. PVC add/update/delete events: + - If `deletionTimestamp` is `nil` and finalizer is missing, the PVC is added to PVC queue. + - If `deletionTimestamp` is `non-nil` and finalizer is present, the PVC is added to PVC queue. +2. Pod add events: + - If pod is terminated, all referenced PVCs are added to PVC queue. +3. Pod update events: + - If pod is changing from non-terminated to terminated state, all referenced PVCs are added to PVC queue. +4. Pod delete events: + - All referenced PVCs are added to PVC queue. + +PVC and pod information are kept in a cache that is done inherently for an informer. + +The PVC queue holds PVCs that need to be processed according to the below rules: +- If PVC is not found in cache, the PVC is skipped. +- If PVC is in cache with `nil` `deletionTimestamp` and missing finalizer, finalizer is added to the PVC. In case the adding finalizer operation fails, the PVC is re-queued into the PVC queue. +- If PVC is in cache with `non-nil` `deletionTimestamp` and finalizer is present, live pod list is done for the PVC namespace. If all pods referencing the PVC are not yet bound to a node or are terminated, the finalizer removal is attempted. In case the finalizer removal operation fails the PVC is re-queued. + +### CLI +In case a PVC has the `deletionTimestamp` set the commands `kubectl get pvc` and `kubectl describe pvc` will display that the PVC is in terminating state. + +### Client/Server Backwards/Forwards compatibility + +N/A + +## Alternatives considered + +1. Check in admission controller whether PVC can be deleted by listing all pods and checking if the PVC is used by a pod. This was discussed and rejected in PR https://github.com/kubernetes/kubernetes/pull/46573 + +There were alternatives discussed in issue https://github.com/kubernetes/kubernetes/issues/45143 + +### Scheduler Live Lookups PVC(s) Instead of Kubelet +The implementation proposes that kubelet live updates PVC(s) used by a pod before it starts the pod in order not to start a pod that uses a PVC that has the `deletionTimestamp` set. + +An alternative is that scheduler will live update PVC(s) used by a pod in order not to schedule a pod that uses a PVC that has the `deletionTimestamp` set. + +But live update represents a performance penalty. As the live update performance penalty is already present in the kubelet it's better to do the live update in kubelet. + +### Scheduler Maintains PVCUsedByPod Information in PVC +Scheduler will maintain information on both pods and PVCs from API server. + +In case a pod is being scheduled and is using PVCs that do not have condition PVCUsedByPod set it will set this condition for these PVCs. + +In case a pod is terminated and was using PVCs the scheduler will update PVCUsedByPod condition for these PVCs accordingly. + +PVC finalizing controller won't watch pods because the information whether a PVC is used by a pod or not is now maintained by the scheduler. + +In case PVC finalizing controller gets an update of a PVC and this PVC has `deletionTimestamp` set it will do live PVC update for this PVC in order to get up-to-date value of its PVCUsedByPod field. In case the PVCUsedByPod is not true it will remove the finalizer information from this PVC. + +### Scheduler In the Role of PVC Finalizing Controller +Scheduler will be responsible for removing the finalizer information from PVCs that are being deleted. + +So scheduler will watch pods and PVCs and will maintain internal cache of pods and PVCs. + +In case a PVC is deleted scheduler will do one of the below: +- In case the PVC is used by a pod it will add the PVC into its internal set of PVCs that are waiting for deletion. +- In case the PVC is not used by a pod it will remove the finalizer information from the PVC metadata. + +Note: scheduler is the source of truth of pods that are being started. The information on active pods may be a little bit outdated that causes that deletion of a PVC may be postponed (pod status in schedular is active while the pod is terminated in API server), but this does not cause any harm. + +The disadvantage is that scheduler will become responsible for PVC deletion postponing that will make scheduler bigger. diff --git a/contributors/design-proposals/storage/raw-block-pv.md b/contributors/design-proposals/storage/raw-block-pv.md new file mode 100644 index 00000000..f180845e --- /dev/null +++ b/contributors/design-proposals/storage/raw-block-pv.md @@ -0,0 +1,768 @@ +# Raw Block Consumption in Kubernetes + +Authors: erinboyd@, screeley44@, mtanino@ + +This document presents a proposal for managing raw block storage in Kubernetes using the persistent volume source API as a consistent model of consumption. + +# Terminology +* Raw Block Device - a physically attached device devoid of a filesystem +* Raw Block Volume - a logical abstraction of the raw block device as defined by a path +* Filesystem on Block - a formatted (ie xfs) filesystem on top of a raw block device + +# Goals +* Enable durable access to block storage +* Provide flexibility for users/vendors to utilize various types of storage devices +* Agree on API changes for block +* Provide a consistent security model for block devices +* Provide a means for running containerized block storage offerings as non-privileged container + +# Non Goals +* Support all storage devices natively in upstream Kubernetes. Non-standard storage devices are expected to be managed using extension + mechanisms. +* Provide a means for full integration into the scheduler based on non-storage related requests (CPU, etc.) +* Provide a means of ensuring specific topology to ensure co-location of the data + +# Value add to Kubernetes + + By extending the API for volumes to specifically request a raw block device, we provide an explicit method for volume consumption, + whereas previously any request for storage was always fulfilled with a formatted fileystem, even when the underlying storage was + block. In addition, the ability to use a raw block device without a filesystem will allow + Kubernetes better support of high performance applications that can utilize raw block devices directly for their storage. + Block volumes are critical to applications like databases (MongoDB, Cassandra) that require consistent I/O performance + and low latency. For mission critical applications, like SAP, block storage is a requirement. + + For applications that use block storage natively (like MongoDB) no additional configuration is required as the mount path passed + to the application provides the device which MongoDB then uses for the storage path in the configuration file (dbpath). Specific + tuning for each application to achieve the highest possibly performance is provided as part of its recommended configurations. + + Specific use cases around improved usage of storage consumption are included in the use cases listed below as follows: + * An admin wishes to expose a block volume to be consumed as a block volume for the user + * An admin wishes to expose a block volume to be consumed as a block volume for an administrative function such + as bootstrapping + * A user wishes to utilize block storage to fully realize the performance of an application tuned to using block devices + * A user wishes to read from a block storage device and write to a filesystem (big data analytics processing) + Future use cases include dynamically provisioning and intelligent discovery of existing devices, which this proposal sets the + foundation for more fully developing these methods. + + +# Design Overview + + The proposed design is based on the idea of leveraging well defined concepts for storage in Kubernetes. The consumption and + definitions for the block devices will be driven through the PVC and PV definitions. Along with Storage + Resource definitions, this will provide the admin with a consistent way of managing all storage. + The API changes proposed in the following section are minimal with the idea of defining a volumeMode to indicate both the definition + and consumption of the devices. Since it's possible to create a volume as a block device and then later consume it by provisioning + a filesystem on top, the design requires explicit intent for how the volume will be used. + The additional benefit of explicitly defining how the volume is to be consumed will provide a means for indicating the method + by which the device should be scrubbed when the claim is deleted, as this method will differ from a raw block device compared to a + filesystem. The ownership and responsibility of defining the retention policy shall be up to the plugin method being utilized and is + not covered in this proposal. + + Limiting use of the volumeMode to block can be executed through the use of storage resource quotas and storageClasses defined by the + administrator. + + To ensure backwards compatibility and a phased transition of this feature, the consensus from the community is to intentionally disable + the volumeMode: Block for both in-tree and external provisioners until a suitable implementation for provisioner versioning has been + accepted and implemented in the community. In addition, in-tree provisioners should be able to gracefully ignore volumeMode API objects + for plugins that haven't been updated to accept this value. + + It is important to note that when a PV is bound, it is either bound as a raw block device or formatted with a filesystem. Therefore, + the PVC drives the request and intended usage of the device by specifying the volumeMode as part of the API. This design lends itself + to future support of dynamic provisioning by also letting the request initiate from the PVC defining the role for the PV. It also + allows flexibility in the implementation and storage plugins to determine their support of this feature. Acceptable values for + volumeMode are 'Block' and 'Filesystem'. Where 'Filesystem' is the default value today and not required to be set in the PV/PVC. + +# Proposed API Changes + +## Persistent Volume Claim API Changes: +In the simplest case of static provisioning, a user asks for a volumeMode of block. The binder will only bind to a PV defined +with the same volumeMode. + +``` +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: myclaim +spec: + volumeMode: Block #proposed API change + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 80Gi +``` + +For dynamic provisioning and the use of the storageClass, the admin also specifically defines the intent of the volume by +indicating the volumeMode as block. The provisioner for this class will validate whether or not it supports block and return +an error if it does not. + +``` +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: myclaim +spec: + storageClassName: local-fast + volumeMode: Block #proposed API change + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 80Gi +``` + +## Persistent Volume API Changes: +For static provisioning the admin creates the volume and also is intentional about how the volume should be consumed. For backwards +compatibility, the absence of volumeMode will default to filesystem which is how volumes work today, which are formatted with a filesystem depending on the plug-in chosen. Recycling will not be a supported reclaim policy as it has been deprecated. The path value in the local PV definition would be overloaded to define the path of the raw block device rather than the fileystem path. +``` +kind: PersistentVolume +apiVersion: v1 +metadata: + name: local-raw-pv + annotations: + "volume.alpha.kubernetes.io/node-affinity": '{ + "requiredDuringSchedulingIgnoredDuringExecution": { + "nodeSelectorTerms": [ + { "matchExpressions": [ + { "key": "kubernetes.io/hostname", + "operator": "In", + "values": ["ip-172-18-11-174.ec2.internal"] + } + ]} + ]} + }' +spec: + volumeMode: Block + capacity: + storage: 10Gi + local: + path: /dev/xvdf + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain +``` +## Pod API Changes: +This change intentionally calls out the use of a block device (volumeDevices) rather than the mount point on a filesystem. +``` +apiVersion: v1 +kind: Pod +metadata: + name: my-db +spec: + containers: + - name: mysql + image: mysql + volumeDevices: #proposed API change + - name: my-db-data + devicePath: /dev/xvda #proposed API change + volumes: + - name: my-db-data + persistentVolumeClaim: + claimName: raw-pvc +``` +## Storage Class non-API Changes: +For dynamic provisioning, it is assumed that values passed in the parameter section are opaque, thus the introduction of utilizing +fstype in the StorageClass can be used by the provisioner to indicate how to create the volume. The proposal for this value is +defined here: +https://github.com/kubernetes/kubernetes/pull/45345 +This section is provided as a general guideline, but each provisioner may implement their parameters independent of what is defined +here. It is our recommendation that the volumeMode in the PVC be the guidance for the provisioner and overrides the value given in the fstype. Therefore a provisioner should be able to ignore the fstype and provision a block device if that is what the user requested via the PVC and the provisioner can support this. + +``` +kind: StorageClass +apiVersion: storage.k8s.io/v1 +metadata: + name: block-volume + provisioner: kubernetes.io/scaleio + parameters: + gateway: https://192.168.99.200:443/api + system: scaleio + protectionDomain: default + storagePool: default + storageMode: ThinProvisionned + secretRef: sio-secret + readOnly: false +``` +The provisioner (if applicable) should validate the parameters and return an error if the combination specified is not supported. +This also allows the use case for leveraging a Storage Class for utilizing pre-defined static volumes. By labeling the Persistent Volumes +with the Storage Class, volumes can be grouped and used according to how they are defined in the class. +``` +kind: StorageClass +apiVersion: storage.k8s.io/v1 +metadata: + name: block-volume +provisioner: no-provisioning +parameters: +``` + +# Use Cases + +## UC1: + +DESCRIPTION: An admin wishes to pre-create a series of local raw block devices to expose as PVs for consumption. The admin wishes to specify the purpose of these devices by specifying 'block' as the volumeMode for the PVs. + +WORKFLOW: + +ADMIN: + +``` +kind: PersistentVolume +apiVersion: v1 +metadata: + name: local-raw-pv +spec: + volumeMode: Block + capacity: + storage: 100Gi + local: + path: /dev/xvdc + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Delete +``` + +## UC2: + +DESCRIPTION: +* A user uses a raw block device for database applications such as MariaDB. +* User creates a persistent volume claim with "volumeMode: Block" option to bind pre-created iSCSI PV. + +WORKFLOW: + +ADMIN: +* Admin creates a disk and exposes it to all kubelet worker nodes. (This is done by storage operation). +* Admin creates an iSCSI persistent volume using storage information such as portal IP, iqn and lun. + +``` +kind: PersistentVolume +apiVersion: v1 +metadata: + name: raw-pv +spec: + volumeMode: Block + capacity: + storage: 100Gi + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Delete + iscsi: + targetPortal: 1.2.3.4:3260 + iqn: iqn.2017-05.com.example:test + lun: 0 +``` + +USER: + +* User creates a persistent volume claim with volumeMode: Block option to bind pre-created iSCSI PV. + +``` +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: raw-pvc +spec: + volumeMode: Block + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 80Gi +``` + +* User creates a Pod yaml which uses raw-pvc PVC. + +``` +apiVersion: v1 +kind: Pod +metadata: + name: my-db +spec: + containers + - namee: mysql + image: mysql + volumeDevices: + - name: my-db-data + devicePath: /dev/xvda + volumes: + - name: my-db-data + persistentVolumeClaim: + claimName: raw-pvc +``` +* During Pod creation, iSCSI Plugin attaches iSCSI volume to the kubelet worker node using storage information. + + +## UC3: + +DESCRIPTION: + +A developer wishes to enable their application to use a local raw block device as the volume for the container. The admin has already created PVs that the user will bind to by specifying 'block' as the volume type of their PVC. + +BACKGROUND: + +For example, an admin has already created the devices locally and wishes to expose them to the user in a consistent manner through the +Persistent Volume API. + +WORKFLOW: + +USER: + +``` +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: local-raw-pvc +spec: + volumeMode: Block + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 80Gi +``` + +## UC4: + +DESCRIPTION: StorageClass with non-dynamically created volumes + +BACKGROUND: The admin wishes to create a storage class that will identify pre-provisioned block PVs based on a user's PVC request for volumeMode: Block. + +WORKFLOW: + +ADMIN: + +``` +kind: StorageClass +apiVersion: storage.k8s.io/v1 +metadata: + name: block-volume +provisioner: no-provisioning +parameters: +``` +* Sample of pre-created volume definition: + +``` +apiVersion: v1 +kind: PersistentVolume +metadata: + name: pv-block-volume +spec: + volumeMode: Block + storageClassName: block-volume + capacity: + storage: 35Gi + accessModes: + - ReadWriteOnce + local: + path: /dev/xvdc +``` +## [FUTURE] UC5: + +DESCRIPTION: StorageClass with dynamically created volumes + +BACKGROUND: The admin wishes to create a storage class that will dynamically create block PVs based on a user's PVC request for volumeMode: Block. The admin desires the volumes be created dynamically and deleted when the PV definition is deleted. + +WORKFLOW: + +ADMIN: + +``` +kind: StorageClass +apiVersion: storage.k8s.io/v1 +metadata: + name: local-fast +provisioner: kubernetes.io/local-block-ssd +parameters: +``` + +***This has implementation details that have yet to be determined. It is included in this proposal for completeness of design **** + +## UC6: + +DESCRIPTION: The developer wishes to request a block device via a Storage Class. + +WORKFLOW: + +USER: + +``` +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: pvc-local-block +spec: + volumeMode: Block + storageClassName: local-fast + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi +``` + +## UC7: + +DESCRIPTION: Admin creates network raw block devices + +BACKGROUND: Admin wishes to pre-create Persistent Volumes in GCE as raw block devices + +WORKFLOW: + +ADMIN: + +``` +apiVersion: "v1" +kind: "PersistentVolume" +metadata: + name: gce-disk-1 +Spec: + volumeMode: Block + capacity: + storage: "10Gi" + accessModes: + - "ReadWriteOnce" + gcePersistentDisk: + pdName: "gce-disk-1" +``` +**Since the PVC object is passed to the provisioner, it will be responsible for validating and handling whether or not it supports the volumeMode being passed** + +## UC8: + +DESCRIPTION: +* A user uses a raw block device for database applications such as mysql to read data from and write the results to a disk that + has a formatted filesystem to be displayed via nginx web server. + +ADMIN: +* Admin creates a 2 block devices and formats one with a filesystem + +``` +kind: PersistentVolume +apiVersion: v1 +metadata: + name: raw-pv +spec: + volumeMode: Block + capacity: + storage: 100Gi + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Delete + gcePersistentDisk: + pdName: "gce-disk-1" + +``` +``` +kind: PersistentVolume +apiVersion: v1 +metadata: + name: gluster-pv +spec: + volumeMode: Filesystem + capacity: + storage: 100Gi + accessModes: + - ReadWriteMany + persistentVolumeReclaimPolicy: Delete + glusterfs: + endpoints: glusterfs-cluster + path: glusterVol +``` +USER: + +* User creates a persistent volume claim with volumeMode: Block option to bind pre-created block volume. + +``` +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: raw-pvc +spec: + volumeMode: Block + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 80Gi +``` +* User creates a persistent volume claim with volumeMode: Filesystem to the pre-created gluster volume. + +``` +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: gluster-pvc +spec: + volumeMode: Filesystem + accessModes: + - ReadWriteMany + resources: + requests: + storage: 50Gi +``` +* User creates a Pod yaml which will utilize both block and filesystem storage by its containers. + +``` +apiVersion: v1 +kind: Pod +metadata: + name: my-db +spec: + volumes: + - name: my-db-data + persistentVolumeClaim: + claimName: raw-pvc + - name: my-nginx-data + persistentVolumeClaim: + claimName: gluster-pvc + containers + - name: mysql + image: mysql + volumeDevices: + - name: my-db-data + devicePath: /var/lib/mysql/data + - name: nginx + image: nginx + ports: + - containerPort: 80 + volumeMounts: + - mountPath: /usr/share/nginx/html + name: my-nginx-data + readOnly: false +``` +## UC9: + +DESCRIPTION: +* A user wishes to read data from a read-only raw block device, an example might be a database for analytics processing. + +USER: +* User creates pod and specifies 'readOnly' as a parameter in the persistent volume claim to indicate they would +like to be bound to a PV with this setting enabled. + +``` +apiVersion: v1 +kind: Pod +metadata: + name: nginx-pod-block-001 +spec: + containers: + - name: nginx-container + image: nginx:latest + ports: + - containerPort: 80 + volumeDevices: + - name: data + devicePath: /dev/xvda + volumes: + - name: data + persistentVolumeClaim: + claimName: block-pvc001 + readOnly: true #flag indicating read-only for container runtime +``` +**Note: the readOnly field already exists in the PersistentVolumeClaimVolumeSource above and will dictate the values set by the container runtime options** + + +# Container Runtime considerations +It is important the values that are passed to the container runtimes are valid and support the current implementation of these various runtimes. Listed below are a table of various runtime and the mapping of their values to what is passed from the kubelet. + +| runtime engine | runtime options | accessMode | +| -------------- |:----------------:| ----------------:| +| docker/runc/rkt | mknod / RWM | RWO | +| docker/runc/rkt | R | ROX | + +The accessModes would be passed as part of the options array and would need validate against the specific runtime engine. +Since rkt doesn't use the CRI, the config values would need to be passed in the legacy method. +Note: the container runtime doesn't require a privileged pod to enable the device as RWX (RMW), but still requires privileges to mount as is consistent with the filesystem implementation today. + +The runtime option would be placed in the DeviceInfo as such: +devices = append(devices, kubecontainer.DeviceInfo{PathOnHost: path, PathInContainer: path, Permissions: "XXX"}) + +The implementation plan would be to rename the current makeDevices to makeGPUDevices and create a separate function to add the raw block devices to the option array to be passed to the container runtime. This would iterate on the paths passed in for the pod/container. + +Since the future of this in Kubernetes for GPUs and other plug-able devices is migrating to a device plugin architecture, there are +still differentiating components of storage that are enough to not to enforce alignment to their convention. Two factors when +considering the usage of device plugins center around discoverability and topology of devices. Since neither of these are requirements +for using raw block devices, the legacy method of populating the devices and appending it to the device array is sufficient. + + +# Plugin interface changes +## New BlockVolume interface proposed design + +``` +// BlockVolume interface provides methods to generate global map path +// and pod device map path. +type BlockVolume interface { + // GetGlobalMapPath returns a global map path which contains + // symbolic links associated to a block device. + // ex. plugins/kubernetes.io/{PluginName}/{DefaultKubeletVolumeDevicesDirName}/{volumePluginDependentPath}/{pod uuid} + GetGlobalMapPath(spec *Spec) (string, error) + // GetPodDeviceMapPath returns a pod device map path + // and name of a symbolic link associated to a block device. + // ex. pods/{podUid}}/{DefaultKubeletVolumeDevicesDirName}/{escapeQualifiedPluginName}/{volumeName} + GetPodDeviceMapPath() (string, string) +} +``` + +## New BlockVolumePlugin interface proposed design + +``` +// BlockVolumePlugin is an extend interface of VolumePlugin and is used for block volumes support. +type BlockVolumePlugin interface { + VolumePlugin + // NewBlockVolumeMapper creates a new volume.BlockVolumeMapper from an API specification. + // - spec: The v1.Volume spec + // - pod: The enclosing pod + NewBlockVolumeMapper(spec *Spec, podRef *v1.Pod, opts VolumeOptions) (BlockVolumeMapper, error) + // NewBlockVolumeUnmapper creates a new volume.BlockVolumeUnmapper from recoverable state. + // - name: The volume name, as per the v1.Volume spec. + // - podUID: The UID of the enclosing pod + NewBlockVolumeUnmapper(name string, podUID types.UID) (BlockVolumeUnmapper, error) + // ConstructBlockVolumeSpec constructs a volume spec based on the given + // pod name, volume name and a pod device map path. + // The spec may have incomplete information due to limited information + // from input. This function is used by volume manager to reconstruct + // volume spec by reading the volume directories from disk. + ConstructBlockVolumeSpec(podUID types.UID, volumeName, mountPath string) (*Spec, error) +} +``` + +## New BlockVolumeMapper/BlockVolumeUnmapper interface proposed design + +``` +// BlockVolumeMapper interface provides methods to set up/map the volume. +type BlockVolumeMapper interface { + BlockVolume + // SetUpDevice prepares the volume to a self-determined directory path, + // which may or may not exist yet and returns combination of physical + // device path of a block volume and error. + // If the plugin is non-attachable, it should prepare the device + // in /dev/ (or where appropriate) and return unique device path. + // Unique device path across kubelet node reboot is required to avoid + // unexpected block volume destruction. + // If the plugin is attachable, it should not do anything here, + // just return empty string for device path. + // Instead, attachable plugin have to return unique device path + // at attacher.Attach() and attacher.WaitForAttach(). + // This may be called more than once, so implementations must be idempotent. + SetUpDevice() (string, error) +} + +// BlockVolumeUnmapper interface provides methods to cleanup/unmap the volumes. +type BlockVolumeUnmapper interface { + BlockVolume + // TearDownDevice removes traces of the SetUpDevice procedure under + // a self-determined directory. + // If the plugin is non-attachable, this method detaches the volume + // from devicePath on kubelet node. + TearDownDevice(mapPath string, devicePath string) error +} +``` + +## Changes for volume mount points + +Currently, a volume which has filesystem is mounted to the following two paths on a kubelet node when the volumes is in-use. +The purpose of those mount points are that Kubernetes manages volume attach/detach status using these mount points and number +of references to these mount points. + +``` +- Global mount path +/var/lib/kubelet/plugins/kubernetes.io/{pluginName}/{volumePluginDependentPath}/ + +- Volume mount path +/var/lib/kubelet/pods/{podUID}/volumes/{escapeQualifiedPluginName}/{volumeName}/ +``` + +Even if the volumeMode is "Block", similar scheme is needed. However, the volume which +doesn't have filesystem can't be mounted. +Therefore, instead of volume mount, we use symbolic link which is associated to raw block device. +Kubelet creates a new symbolic link under the new `global map path` and `pod device map path`. + +#### Global map path for "Block" volumeMode volume +Kubelet creates a new symbolic link under the new global map path when volume is attached to a Pod. +Number of symbolic links are equal to the number of Pods which use the same volume. Kubelet needs +to manage both creation and deletion of symbolic links under the global map path. The name of the +symbolic link is same as pod uuid. +There are two usages of Global map path. + +1. Manage number of references from multiple pods +1. Retrieve `{volumePluginDependentPath}` during `Block volume reconstruction` + +``` +/var/lib/kubelet/plugins/kubernetes.io/{pluginName}/volumeDevices/{volumePluginDependentPath}/{pod uuid1} +/var/lib/kubelet/plugins/kubernetes.io/{pluginName}/volumeDevices/{volumePluginDependentPath}/{pod uuid2} +... +``` + +- {volumePluginDependentPath} example: +``` +FC plugin: {wwn}-lun-{lun} or {wwid} +ex. /var/lib/kubelet/plugins/kubernetes.io/fc/volumeDevices/500a0982991b8dc5-lun-0/f527ca5b-6d87-11e5-aa7e-080027ff6387 +iSCSI plugin: {portal ip}-{iqn}-lun-{lun} +ex. /var/lib/kubelet/plugins/kubernetes.io/iscsi/volumeDevices/1.2.3.4:3260-iqn.2001-04.com.example:storage.kube.sys1.xyz-lun-1/f527ca5b-6d87-11e5-aa7e-080027ff6387 + ``` + +#### Pod device map path for "Block" volumeMode volume +Kubelet creates a symbolic link under the new pod device map path. The file of {volumeName} is +symbolic link and the link is associated to raw block device. If a Pod has multiple block volumes, +multiple symbolic links under the pod device map path will be created with each volume name. +The usage of pod device map path is; + +1. Retrieve raw block device path(ex. /dev/sdX) during `Container initialization` and `Block volume reconstruction` + +``` +/var/lib/kubelet/pods/{podUID}/volumeDevices/{escapeQualifiedPluginName}/{volumeName1} +/var/lib/kubelet/pods/{podUID}/volumeDevices/{escapeQualifiedPluginName}/{volumeName2} +... +``` + +# Volume binding matrix for statically provisioned volumes: + +| PV volumeMode | PVC volumeMode | Result | +| --------------|:---------------:| ----------------:| +| unspecified | unspecified | BIND | +| unspecified | Block | NO BIND | +| unspecified | Filesystem | BIND | +| Block | unspecified | NO BIND | +| Block | Block | BIND | +| Block | Filesystem | NO BIND | +| Filesystem | Filesystem | BIND | +| Filesystem | Block | NO BIND | +| Filesystem | unspecified | BIND | + + + +* unspecified defaults to 'file/ext4' today for backwards compatibility and in mount_linux.go + + +# Volume binding considerations for dynamically provisioned volumes: +The value used for the plugin to indicate is it provisioning block will be plugin dependent and is an opaque parameter. Binding will also be plugin dependent and must handle the parameter being passed and indicate whether or not it supports block. + +# Implementation Plan, Features & Milestones + +Phase 1: v1.9 +Feature: Pre-provisioned PVs to precreated devices + + Milestone 1: API changes + + Milestone 2: Restricted Access + + Milestone 3: Changes to the mounter interface as today it is assumed 'file' as the default. + + Milestone 4: Expose volumeMode to users via kubectl + + Milestone 5: PV controller binding changes for block devices + + Milestone 6: Container Runtime changes + + Milestone 7: Initial Plugin changes (FC & Local storage) + + Milestone 8: Disabling of provisioning where volumeMode == Block is not supported + +Phase 2: v1.10 +Feature: Discovery of block devices + + Milestone 1: Dynamically provisioned PVs to dynamically allocated devices + + Milestone 2: Privileged container concerns + + Milestone 3: Plugin changes with dynamic provisioning support (GCE, AWS & GlusterFS) + + Milestone 4: Flex volume update diff --git a/contributors/design-proposals/volume-hostpath-qualifiers.md b/contributors/design-proposals/storage/volume-hostpath-qualifiers.md index b207253c..8afcc4e0 100644 --- a/contributors/design-proposals/volume-hostpath-qualifiers.md +++ b/contributors/design-proposals/storage/volume-hostpath-qualifiers.md @@ -144,7 +144,3 @@ I don't think this problem is severe enough that we need to push to solve it; rather I think we can simply accept this minor race, and if runtimes eventually allow this we can begin to leverage them. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/volume-metrics.md b/contributors/design-proposals/storage/volume-metrics.md index 65f3c513..4bea9645 100644 --- a/contributors/design-proposals/volume-metrics.md +++ b/contributors/design-proposals/storage/volume-metrics.md @@ -17,7 +17,7 @@ higher than individual volume plugins. ### Metric format and collection Volume metrics emitted will fall under category of service metrics -as defined in [Kubernetes Monitoring Architecture](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/monitoring_architecture.md). +as defined in [Kubernetes Monitoring Architecture](/contributors/design-proposals/instrumentation/monitoring_architecture.md). The metrics will be emitted using [Prometheus format](https://prometheus.io/docs/instrumenting/exposition_formats/) and available for collection @@ -27,7 +27,7 @@ from `/metrics` HTTP endpoint of kubelet and controller-manager. Any collector which can parse Prometheus metric format should be able to collect metrics from these endpoints. -A more detailed description of monitoring pipeline can be found in [Monitoring architecture](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/monitoring_architecture.md#monitoring-pipeline) document. +A more detailed description of monitoring pipeline can be found in [Monitoring architecture](/contributors/design-proposals/instrumentation/monitoring_architecture.md#monitoring-pipeline) document. ### Metric Types @@ -50,13 +50,13 @@ Following is a sample of metrics (not exhaustive) that will be added by this pro ``` storage_operation_duration_seconds { volume_plugin = "aws-ebs", operation_name = "volume_attach" } storage_operation_duration_seconds { volume_plugin = "aws-ebs", operation_name = "volume_detach" } -storage_operation_duration_seconds { volume_plugin = "glusterfs", operation_name = "provision" } +storage_operation_duration_seconds { volume_plugin = "glusterfs", operation_name = "volume_provision" } storage_operation_duration_seconds { volume_plugin = "gce-pd", operation_name = "volume_delete" } storage_operation_duration_seconds { volume_plugin = "vsphere", operation_name = "volume_mount" } storage_operation_duration_seconds { volume_plugin = "iscsi" , operation_name = "volume_unmount" } -storage_operation_duration_seconds { volume_plugin = "aws-ebs", operation_name = "mount_device" } storage_operation_duration_seconds { volume_plugin = "aws-ebs", operation_name = "unmount_device" } -storage_operation_duration_seconds { volume_plugin = "cinder" , operation_name = "verify_volume" } +storage_operation_duration_seconds { volume_plugin = "cinder" , operation_name = "verify_volumes_are_attached" } +storage_operation_duration_seconds { volume_plugin = "<n/a>" , operation_name = "verify_volumes_are_attached_per_node" } ``` Similarly errors will be named: @@ -64,13 +64,13 @@ Similarly errors will be named: ``` storage_operation_errors_total { volume_plugin = "aws-ebs", operation_name = "volume_attach" } storage_operation_errors_total { volume_plugin = "aws-ebs", operation_name = "volume_detach" } -storage_operation_errors_total { volume_plugin = "glusterfs", operation_name = "provision" } +storage_operation_errors_total { volume_plugin = "glusterfs", operation_name = "volume_provision" } storage_operation_errors_total { volume_plugin = "gce-pd", operation_name = "volume_delete" } storage_operation_errors_total { volume_plugin = "vsphere", operation_name = "volume_mount" } storage_operation_errors_total { volume_plugin = "iscsi" , operation_name = "volume_unmount" } -storage_operation_errors_total { volume_plugin = "aws-ebs", operation_name = "mount_device" } storage_operation_errors_total { volume_plugin = "aws-ebs", operation_name = "unmount_device" } -storage_operation_errors_total { volume_plugin = "cinder" , operation_name = "verify_volume" } +storage_operation_errors_total { volume_plugin = "cinder" , operation_name = "verify_volumes_are_attached" } +storage_operation_errors_total { volume_plugin = "<n/a>" , operation_name = "verify_volumes_are_attached_per_node" } ``` ### Implementation Detail diff --git a/contributors/design-proposals/volume-ownership-management.md b/contributors/design-proposals/storage/volume-ownership-management.md index a9fb1cfe..a9fb1cfe 100644 --- a/contributors/design-proposals/volume-ownership-management.md +++ b/contributors/design-proposals/storage/volume-ownership-management.md diff --git a/contributors/design-proposals/volume-provisioning.md b/contributors/design-proposals/storage/volume-provisioning.md index ff68d280..42d08512 100644 --- a/contributors/design-proposals/volume-provisioning.md +++ b/contributors/design-proposals/storage/volume-provisioning.md @@ -383,7 +383,7 @@ StorageClass and it MUST NOT be copied around the system e.g. in annotations of PVs. See issue #34822. * External provisioners running in pod should have appropriate credentials -mouted as Secret inside pods that run the provisioner. Namespace with the pods +mounted as Secret inside pods that run the provisioner. Namespace with the pods and Secret instance should be well secured. ### `StorageClass` API @@ -494,7 +494,3 @@ parameters: # Cloud Providers Since the `volume.alpha.kubernetes.io/storage-class` is in use a `StorageClass` must be defined to support provisioning. No default is assumed as before. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/volume-selectors.md b/contributors/design-proposals/storage/volume-selectors.md index c1915f99..5af92d0c 100644 --- a/contributors/design-proposals/volume-selectors.md +++ b/contributors/design-proposals/storage/volume-selectors.md @@ -260,9 +260,3 @@ spec: ebs-volume-type: gp-ssd aws-availability-zone: us-east-1 ``` - - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/volume-snapshotting.md b/contributors/design-proposals/storage/volume-snapshotting.md index 0d569618..eeb71b44 100644 --- a/contributors/design-proposals/volume-snapshotting.md +++ b/contributors/design-proposals/storage/volume-snapshotting.md @@ -466,7 +466,7 @@ metadata: name: test-pd spec: containers: - - image: gcr.io/google_containers/test-webserver + - image: k8s.gcr.io/test-webserver name: test-container volumeMounts: - mountPath: /test-pd @@ -486,7 +486,7 @@ metadata: name: test-pd spec: containers: - - image: gcr.io/google_containers/test-webserver + - image: k8s.gcr.io/test-webserver name: test-container volumeMounts: - mountPath: /test-pd @@ -516,8 +516,3 @@ spec: * POST https://www.googleapis.com/compute/v1/projects/example-project/zones/us-central1-f/disks/example-disk/createSnapshot - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/volume-snapshotting.png b/contributors/design-proposals/storage/volume-snapshotting.png Binary files differindex 1b1ea748..1b1ea748 100644 --- a/contributors/design-proposals/volume-snapshotting.png +++ b/contributors/design-proposals/storage/volume-snapshotting.png diff --git a/contributors/design-proposals/storage/volume-topology-scheduling.md b/contributors/design-proposals/storage/volume-topology-scheduling.md new file mode 100644 index 00000000..5885936b --- /dev/null +++ b/contributors/design-proposals/storage/volume-topology-scheduling.md @@ -0,0 +1,661 @@ +# Volume Topology-aware Scheduling + +Authors: @msau42 + +This document presents a detailed design for making the default Kubernetes +scheduler aware of volume topology constraints, and making the +PersistentVolumeClaim (PVC) binding aware of scheduling decisions. + + +## Goals +* Allow a Pod to request one or more topology-constrained Persistent +Volumes (PV) that are compatible with the Pod's other scheduling +constraints, such as resource requirements and affinity/anti-affinity +policies. +* Support arbitrary PV topology constraints (i.e. node, +rack, zone, foo, bar). +* Support topology constraints for statically created PVs and dynamically +provisioned PVs. +* No scheduling latency performance regression for Pods that do not use +topology-constrained PVs. + + +## Non Goals +* Fitting a pod after the initial PVC binding has been completed. + * The more constraints you add to your pod, the less flexible it becomes +in terms of placement. Because of this, tightly constrained storage, such as +local storage, is only recommended for specific use cases, and the pods should +have higher priority in order to preempt lower priority pods from the node. +* Binding decision considering scheduling constraints from two or more pods +sharing the same PVC. + * The scheduler itself only handles one pod at a time. It’s possible the +two pods may not run at the same time either, so there’s no guarantee that you +will know both pod’s requirements at once. + * For two+ pods simultaneously sharing a PVC, this scenario may require an +operator to schedule them together. Another alternative is to merge the two +pods into one. + * For two+ pods non-simultaneously sharing a PVC, this scenario could be +handled by pod priorities and preemption. + + +## Problem +Volumes can have topology constraints that restrict the set of nodes that the +volume can be accessed on. For example, a GCE PD can only be accessed from a +single zone, and a local disk can only be accessed from a single node. In the +future, there could be other topology constraints, such as rack or region. + +A pod that uses such a volume must be scheduled to a node that fits within the +volume’s topology constraints. In addition, a pod can have further constraints +and limitations, such as the pod’s resource requests (cpu, memory, etc), and +pod/node affinity and anti-affinity policies. + +Currently, the process of binding and provisioning volumes are done before a pod +is scheduled. Therefore, it cannot take into account any of the pod’s other +scheduling constraints. This makes it possible for the PV controller to bind a +PVC to a PV or provision a PV with constraints that can make a pod unschedulable. + +### Examples +* In multizone clusters, the PV controller has a hardcoded heuristic to provision +PVCs for StatefulSets spread across zones. If that zone does not have enough +cpu/memory capacity to fit the pod, then the pod is stuck in pending state because +its volume is bound to that zone. +* Local storage exasperates this issue. The chance of a node not having enough +cpu/memory is higher than the chance of a zone not having enough cpu/memory. +* Local storage PVC binding does not have any node spreading logic. So local PV +binding will very likely conflict with any pod anti-affinity policies if there is +more than one local PV on a node. +* A pod may need multiple PVCs. As an example, one PVC can point to a local SSD for +fast data access, and another PVC can point to a local HDD for logging. Since PVC +binding happens without considering if multiple PVCs are related, it is very likely +for the two PVCs to be bound to local disks on different nodes, making the pod +unschedulable. +* For multizone clusters and deployments requesting multiple dynamically provisioned +zonal PVs, each PVC Is provisioned independently, and is likely to provision each PV +In different zones, making the pod unschedulable. + +To solve the issue of initial volume binding and provisioning causing an impossible +pod placement, volume binding and provisioning should be more tightly coupled with +pod scheduling. + + +## Background +In 1.7, we added alpha support for [local PVs](local-storage-pv.md) with node affinity. +You can specify a PV object with node affinity, and if a pod is using such a PV, +the scheduler will evaluate the PV node affinity in addition to the other +scheduling predicates. So far, the PV node affinity only influences pod +scheduling once the PVC is already bound. The initial PVC binding decision was +unchanged. This proposal addresses the initial PVC binding decision. + + +## Design +The design can be broken up into a few areas: +* User-facing API to invoke new behavior +* Integrating PV binding with pod scheduling +* Binding multiple PVCs as a single transaction +* Recovery from kubelet rejection of pod +* Making dynamic provisioning topology-aware + +For the alpha phase, only the user-facing API and PV binding and scheduler +integration are necessary. The remaining areas can be handled in beta and GA +phases. + +### User-facing API +In alpha, this feature is controlled by a feature gate, VolumeScheduling, and +must be configured in the kube-scheduler and kube-controller-manager. + +A new StorageClass field will be added to control the volume binding behavior. + +``` +type StorageClass struct { + ... + + VolumeBindingMode *VolumeBindingMode +} + +type VolumeBindingMode string + +const ( + VolumeBindingImmediate VolumeBindingMode = "Immediate" + VolumeBindingWaitForFirstConsumer VolumeBindingMode = "WaitForFirstConsumer" +) +``` + +`VolumeBindingImmediate` is the default and current binding method. + +This approach allows us to introduce the new binding behavior gradually and to +be able to maintain backwards compatibility without deprecation of previous +behavior. However, it has a few downsides: +* StorageClass will be required to get the new binding behavior, even if dynamic +provisioning is not used (in the case of local storage). +* We have to maintain two different paths for volume binding. +* We will be depending on the storage admin to correctly configure the +StorageClasses for the volume types that need the new binding behavior. +* User experience can be confusing because PVCs could have different binding +behavior depending on the StorageClass configuration. We will mitigate this by +adding a new PVC event to indicate if binding will follow the new behavior. + +### Integrating binding with scheduling +For the alpha phase, the focus is on static provisioning of PVs to support +persistent local storage. + +For the new volume binding mode, the proposed new workflow is: +1. Admin statically creates PVs and/or StorageClasses. +2. User creates unbound PVC and there are no prebound PVs for it. +3. **NEW:** PVC binding and provisioning is delayed until a pod is created that +references it. +4. User creates a pod that uses the PVC. +5. Pod starts to get processed by the scheduler. +6. **NEW:** A new predicate function, called MatchUnboundPVCs, will look at all of +a Pod’s unbound PVCs, and try to find matching PVs for that node based on the +PV topology. If there are no matching PVs, then it checks if dynamic +provisioning is possible for that node. +7. **NEW:** The scheduler continues to evaluate priorities. A new priority +function, called PrioritizeUnboundPVCs, will get the PV matches per PVC per +node, and compute a priority score based on various factors. +8. **NEW:** After evaluating all the existing predicates and priorities, the +scheduler will pick a node, and call a new assume function, AssumePVCs, +passing in the Node. The assume function will check if any binding or +provisioning operations need to be done. If so, it will update the PV cache to +mark the PVs with the chosen PVCs. +9. **NEW:** If PVC binding or provisioning is required, we do NOT AssumePod. +Instead, a new bind function, BindPVCs, will be called asynchronously, passing +in the selected node. The bind function will prebind the PV to the PVC, or +trigger dynamic provisioning. Then, it always sends the Pod through the +scheduler again for reasons explained later. +10. When a Pod makes a successful scheduler pass once all PVCs are bound, the +scheduler assumes and binds the Pod to a Node. +11. Kubelet starts the Pod. + +This diagram depicts the new additions to the default scheduler: + + +This new workflow will have the scheduler handle unbound PVCs by choosing PVs +and prebinding them to the PVCs. The PV controller completes the binding +transaction, handling it as a prebound PV scenario. + +Prebound PVCs and PVs will still immediately be bound by the PV controller. + +Manual recovery by the user will be required in following error conditions: +* A Pod has multiple PVCs, and only a subset of them successfully bind. + +The primary cause for these errors is if a user or external entity +binds a PV between the time that the scheduler chose the PV and when the +scheduler actually made the API update. Some workarounds to +avoid these error conditions are to: +* Prebind the PV instead. +* Separate out volumes that the user prebinds from the volumes that are +available for the system to choose from by StorageClass. + +#### PV Controller Changes +When the feature gate is enabled, the PV controller needs to skip binding +unbound PVCs with VolumBindingWaitForFirstConsumer and no prebound PVs +to let it come through the scheduler path. + +Dynamic provisioning will also be skipped if +VolumBindingWaitForFirstConsumer is set. The scheduler will signal to +the PV controller to start dynamic provisioning by setting the +`annStorageProvisioner` annotation in the PVC. + +No other state machine changes are required. The PV controller continues to +handle the remaining scenarios without any change. + +The methods to find matching PVs for a claim and prebind PVs need to be +refactored for use by the new scheduler functions. + +#### Scheduler Changes + +##### Predicate +A new predicate function checks all of a Pod's unbound PVCs can be satisfied +by existing PVs or dynamically provisioned PVs that are +topologically-constrained to the Node. +``` +MatchUnboundPVCs(pod *v1.Pod, node *v1.Node) (canBeBound bool, err error) +``` +1. If all the Pod’s PVCs are bound, return true. +2. Otherwise try to find matching PVs for all of the unbound PVCs in order of +decreasing requested capacity. +3. Walk through all the PVs. +4. Find best matching PV for the PVC where PV topology is satisfied by the Node. +5. Temporarily cache this PV choice for the PVC per Node, for fast +processing later in the priority and bind functions. +6. Return true if all PVCs are matched. +7. If there are still unmatched PVCs, check if dynamic provisioning is possible. +For this alpha phase, the provisioner is not topology aware, so the predicate +will just return true if there is a provisioner specified in the StorageClass +(internal or external). +8. Otherwise return false. + +##### Priority +After all the predicates run, there is a reduced set of Nodes that can fit a +Pod. A new priority function will rank the remaining nodes based on the +unbound PVCs and their matching PVs. +``` +PrioritizeUnboundPVCs(pod *v1.Pod, filteredNodes HostPriorityList) (rankedNodes HostPriorityList, err error) +``` +1. For each Node, get the cached PV matches for the Pod’s PVCs. +2. Compute a priority score for the Node using the following factors: + 1. How close the PVC’s requested capacity and PV’s capacity are. + 2. Matching static PVs is preferred over dynamic provisioning because we + assume that the administrator has specifically created these PVs for + the Pod. + +TODO (beta): figure out weights and exact calculation + +##### Assume +Once all the predicates and priorities have run, then the scheduler picks a +Node. Then we can bind or provision PVCs for that Node. For better scheduler +performance, we’ll assume that the binding will likely succeed, and update the +PV cache first. Then the actual binding API update will be made +asynchronously, and the scheduler can continue processing other Pods. + +For the alpha phase, the AssumePVCs function will be directly called by the +scheduler. We’ll consider creating a generic scheduler interface in a +subsequent phase. + +``` +AssumePVCs(pod *v1.Pod, node *v1.Node) (pvcBindingRequired bool, err error) +``` +1. If all the Pod’s PVCs are bound, return false. +2. For static PV binding: + 1. Get the cached matching PVs for the PVCs on that Node. + 2. Validate the actual PV state. + 3. Mark PV.ClaimRef in the PV cache. + 4. Cache the PVs that need binding in the Pod object. +3. For in-tree and external dynamic provisioning: + 1. Cache the PVCs that need provisioning in the Pod object. +4. Return true. + +##### Bind +If AssumePVCs returns pvcBindingRequired, then the BindPVCs function is called +as a go routine. Otherwise, we can continue with assuming and binding the Pod +to the Node. + +For the alpha phase, the BindUnboundPVCs function will be directly called by the +scheduler. We’ll consider creating a generic scheduler interface in a subsequent +phase. + +``` +BindUnboundPVCs(pod *v1.Pod, node *v1.Node) (err error) +``` +1. For static PV binding: + 1. Prebind the PV by updating the `PersistentVolume.ClaimRef` field. + 2. If the prebind fails, revert the cache updates. +2. For in-tree and external dynamic provisioning: + 1. Set `annStorageProvisioner` on the PVC. +3. Send Pod back through scheduling, regardless of success or failure. + 1. In the case of success, we need one more pass through the scheduler in +order to evaluate other volume predicates that require the PVC to be bound, as +described below. + 2. In the case of failure, we want to retry binding/provisioning. + +TODO: pv controller has a high resync frequency, do we need something similar +for the scheduler too + +##### Access Control +Scheduler will need PV update permissions for prebinding static PVs, and PVC +modify permissions for triggering dynamic provisioning. + +##### Pod preemption considerations +The MatchUnboundPVs predicate does not need to be re-evaluated for pod +preemption. Preempting a pod that uses a PV will not free up capacity on that +node because the PV lifecycle is independent of the Pod’s lifecycle. + +##### Other scheduler predicates +Currently, there are a few existing scheduler predicates that require the PVC +to be bound. The bound assumption needs to be changed in order to work with +this new workflow. + +TODO: how to handle race condition of PVCs becoming bound in the middle of +running predicates? One possible way is to mark at the beginning of scheduling +a Pod if all PVCs were bound. Then we can check if a second scheduler pass is +needed. + +###### Max PD Volume Count Predicate +This predicate checks the maximum number of PDs per node is not exceeded. It +needs to be integrated into the binding decision so that we don’t bind or +provision a PV if it’s going to cause the node to exceed the max PD limit. But +until it is integrated, we need to make one more pass in the scheduler after all +the PVCs are bound. The current copy of the predicate in the default scheduler +has to remain to account for the already-bound volumes. + +###### Volume Zone Predicate +This predicate makes sure that the zone label on a PV matches the zone label of +the node. If the volume is not bound, this predicate can be ignored, as the +binding logic will take into account zone constraints on the PV. + +However, this assumes that zonal PVs like GCE PDs and AWS EBS have been updated +to use the new PV topology specification, which is not the case as of 1.8. So +until those plugins are updated, the binding and provisioning decisions will be +topology-unaware, and we need to make one more pass in the scheduler after all +the PVCs are bound. + +This predicate needs to remain in the default scheduler to handle the +already-bound volumes using the old zonal labeling. It can be removed once that +mechanism is deprecated and unsupported. + +###### Volume Node Predicate +This is a new predicate added in 1.7 to handle the new PV node affinity. It +evaluates the node affinity against the node’s labels to determine if the pod +can be scheduled on that node. If the volume is not bound, this predicate can +be ignored, as the binding logic will take into account the PV node affinity. + +##### Caching +There are two new caches needed in the scheduler. + +The first cache is for handling the PV/PVC API binding updates occurring +asynchronously with the main scheduler loop. `AssumePVCs` needs to store +the updated API objects before `BindUnboundPVCs` makes the API update, so +that future binding decisions will not choose any assumed PVs. In addition, +if the API update fails, the cached updates need to be reverted and restored +with the actual API object. The cache will return either the cached-only +object, or the informer object, whichever one is latest. Informer updates +will always override the cached-only object. The new predicate and priority +functions must get the objects from this cache intead of from the informer cache. +This cache only stores pointers to objects and most of the time will only +point to the informer object, so the memory footprint per object is small. + +The second cache is for storing temporary state as the Pod goes from +predicates to priorities and then assume. This all happens serially, so +the cache can be cleared at the beginning of each pod scheduling loop. This +cache is used for: +* Indicating if all the PVCs are already bound at the beginning of the pod +scheduling loop. This is to handle situations where volumes may have become +bound in the middle of processing the predicates. We need to ensure that +all the volume predicates are fully run once all PVCs are bound. +* Caching PV matches per node decisions that the predicate had made. This is +an optimization to avoid walking through all the PVs again in priority and +assume functions. + +#### Performance and Optimizations +Let: +* N = number of nodes +* V = number of all PVs +* C = number of claims in a pod + +C is expected to be very small (< 5) so shouldn’t factor in. + +The current PV binding mechanism just walks through all the PVs once, so its +running time O(V). + +Without any optimizations, the new PV binding mechanism has to run through all +PVs for every node, so its running time is O(NV). + +A few optimizations can be made to improve the performance: + +1. Optimizing for PVs that don’t use node affinity (to prevent performance +regression): + 1. Index the PVs by StorageClass and only search the PV list with matching +StorageClass. + 2. Keep temporary state in the PVC cache if we previously succeeded or +failed to match PVs, and if none of the PVs have node affinity. Then we can +skip PV matching on subsequent nodes, and just return the result of the first +attempt. +2. Optimizing for PVs that have node affinity: + 1. When a static PV is created, if node affinity is present, evaluate it +against all the nodes. For each node, keep an in-memory map of all its PVs +keyed by StorageClass. When finding matching PVs for a particular node, try to +match against the PVs in the node’s PV map instead of the cluster-wide PV list. + +For the alpha phase, the optimizations are not required. However, they should +be required for beta and GA. + +#### Packaging +The new bind logic that is invoked by the scheduler can be packaged in a few +ways: +* As a library to be directly called in the default scheduler +* As a scheduler extender + +We propose taking the library approach, as this method is simplest to release +and deploy. Some downsides are: +* The binding logic will be executed using two different caches, one in the +scheduler process, and one in the PV controller process. There is the potential +for more race conditions due to the caches being out of sync. +* Refactoring the binding logic into a common library is more challenging +because the scheduler’s cache and PV controller’s cache have different interfaces +and private methods. + +##### Extender cons +However, the cons of the extender approach outweighs the cons of the library +approach. + +With an extender approach, the PV controller could implement the scheduler +extender HTTP endpoint, and the advantage is the binding logic triggered by the +scheduler can share the same caches and state as the PV controller. + +However, deployment of this scheduler extender in a master HA configuration is +extremely complex. The scheduler has to be configured with the hostname or IP of +the PV controller. In a HA setup, the active scheduler and active PV controller +could run on the same, or different node, and the node can change at any time. +Exporting a network endpoint in the controller manager process is unprecedented +and there would be many additional features required, such as adding a mechanism +to get a stable network name, adding authorization and access control, and +dealing with DDOS attacks and other potential security issues. Adding to those +challenges is the fact that there are countless ways for users to deploy +Kubernetes. + +With all this complexity, the library approach is the most feasible in a single +release time frame, and aligns better with the current Kubernetes architecture. + +#### Downsides + +##### Unsupported Use Cases +The following use cases will not be supported for PVCs with a StorageClass with +VolumeBindingWaitForFirstConsumer: +* Directly setting Pod.Spec.NodeName +* DaemonSets + +These two use cases will bypass the default scheduler and thus will not +trigger PV binding. + +##### Custom Schedulers +Custom schedulers, controllers and operators that handle pod scheduling and want +to support this new volume binding mode will also need to handle the volume +binding decision. + +There are a few ways to take advantage of this feature: +* Custom schedulers could be implemented through the scheduler extender +interface. This allows the default scheduler to be run in addition to the +custom scheduling logic. +* The new code for this implementation will be packaged as a library to make it +easier for custom schedulers to include in their own implementation. + +In general, many advanced scheduling features have been added into the default +scheduler, such that it is becoming more difficult to run without it. + +##### HA Master Upgrades +HA masters adds a bit of complexity to this design because the active scheduler +process and active controller-manager (PV controller) process can be on different +nodes. That means during an HA master upgrade, the scheduler and controller-manager +can be on different versions. + +The scenario where the scheduler is newer than the PV controller is fine. PV +binding will not be delayed and in successful scenarios, all PVCs will be bound +before coming to the scheduler. + +However, if the PV controller is newer than the scheduler, then PV binding will +be delayed, and the scheduler does not have the logic to choose and prebind PVs. +That will cause PVCs to remain unbound and the Pod will remain unschedulable. + +TODO: One way to solve this is to have some new mechanism to feature gate system +components based on versions. That way, the new feature is not turned on until +all dependencies are at the required versions. + +For alpha, this is not concerning, but it needs to be solved by GA. + +#### Other Alternatives Considered + +##### One scheduler function +An alternative design considered was to do the predicate, priority and bind +functions all in one function at the end right before Pod binding, in order to +reduce the number of passes we have to make over all the PVs. However, this +design does not work well with pod preemption. Pod preemption needs to be able +to evaluate if evicting a lower priority Pod will make a higher priority Pod +schedulable, and it does this by re-evaluating predicates without the lower +priority Pod. + +If we had put the MatchUnboundPVCs predicate at the end, then pod preemption +wouldn’t have an accurate filtered nodes list, and could end up preempting pods +on a Node that the higher priority pod still cannot run on due to PVC +requirements. For that reason, the PVC binding decision needs to be have its +predicate function separated out and evaluated with the rest of the predicates. + +##### Pull entire PVC binding into the scheduler +The proposed design only has the scheduler initiating the binding transaction +by prebinding the PV. An alternative is to pull the whole two-way binding +transaction into the scheduler, but there are some complex scenarios that +scheduler’s Pod sync loop cannot handle: +* PVC and PV getting unexpectedly unbound or lost +* PVC and PV state getting partially updated +* PVC and PV deletion and cleanup + +Handling these scenarios in the scheduler’s Pod sync loop is not possible, so +they have to remain in the PV controller. + +##### Keep all PVC binding in the PV controller +Instead of initiating PV binding in the scheduler, have the PV controller wait +until the Pod has been scheduled to a Node, and then try to bind based on the +chosen Node. A new scheduling predicate is still needed to filter and match +the PVs (but not actually bind). + +The advantages are: +* Existing scenarios where scheduler is bypassed will work. +* Custom schedulers will continue to work without any changes. +* Most of the PV logic is still contained in the PV controller, simplifying HA +upgrades. + +Major downsides of this approach include: +* Requires PV controller to watch Pods and potentially change its sync loop +to operate on pods, in order to handle the multiple PVCs in a pod scenario. +This is a potentially big change that would be hard to keep separate and +feature-gated from the current PV logic. +* Both scheduler and PV controller processes have to make the binding decision, +but because they are done asynchronously, it is possible for them to choose +different PVs. The scheduler has to cache its decision so that it won't choose +the same PV for another PVC. But by the time PV controller handles that PVC, +it could choose a different PV than the scheduler. + * Recovering from this inconsistent decision and syncing the two caches is +very difficult. The scheduler could have made a cascading sequence of decisions +based on the first inconsistent decision, and they would all have to somehow be +fixed based on the real PVC/PV state. +* If the scheduler process restarts, it loses all its in-memory PV decisions and +can make a lot of wrong decisions after the restart. +* All the volume scheduler predicates that require PVC to be bound will not get +evaluated. To solve this, all the volume predicates need to also be built into +the PV controller when matching possible PVs. + +##### Move PVC binding to kubelet +Looking into the future, with the potential for NUMA-aware scheduling, you could +have a sub-scheduler on each node to handle the pod scheduling within a node. It +could make sense to have the volume binding as part of this sub-scheduler, to make +sure that the volume selected will have NUMA affinity with the rest of the +resources that the pod requested. + +However, there are potential security concerns because kubelet would need to see +unbound PVs in order to bind them. For local storage, the PVs could be restricted +to just that node, but for zonal storage, it could see all the PVs in that zone. + +In addition, the sub-scheduler is just a thought at this point, and there are no +concrete proposals in this area yet. + +### Binding multiple PVCs in one transaction +There are no plans to handle this, but a possible solution is presented here if the +need arises in the future. Since the scheduler is serialized, a partial binding +failure should be a rare occurrence and would only be caused if there is a user or +other external entity also trying to bind the same volumes. + +One possible approach to handle this is to rollback previously bound PVCs on +error. However, volume binding cannot be blindly rolled back because there could +be user's data on the volumes. + +For rollback, PersistentVolumeClaims will have a new status to indicate if it's +clean or dirty. For backwards compatibility, a nil value is defaulted to dirty. +The PV controller will set the status to clean if the PV is Available and unbound. +Kubelet will set the PV status to dirty during Pod admission, before adding the +volume to the desired state. + +If scheduling fails, update all bound PVCs with an annotation, +"pv.kubernetes.io/rollback". The PV controller will only unbind PVCs that +are clean. Scheduler and kubelet needs to reject pods with PVCs that are +undergoing rollback. + +### Recovering from kubelet rejection of pod +We can use the same rollback mechanism as above to handle this case. +If kubelet rejects a pod, it will go back to scheduling. If the scheduler +cannot find a node for the pod, then it will encounter scheduling failure and +initiate the rollback. + +### Making dynamic provisioning topology aware +TODO (beta): Design details + +For alpha, we are not focusing on this use case. But it should be able to +follow the new workflow closely with some modifications. +* The FindUnboundPVCs predicate function needs to get provisionable capacity per +topology dimension from the provisioner somehow. +* The PrioritizeUnboundPVCs priority function can add a new priority score factor +based on available capacity per node. +* The BindUnboundPVCs bind function needs to pass in the node to the provisioner. +The internal and external provisioning APIs need to be updated to take in a node +parameter. + + +## Testing + +### E2E tests +* StatefulSet, replicas=3, specifying pod anti-affinity + * Positive: Local PVs on each of the nodes + * Negative: Local PVs only on 2 out of the 3 nodes +* StatefulSet specifying pod affinity + * Positive: Multiple local PVs on a node + * Negative: Only one local PV available per node +* Multiple PVCs specified in a pod + * Positive: Enough local PVs available on a single node + * Negative: Not enough local PVs available on a single node +* Fallback to dynamic provisioning if unsuitable static PVs + +### Unit tests +* All PVCs found a match on first node. Verify match is best suited based on +capacity. +* All PVCs found a match on second node. Verify match is best suited based on +capacity. +* Only 2 out of 3 PVCs have a match. +* Priority scoring doesn’t change the given priorityList order. +* Priority scoring changes the priorityList order. +* Don’t match PVs that are prebound + + +## Implementation Plan + +### Alpha +* New feature gate for volume topology scheduling +* StorageClass API change +* Refactor PV controller methods into a common library +* PV controller: Delay binding and provisioning unbound PVCs +* Predicate: Filter nodes and find matching PVs +* Predicate: Check if provisioner exists for dynamic provisioning +* Update existing predicates to skip unbound PVC +* Bind: Trigger PV binding +* Bind: Trigger dynamic provisioning +a Pod (only if alpha is enabled) + +### Beta +* Scheduler cache: Optimizations for no PV node affinity +* Priority: capacity match score +* Plugins: Convert all zonal volume plugins to use new PV node affinity (GCE PD, +AWS EBS, what else?) +* Make dynamic provisioning topology aware + +### GA +* Predicate: Handle max PD per node limit +* Scheduler cache: Optimizations for PV node affinity + + +## Open Issues +* Can generic device resource API be leveraged at all? Probably not, because: + * It will only work for local storage (node specific devices), and not zonal +storage. + * Storage already has its own first class resources in K8s (PVC/PV) with an +independent lifecycle. The current resource API proposal does not have an a way to +specify identity/persistence for devices. +* Will this be able to work with the node sub-scheduler design for NUMA-aware +scheduling? + * It’s still in a very early discussion phase. diff --git a/contributors/design-proposals/storage/volume-topology-scheduling.png b/contributors/design-proposals/storage/volume-topology-scheduling.png Binary files differnew file mode 100644 index 00000000..b952ff39 --- /dev/null +++ b/contributors/design-proposals/storage/volume-topology-scheduling.png diff --git a/contributors/design-proposals/volumes.md b/contributors/design-proposals/storage/volumes.md index 874dc2af..a963b279 100644 --- a/contributors/design-proposals/volumes.md +++ b/contributors/design-proposals/storage/volumes.md @@ -476,7 +476,3 @@ spec: The cluster operator would need to manually `chgrp` and `chmod` the `/tmp/example-pod` on the host in order for the volume to be usable from the pod. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/design-proposals/testing/OWNERS b/contributors/design-proposals/testing/OWNERS new file mode 100644 index 00000000..48c9f03c --- /dev/null +++ b/contributors/design-proposals/testing/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-testing-leads +approvers: + - sig-testing-leads +labels: + - sig/testing diff --git a/contributors/design-proposals/flakiness-sla.md b/contributors/design-proposals/testing/flakiness-sla.md index 0b473d73..b76be2f0 100644 --- a/contributors/design-proposals/flakiness-sla.md +++ b/contributors/design-proposals/testing/flakiness-sla.md @@ -67,7 +67,7 @@ may chose to update our flakiness SLA. ### Monitoring and enforcement infrastructure SIG-Testing are currently responsible for the submit-queue infrastructure -and will be responsibile for designing, implementing and deploying the +and will be responsible for designing, implementing and deploying the relevant monitoring and enforcement mechanisms for this proposal. ### Assessing success diff --git a/contributors/design-proposals/vault-based-kms-class-diagram.png b/contributors/design-proposals/vault-based-kms-class-diagram.png Binary files differnew file mode 100644 index 00000000..13d57884 --- /dev/null +++ b/contributors/design-proposals/vault-based-kms-class-diagram.png diff --git a/contributors/design-proposals/vault-based-kms-provider.md b/contributors/design-proposals/vault-based-kms-provider.md new file mode 100644 index 00000000..097e2f13 --- /dev/null +++ b/contributors/design-proposals/vault-based-kms-provider.md @@ -0,0 +1,275 @@ +# Vault based KMS provider for envelope encryption of secrets in etcd3 + +## Abstract + +Kubernetes, starting with the release 1.7, adds Alpha support ( via PRs +[41939](https://github.com/kubernetes/kubernetes/pull/41939) and +[46460](https://github.com/kubernetes/kubernetes/pull/46460)) to encrypt secrets +and resources in etcd3 via a configured Provider. This release supports three +providers viz. aesgcm, aescbc, secretbox. These providers store the encryption +key(s) locally in a server configuration file. The provider encrypts and +decrypts secrets in-process. Building upon these, a KMS provider framework with +an option to support different KMS providers like google cloud KMS is being +added via PRs [48574](https://github.com/kubernetes/kubernetes/pull/48575) and +[49350](https://github.com/kubernetes/kubernetes/pull/49350). The new KMS +provider framework uses an envelope encryption scheme. + + +This proposal adopts the KMS provider framework and adds a new KMS provider that +uses Hashicorp Vault with a transit backend, to encrypt and decrypt the DEK +stored in encrypted form in etcd3 along with encrypted secrets. + + +Vault is widely used for Data encryption and securely storing secrets. +Externalizing encryption/decryption of kubernetes secrets to vault provides +various benefits + +* Choice of industry standard encryption algorithms and strengths without having +to implement specific providers for each (in K8S). +* Reduced risk of encryption key compromise. + * encryption key is stored and managed in Vault. + * encryption key does not need to leave the Vault. +* Vault provides ability to define access control suitable for a wide range of deployment scenarios and security needs. +* Vault provides In-built auditing of vault API calls. +* Ability for a customer already using Vault to leverage the instance to also +secure keys used to encrypt secrets managed within a Kubernetes cluster +* Separation of Kubernetes cluster management responsibilities from encryption key +management and administration allowing an organization to better leverage +competencies and skills within the DevOps teams. + +Note, that the Vault Provider in this proposal + +1. **requires** Vault transit backend. +2. supports a wide range of authentication backends supported by vault (see below +for exact list). +3. does not depend on specific storage backend or any other specific configuration. + +This proposal assumes familiarity with Vault and the transit back-end. + +## High level design +As with existing providers, the Vault based provider will implement the +interface ``envelope.Service``. Based on value of *name* in the KMS provider +configuration, the ``EnvelopeTransformer`` module will use an instance of the +Vault provider for decryption and encryption of DEK before storing and after +reading from the storage. + +The KEK will be stored and managed in Vault backend. The Vault based provider +configured in KMS Transformer configuration will make REST requests to encrypt +and decrypt DEKs over a secure channel (must enable TLS). KMS Transformer will +store the DEKs in etcd in encrypted form along with encrypted secrets. As with +existing providers, encrypted DEKs will be stored with metadata used to identify +the provider and KEK to be used for decryption. + +The provider will support following authentication back-ends + +* Vault token based, +* TLS cert based, +* Vault AppRole based. + +Deployers can choose an authentication mechanism best suited to their +requirements. +The provider will work with vault REST APIs and will not require Vault to be +configured or deployed in any specific way other than requiring a Transit +Backend. + +### Diagram illustrating interfaces and implementations + + + +### Pseudocode +#### Prefix Metadata +Every encrypted secret will have the following metadata prefixed. +```json +k8s:enc:kms:<api-version>:vault:len(<KEK-key-name>:<KEK-key-version>:<DEK +encrypted with KEK>):<KEK-key-name>:<KEK-key-version>:<DEK encrypted with KEK> +``` + +* ``<api-version>`` represents api version in the providers configuration file. +* ``vault`` represents the KMS service *kind* value. It is a fixed value for Vault +based provider. +* ``KEK-key-name`` is determined from the vault service configuration in providers +configuration file +* ``KEK-key-version`` is an internal identifier used by vault to identify specific +key version used to encrypt and decrypt. Vault sends ``kek-key-version`` +prefixed with encrypted data in the response to an encrypt request. The +``kek-key-version`` will be stored as part of prefix and returned back to Vault +during a decrypt request. + +Of the above metadata, + +* ``EnvelopeTransformer`` will add +``k8s:enc:kms:<api-version>:vault:len(<KEK-key-name>:<KEK-key-version>:<DEK +encrypted with KEK>)`` +* while the ``vaultEnvelopeService`` will add +``<KEK-key-name>:<KEK-key-version>:<DEK encrypted with KEK>``. + + +#### For each write of DEK +``EnvelopeTransformer`` will write encrypted DEK along with encrypted secret in +etcd. + +Here's the pseudocode for ``vaultEnvelopeService.encrypt()``, invoked on each +write of DEK. + + KEY_NAME = <first key-name from vault provider config> + PLAIN_DEK = <value of DEK> + ENCRYPTED_DEK_WITH_KEY_VERSION = encrypt(base64(PLAIN_DEK), KEY_NAME) + + // output from vault will have an extra prefix "vault" (other than key version) which will be stripped. + + STORED_DEK = KEY_NAME:<ENCRYPTED_DEK_WITH_KEY_VERSION> + +#### For each read of DEK +``EnvelopeTransformer`` will read encrypted DEK along with encrypted secret from +etcd + +Here's the pseudocode ``vaultEnvelopeService.decrypt()`` invoked on each read of +DEK. + + // parse the provider kind, key name and encrypted DEK prefixed with key version + KEY_NAME = //key-name from the prefix + ENCRYPTED_DEK_WITH_KEY_VERSION = //<key version>:<encrypted DEK> from the stored value + + // add "vault" prefix to ENCRYPTED_DEK_WITH_KEY_VERSION as required by vault decrypt API + + base64Encoded = decrypt(vault:ENCRYPTED_DEK_WITH_KEY_VERSION, KEY_NAME) + + PLAIN_DEK = base64.Decode(base64Encoded) + +#### Example + + DEK = "the quick brown fox" + provider kind = "vault" + api version version = "v1" + Key name = "kube-secret-enc-key" + key version = v1 + ciphertext returned from vault = vault:v1:aNOTZn0aUDMDbWAQL1E31tH/7zr7oslRjkSpRW0+BPdMfSJntyXZNCAwIbkTtn0= + prefixed DEK used to tag secrets = vault:kube-secret-enc-key:v1:aNOTZn0aUDMDbWAQL1E31tH/7zr7oslRjkSpRW0+BPdMfSJntyXZNCAwIbkTtn0= + +### Configuration + +No new configuration file or startup parameter will be introduced. + +The vault provider will be specified in the existing configuration file used to +configure any of the encryption providers. The location of this configuration +file is identified by the existing startup parameter: +`--experimental-encryption-provider-config` . + +Vault provider configuration will be identified by value "**vault**" for the +``name`` attribute in ``kms`` provider. + +The actual configuration of the vault provider will be in a separate +configuration identified by the ``configfile`` attribute in the KMS provider. + +Here is a sample configuration file with the vault provider configured: + + kind: EncryptionConfig + apiVersion: v1 + resources: + - resources: + - secrets + providers: + - kms: + name: vault + cachesize: 10 + configfile: /home/myvault/vault-config.yaml + +#### Minimal required Configuration +The Vault based Provider needs the following configuration elements, at a +minimum: + +1. ``addr`` Vault service base endpoint eg. https://example.com:8200 +2. ``key-names`` list of names of the keys in Vault to be used. eg: key-name: +kube-secret-enc-key. + +Note : key name does not need to be changed if the key is rotated in Vault, the +rotated key is identified by key version which is prefix to ciphertext. + +A new key can be added in the list. Encryption will be done using the first key +in the list. Decryption can happen using any of the keys in the list based on +the prefix to the encrypted DEK stored in etcd + +#### Authentication Configuration +##### Vault Server Authentication + +For the Kubernetes cluster to authenticate the vault server, TLS must be enabled : +1. ``ca-cert`` location of x509 certificate to authenticate the vault server eg: +``/var/run/kubernetes/ssl/vault.crt`` + +##### Client Authentication Choices + +For client authentication, one of following **must** be used: (provider will +reject the configuration if parameters for more than one authentication backends +are specified ) + +###### X509 based authentication +1. ``client-cert``: location of x509 certificate to authenticate kubernetes API +server to vault server eg. ``/var/run/kubernetes/ssl/valut-client-cert.pem`` +2. ``client-key`` : location of x509 private key to authenticate kubernetes API +server to vault server eg. ``/var/run/kubernetes/ssl/vault-client-key.pem`` + +Here's a sample ``vault-config.yaml`` configuration with ``client-cert``: +``` + key-names: + - kube-secret-enc-key + addr: https://example.com:8200 + ca-cert:/var/run/kubernetes/ssl/vault.crt + client-cert:/var/run/kubernetes/ssl/vault-client-cert.pem + client-key:/var/run/kubernetes/ssl/vault-client-key.pem +``` + +###### Vault token based authentication +1. ``token`` : limited access vault token required by kubernetes API sever to +authenticate itself while making requests to vault eg: +8dad1053-4a4e-f359-2eab-d57968eb277f + +Here's a sample ``vault-config.yaml`` configuration using a Vault Token for authentication. +the Kubernetes cluster as a client to Vault: +``` + key-names: + -kube-secret-enc-key + addr: https://example.com:8200 + ca-cert:/var/run/kubernetes/ssl/vault.crt + token: 8dad1053-4a4e-f359-2eab-d57968eb277f +``` + +###### Vault AppRole based authentication +1. ``role-id`` : RoleID of the AppRole +2. ``secret-id`` : secret Id only if associated with the appRole. + +Here's a sample configuration file using a Vault AppRole for authentication. +``` + key-names: + - kube-secret-enc-key + addr: https://localhost:8200 + ca-cert: /var/run/kubernetes/ssl/vault.crt + role-id: db02de05-fa39-4855-059b-67221c5c2f63 +``` + +## Key Generation and rotation +The KEK is generated in Vault and rotated using direct API call or CLI to Vault +itself. The Key never leaves the vault. + +Note that when a key is rotated, Vault does not allow choosing a different +encryption algorithm or key size. If a key for different encryption algorithm or +a different key size is desired, new key needs to be generated in Vault and the +corresponding key name be added in the configuration. Subsequent encryption will +be done using the first key in the list. Decryption can happen using any of the +keys in the list based on the prefix to the encrypted DEK. + +## Backward compatibility +1. Unencrypted secrets and secrets encrypted using other non-KMS providers will +continue to be readable upon adding vault as a new KMS provider. +2. If a Vault KMS is added as first provider, the secrets created or modified +thereafter will be encrypted by vault provider. + +## Performance +1. KMS provider framework uses LRU cache to minimize the requests to KMS for +encryption and decryption of DEKs. +2. Note that there will be a request to KMS for every cache miss causing a +performance impact. Hence, depending on the cache size, there will be a +performance impact. +3. Response time. + 4. will depend on choice of encryption algorithm and strength. + 5. will depend on specific vault configurations like storage backend, +authentication mechanism, token polices etc. diff --git a/contributors/devel/README.md b/contributors/devel/README.md index 5e5490ee..2c416296 100644 --- a/contributors/devel/README.md +++ b/contributors/devel/README.md @@ -8,20 +8,13 @@ Guide](http://kubernetes.io/docs/admin/). ## The process of developing and contributing code to the Kubernetes project -* **Welcome to Kubernetes (New Developer Guide)** - ([welcome-to-kubernetes-new-developer-guide.md](welcome-to-kubernetes-new-developer-guide.md)): - An introductory guide to contributing to K8s. - -* **On Collaborative Development** ([collab.md](collab.md)): Info on pull requests and code reviews. +* **Contributor Guide** + ([Please start here](/contributors/guide/README.md)) to learn about how to contribute to Kubernetes * **GitHub Issues** ([issues.md](issues.md)): How incoming issues are triaged. * **Pull Request Process** ([pull-requests.md](pull-requests.md)): When and why pull requests are closed. -* **Kubernetes On-Call Rotations** ([on-call-rotations.md](on-call-rotations.md)): Descriptions of on-call rotations for build and end-user support. - -* **Faster PR reviews** ([faster_reviews.md](faster_reviews.md)): How to get faster PR reviews. - * **Getting Recent Builds** ([getting-builds.md](getting-builds.md)): How to get recent builds including the latest builds that pass CI. * **Automated Tools** ([automation.md](automation.md)): Descriptions of the automation that is running on our github repository. @@ -73,17 +66,13 @@ Guide](http://kubernetes.io/docs/admin/). * **Authentication** ([Authentication](http://kubernetes.io/docs/admin/authentication/)): The current and planned states of authentication tokens. -* **Authorization Plugins** ([Authorization](http://kubernetes.github.io/docs/admin/authorization/)): +* **Authorization Plugins** ([Authorization](http://kubernetes.io/docs/admin/authorization/)): Authorization applies to all HTTP requests on the main apiserver port. This doc explains the available authorization implementations. -* **Admission Control Plugins** ([admission_control](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control.md)) +* **Admission Control Plugins** ([admission_control](/contributors/design-proposals/api-machinery/admission_control.md)) ## Building releases See the [kubernetes/release](https://github.com/kubernetes/release) repository for details on creating releases and related tools and helper scripts. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/adding-an-APIGroup.md b/contributors/devel/adding-an-APIGroup.md index fee831ed..8bbc2e93 100644 --- a/contributors/devel/adding-an-APIGroup.md +++ b/contributors/devel/adding-an-APIGroup.md @@ -2,8 +2,3 @@ Adding an API Group =============== Please refer to [api_changes.md](api_changes.md#making-a-new-api-group). - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/api-conventions.md b/contributors/devel/api-conventions.md index e2c7bab6..2ac93bf7 100644 --- a/contributors/devel/api-conventions.md +++ b/contributors/devel/api-conventions.md @@ -8,7 +8,7 @@ Kubernetes API structure, and developers wanting to extend the Kubernetes API. An introduction to using resources with kubectl can be found in [the object management overview](https://kubernetes.io/docs/tutorials/object-management-kubectl/object-management/).* **Table of Contents** -<!-- BEGIN MUNGE: GENERATED_TOC --> + - [Types (Kinds)](#types-kinds) - [Resources](#resources) @@ -45,7 +45,6 @@ An introduction to using resources with kubectl can be found in [the object mana - [WebSockets and SPDY](#websockets-and-spdy) - [Validation](#validation) -<!-- END MUNGE: GENERATED_TOC --> The conventions of the [Kubernetes API](https://kubernetes.io/docs/api/) (and related APIs in the ecosystem) are intended to ease client development and ensure that configuration @@ -225,9 +224,12 @@ an object was created after which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource will be deleted (no longer visible from resource lists, and -not reachable by name) after the time in this field. Once set, this value may -not be unset or be set further into the future, although it may be shortened or -the resource may be deleted prior to this time. +not reachable by name) after the time in this field except when the object has +a finalizer set. In case the finalizer is set the deletion of the object is +postponed at least until the finalizer is removed. +Once the deletionTimestamp is set, this value may not be unset or be set further +into the future, although it may be shortened or the resource may be deleted +prior to this time. * labels: a map of string keys and values that can be used to organize and categorize objects (see [the labels docs](https://kubernetes.io/docs/user-guide/labels/)) * annotations: a map of string keys and values that can be used by external @@ -350,7 +352,7 @@ Some resources in the v1 API contain fields called **`phase`**, and associated `message`, `reason`, and other status fields. The pattern of using `phase` is deprecated. Newer API types should use conditions instead. Phase was essentially a state-machine enumeration field, that contradicted -[system-design principles](../design-proposals/principles.md#control-logic) and hampered +[system-design principles](../design-proposals/architecture/principles.md#control-logic) and hampered evolution, since [adding new enum values breaks backward compatibility](api_changes.md). Rather than encouraging clients to infer implicit properties from phases, we intend to explicitly expose the conditions @@ -374,7 +376,7 @@ only provided with reasonable effort, and is not guaranteed to not be lost. Status information that may be large (especially proportional in size to collections of other resources, such as lists of references to other objects -- see below) and/or rapidly changing, such as -[resource usage](../design-proposals/resources.md#usage-data), should be put into separate +[resource usage](../design-proposals/scheduling/resources.md#usage-data), should be put into separate objects, with possibly a reference from the original object. This helps to ensure that GETs and watch remain reasonably efficient for the majority of clients, which may not need that data. @@ -1228,10 +1230,11 @@ policy field. The "name" portion of the annotation should follow the below conventions for annotations. When an annotation gets promoted to a field, the name transformation should then be mechanical: `foo-bar` becomes `fooBar`. -Other advice regarding use of labels, annotations, and other generic map keys by +Other advice regarding use of labels, annotations, taints, and other generic map keys by Kubernetes components and tools: - - Key names should be all lowercase, with words separated by dashes, such as -`desired-replicas` + - Key names should be all lowercase, with words separated by dashes instead of camelCase + - For instance, prefer `foo.kubernetes.io/foo-bar over` `foo.kubernetes.io/fooBar`, prefer + `desired-replicas` over `DesiredReplicas` - Prefix the key with `kubernetes.io/` or `foo.kubernetes.io/`, preferably the latter if the label/annotation is specific to `foo` - For instance, prefer `service-account.kubernetes.io/name` over @@ -1241,7 +1244,6 @@ the resource doesn't need to know about, experimental fields that aren't intended to be generally used API fields, etc. Beware that annotations aren't automatically handled by the API conversion machinery. - ## WebSockets and SPDY Some of the API operations exposed by Kubernetes involve transfer of binary @@ -1314,6 +1316,3 @@ be less than 256", "must be greater than or equal to 0". Do not use words like "larger than", "bigger than", "more than", "higher than", etc. * When specifying numeric ranges, use inclusive ranges when possible. -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/api_changes.md b/contributors/devel/api_changes.md index dc06aeb5..8104946c 100755..100644 --- a/contributors/devel/api_changes.md +++ b/contributors/devel/api_changes.md @@ -3,7 +3,6 @@ A set of API conventions, which applies to new APIs and to changes, can be found at [API Conventions](api-conventions.md). **Table of Contents** -<!-- BEGIN MUNGE: GENERATED_TOC --> - [So you want to change the API?](#so-you-want-to-change-the-api) - [Operational overview](#operational-overview) @@ -30,15 +29,13 @@ found at [API Conventions](api-conventions.md). - [Alpha, Beta, and Stable Versions](#alpha-beta-and-stable-versions) - [Adding Unstable Features to Stable Versions](#adding-unstable-features-to-stable-versions) -<!-- END MUNGE: GENERATED_TOC --> # So you want to change the API? Before attempting a change to the API, you should familiarize yourself with a number of existing API types and with the [API conventions](api-conventions.md). If creating a new API type/resource, we also recommend that you first send a PR -containing just a proposal for the new API types, and that you initially target -the extensions API (pkg/apis/extensions). +containing just a proposal for the new API types. The Kubernetes API has two major components - the internal structures and the versioned APIs. The versioned APIs are intended to be stable, while the @@ -310,7 +307,7 @@ unacceptable. Compatibility for experimental or alpha APIs is not strictly required, but breaking compatibility should not be done lightly, as it disrupts all users of the feature. Experimental APIs may be removed. Alpha and beta API versions may be deprecated and eventually removed wholesale, as described in the -[versioning document](../design-proposals/versioning.md). +[versioning document](../design-proposals/release/versioning.md). If your change is going to be backward incompatible or might be a breaking change for API consumers, please send an announcement to @@ -322,15 +319,11 @@ If you found that your change accidentally broke clients, it should be reverted. In short, the expected API evolution is as follows: -* `extensions/v1alpha1` -> * `newapigroup/v1alpha1` -> ... -> `newapigroup/v1alphaN` -> * `newapigroup/v1beta1` -> ... -> `newapigroup/v1betaN` -> * `newapigroup/v1` -> * `newapigroup/v2alpha1` -> ... -While in extensions we have no obligation to move forward with the API at all -and may delete or break it at any time. - While in alpha we expect to move forward with it, but may break it. Once in beta we will preserve forward compatibility, but may introduce new @@ -456,7 +449,7 @@ The conversion code resides with each versioned API. There are two files: Since auto-generated conversion functions are using manually written ones, those manually written should be named with a defined convention, i.e. a -function converting type X in pkg a to type Y in pkg b, should be named: +function converting type `X` in pkg `a` to type `Y` in pkg `b`, should be named: `convert_a_X_To_b_Y`. Also note that you can (and for efficiency reasons should) use auto-generated @@ -478,7 +471,7 @@ the build system uses custom cache. `make all` will invoke `make generated_files` as well. The `make generated_files` will also regenerate the `zz_generated.deepcopy.go`, -`zz_generated.defaults.go`, and `api/openapi-spec/openapi-spec`. +`zz_generated.defaults.go`, and `api/openapi-spec/swagger.json`. If regeneration is somehow not possible due to compile errors, the easiest workaround is to remove the files causing errors and rerun the command. @@ -502,10 +495,10 @@ The generators that create go code have a `--go-header-file` flag which should be a file that contains the header that should be included. This header is the copyright that should be present at the top of the generated file and should be checked with the -[`repo-infra/verify/verify-boilerplane.sh`](https://github.com/kubernetes/repo-infra/blob/master/verify/verify-boilerplate.sh) +[`repo-infra/verify/verify-boilerplane.sh`](https://git.k8s.io/repo-infra/verify/verify-boilerplate.sh) script at a later stage of the build. -To invoke these generators, you can run `make generated`, which runs a bunch of +To invoke these generators, you can run `make update`, which runs a bunch of [scripts](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/hack/update-all.sh#L63-L78). Please continue to read the next a few sections, because some generators have prerequisites, also because they introduce how to invoke the generators @@ -531,7 +524,7 @@ run it several times to ensure there are no incompletely calculated fields. `client-gen` is a tool to generate clientsets for top-level API objects. -`client-gen` requires the `// +genclient=true` annotation on each +`client-gen` requires the `// +genclient` annotation on each exported type in both the internal `pkg/apis/<group>/types.go` as well as each specifically versioned `staging/src/k8s.io/api/<group>/<version>/types.go`. @@ -549,13 +542,17 @@ Once you added the annotations, generate the client with hack/update-codegen.sh ``` +Note that you can use the optional `// +groupGoName=` to specify a CamelCase +custom Golang identifier to de-conflict e.g. `policy.authorization.k8s.io` and +`policy.k8s.io`. These two would both map to `Policy()` in clientsets. + client-gen is flexible. See [this document](generating-clientset.md) if you need client-gen for non-kubernetes API. ### Generate Listers `lister-gen` is a tool to generate listers for a client. It reuses the -`//+genclient=true` and the `// +groupName=` annotations, so you do not need to +`//+genclient` and the `// +groupName=` annotations, so you do not need to specify extra annotations. Your previous run of `hack/update-codegen.sh` has invoked `lister-gen`. @@ -563,7 +560,7 @@ Your previous run of `hack/update-codegen.sh` has invoked `lister-gen`. ### Generate Informers `informer-gen` generates the very useful Informers which watch API -resources for changes. It reuses the `//+genclient=true` and the +resources for changes. It reuses the `//+genclient` and the `//+groupName=` annotations, so you do not need to specify extra annotations. Your previous run of `hack/update-codegen.sh` has invoked `informer-gen`. @@ -575,12 +572,13 @@ of api objects - this is to improve the overall system performance. The auto-generated code resides with each versioned API: - - `staging/src/k8s.io/api/<group>/<version>/types.generated.go` + - `staging/src/k8s.io/api/<group>/<version>/generated.proto` + - `staging/src/k8s.io/api/<group>/<version>/generated.pb.go` To regenerate them run: ```sh -hack/update-codecgen.sh +hack/update-generated-protobuf.sh ``` ## Making a new API Version @@ -599,7 +597,7 @@ Due to the fast changing nature of the project, the following content is probabl * You must add the new version to [hack/lib/init.sh#KUBE_AVAILABLE_GROUP_VERSIONS](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/hack/lib/init.sh#L53). * You must add the new version to - [staging/src/k8s.io/kube-gen/cmd/go-to-protobuf/protobuf/cmd.go](https://github.com/kubernetes/kubernetes/blob/2e6be8583d00916f1896d2b53e550162f1558ccf/staging/src/k8s.io/kube-gen/cmd/go-to-protobuf/protobuf/cmd.go#L64) + [hack/update-generated-protobuf-dockerized.sh](https://github.com/kubernetes/kubernetes/blob/v1.8.2/hack/update-generated-protobuf-dockerized.sh#L44) to generate protobuf IDL and marshallers. * You must add the new version to [cmd/kube-apiserver/app#apiVersionPriorities](https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/cmd/kube-apiserver/app/aggregator.go#L172) @@ -751,8 +749,7 @@ cases, objects will be automatically converted to the new version; in other cases, a manual upgrade may be necessary; a manual upgrade may require downtime for anything relying on the new feature, and may require manual conversion of objects to the new version; when manual conversion is necessary, the project -will provide documentation on the process (for an example, see [v1 conversion -tips](../api.md#v1-conversion-tips)) +will provide documentation on the process - Cluster Reliability: since the feature has e2e tests, enabling the feature via a flag should not create new bugs in unrelated features; because the feature is new, it may have minor bugs @@ -788,8 +785,10 @@ For example, consider the following object: ```go // API v6. type Frobber struct { - Height int `json:"height"` - Param string `json:"param"` + // height ... + Height *int32 `json:"height" protobuf:"varint,1,opt,name=height"` + // param ... + Param string `json:"param" protobuf:"bytes,2,opt,name=param"` } ``` @@ -798,65 +797,154 @@ A developer is considering adding a new `Width` parameter, like this: ```go // API v6. type Frobber struct { - Height int `json:"height"` - Width int `json:"height"` - Param string `json:"param"` + // height ... + Height *int32 `json:"height" protobuf:"varint,1,opt,name=height"` + // param ... + Param string `json:"param" protobuf:"bytes,2,opt,name=param"` + // width ... + Width *int32 `json:"width,omitempty" protobuf:"varint,3,opt,name=width"` } ``` However, the new feature is not stable enough to be used in a stable version (`v6`). Some reasons for this might include: -- the final representation is undecided (e.g. should it be called `Width` or -`Breadth`?) -- the implementation is not stable enough for general use (e.g. the `Area()` -routine sometimes overflows.) +- the final representation is undecided (e.g. should it be called `Width` or `Breadth`?) +- the implementation is not stable enough for general use (e.g. the `Area()` routine sometimes overflows.) -The developer cannot add the new field until stability is met. However, +The developer cannot add the new field unconditionally until stability is met. However, sometimes stability cannot be met until some users try the new feature, and some users are only able or willing to accept a released version of Kubernetes. In that case, the developer has a few options, both of which require staging work over several releases. - -A preferred option is to first make a release where the new value (`Width` in -this example) is specified via an annotation, like this: - -```go -kind: frobber -version: v6 -metadata: - name: myfrobber - annotations: - frobbing.alpha.kubernetes.io/width: 2 -height: 4 -param: "green and blue" -``` - -This format allows users to specify the new field, but makes it clear that they -are using a Alpha feature when they do, since the word `alpha` is in the -annotation key. +#### Alpha field in existing API version + +Previously, annotations were used for experimental alpha features, but are no longer recommended for several reasons: + +* They expose the cluster to "time-bomb" data added as unstructured annotations against an earlier API server (https://issue.k8s.io/30819) +* They cannot be migrated to first-class fields in the same API version (see the issues with representing a single value in multiple places in [backward compatibility gotchas](#backward-compatibility-gotchas)) + +The preferred approach adds an alpha field to the existing object, and ensures it is disabled by default: + +1. Add a feature gate to the API server to control enablement of the new field (and associated function): + + In [staging/src/k8s.io/apiserver/pkg/features/kube_features.go](https://git.k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/features/kube_features.go): + + ```go + // owner: @you + // alpha: v1.11 + // + // Add multiple dimensions to frobbers. + Frobber2D utilfeature.Feature = "Frobber2D" + + var defaultKubernetesFeatureGates = map[utilfeature.Feature]utilfeature.FeatureSpec{ + ... + Frobber2D: {Default: false, PreRelease: utilfeature.Alpha}, + } + ``` + +2. Add the field to the API type: + + * ensure the field is [optional](api-conventions.md#optional-vs-required) + * add the `omitempty` struct tag + * add the `// +optional` comment tag + * ensure the field is entirely absent from API responses when empty (optional fields should be pointers, anyway) + * include details about the alpha-level in the field description + + ```go + // API v6. + type Frobber struct { + // height ... + Height int32 `json:"height" protobuf:"varint,1,opt,name=height"` + // param ... + Param string `json:"param" protobuf:"bytes,2,opt,name=param"` + // width indicates how wide the object is. + // This field is alpha-level and is only honored by servers that enable the Frobber2D feature. + // +optional + Width *int32 `json:"width,omitempty" protobuf:"varint,3,opt,name=width"` + } + ``` + +3. Before persisting the object to storage, clear disabled alpha fields. +One possible place to do this is in the REST storage strategy's PrepareForCreate/PrepareForUpdate methods: + + ```go + func (frobberStrategy) PrepareForCreate(ctx genericapirequest.Context, obj runtime.Object) { + frobber := obj.(*api.Frobber) + + if !utilfeature.DefaultFeatureGate.Enabled(features.Frobber2D) { + frobber.Width = nil + } + } + + func (frobberStrategy) PrepareForUpdate(ctx genericapirequest.Context, obj, old runtime.Object) { + newFrobber := obj.(*api.Frobber) + oldFrobber := old.(*api.Frobber) + + if !utilfeature.DefaultFeatureGate.Enabled(features.Frobber2D) { + newFrobber.Width = nil + oldFrobber.Width = nil + } + } + ``` + +4. In validation, ensure the alpha field is not set if the feature gate is disabled (covers cases we might miss in the above): + + ```go + func ValidateFrobber(f *api.Frobber, fldPath *field.Path) field.ErrorList { + ... + if utilfeature.DefaultFeatureGate.Enabled(features.Frobber2D) { + ... normal validation of width field ... + } else if f.Width != nil { + allErrs = append(allErrs, field.Forbidden(fldPath.Child("width"), "disabled by feature-gate")) + } + ... + } + ``` + +Eventually, the API machinery will handle a lot of these things automatically from +declarative inputs. + +In future Kubernetes versions: + +* if the feature progresses to beta or stable status, the feature gate can be removed or be enabled by default. +* if the schema of the alpha field must change in an incompatible way, a new field name must be used. +* if the feature is abandoned, or the field name is changed, the field should be removed from the go struct, with a tombstone comment ensuring the field name and protobuf tag are not reused: + + ```go + // API v6. + type Frobber struct { + // height ... + Height int32 `json:"height" protobuf:"varint,1,opt,name=height"` + // param ... + Param string `json:"param" protobuf:"bytes,2,opt,name=param"` + + // +k8s:deprecated=width,protobuf=3 + } + ``` + +#### New alpha API version Another option is to introduce a new type with an new `alpha` or `beta` version designator, like this: ``` -// API v6alpha2 +// API v7alpha1 type Frobber struct { - Height int `json:"height"` - Width int `json:"height"` - Param string `json:"param"` + // height ... + Height *int32 `json:"height" protobuf:"varint,1,opt,name=height"` + // param ... + Param string `json:"param" protobuf:"bytes,2,opt,name=param"` + // width ... + Width *int32 `json:"width,omitempty" protobuf:"varint,3,opt,name=width"` } ``` The latter requires that all objects in the same API group as `Frobber` to be -replicated in the new version, `v6alpha2`. This also requires user to use a new +replicated in the new version, `v7alpha1`. This also requires user to use a new client which uses the other version. Therefore, this is not a preferred option. A related issue is how a cluster manager can roll back from a new version with a new feature, that is already being used by users. See https://github.com/kubernetes/kubernetes/issues/4855. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/arch-roadmap-1.png b/contributors/devel/arch-roadmap-1.png Binary files differnew file mode 100644 index 00000000..660d8206 --- /dev/null +++ b/contributors/devel/arch-roadmap-1.png diff --git a/contributors/devel/architectural-roadmap.md b/contributors/devel/architectural-roadmap.md new file mode 100644 index 00000000..afe37b1a --- /dev/null +++ b/contributors/devel/architectural-roadmap.md @@ -0,0 +1,1132 @@ +# Kubernetes Architectural Roadmap + +**Shared with the community** + +Status: First draft + +Last update: 4/20/2017 + +Authors: Brian Grant, Tim Hockin, and Clayton Coleman + +Intended audience: Kubernetes contributors + +* * * + +<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc --> +**Table of Contents** + +- [Kubernetes Architectural Roadmap](#kubernetes-architectural-roadmap) + - [Summary/TL;DR](#summarytldr) + - [Background](#background) + - [System Layers](#system-layers) + - [The Nucleus: API and Execution](#the-nucleus-api-and-execution) + - [The API and cluster control plane](#the-api-and-cluster-control-plane) + - [Execution](#execution) + - [The Application Layer: Deployment and Routing](#the-application-layer-deployment-and-routing) + - [The Governance Layer: Automation and Policy Enforcement](#the-governance-layer-automation-and-policy-enforcement) + - [The Interface Layer: Libraries and Tools](#the-interface-layer-libraries-and-tools) + - [The Ecosystem](#the-ecosystem) + - [Managing the matrix](#managing-the-matrix) + - [Layering of the system as it relates to security](#layering-of-the-system-as-it-relates-to-security) + - [Next Steps](#next-steps) + +<!-- markdown-toc end --> + + +## Summary/TL;DR + +This document describes the ongoing architectural development of the Kubernetes system, and the +motivations behind it. System developers wanting to extend or customize +Kubernetes should use this document as a guide to inform where and how best to implement these +enhancements. Application developers wanting to develop large, portable and/or future-proof +Kubernetes applications may refer to this document for guidance on which parts of Kubernetes they +can rely on being present now and in the future. + +The layers of the architecture are named and described (see the diagram below). Distinctions are +drawn between what exists today and what we plan to provide in future, and why. + +Succinctly, the layers comprise: + +1. **_The Nucleus_** which provides standardized API and execution machinery, including basic REST + mechanics, security, individual Pod, container, network interface and storage volume management, + all of which are extensible via well-defined interfaces. The Nucleus is non-optional and + expected to be the most stable part of the system. + +2. **_The Application Management Layer_** which provides basic deployment and routing, including + self-healing, scaling, service discovery, load balancing and traffic routing. This is + often referred to as orchestration and the service fabric. Default implementations of all + functions are provided, but conformant replacements are permitted. + +3. **_The Governance Layer_** which provides higher level automation and policy enforcement, + including single- and multi-tenancy, metrics, intelligent autoscaling and provisioning, and + schemes for authorization, quota, network, and storage policy expression and enforcement. These + are optional, and achievable via other solutions. + +4. **_The Interface Layer_** which provides commonly used libraries, tools, UI's and systems used to + interact with the Kubernetes API. + +5. **_The Ecosystem_** which includes everything else associated with Kubernetes, and is not really + "part of" Kubernetes at all. This is where most of the development happens, and includes CI/CD, + middleware, logging, monitoring, data processing, PaaS, serverless/FaaS systems, workflow, + container runtimes, image registries, node and cloud provider management, and many others. + + +## Background + +Kubernetes is a platform for deploying and managing containers. For more information about the +mission, scope, and design of Kubernetes, see [What Is +Kubernetes](http://kubernetes.io/docs/whatisk8s/) and the [architectural +overview](/contributors/design-proposals/architecture/architecture.md). The +latter also describes the current breakdown of the system into components/processes. + +Contributors to Kubernetes need to know what functionality they can +rely upon when adding new features to different parts of the system. + +Additionally, one of the problems that faces platforms like Kubernetes +is to define what is "in" and what is “out”. While Kubernetes must +offer some base functionality on which users can rely when running +their containerized applications or building their extensions, +Kubernetes cannot and should not try to solve every problem that users +have. Adding to the difficulty is that, unlike some other types of +infrastructure services such as databases, load balancers, or +messaging systems, there are few obvious, natural boundaries for the +“built-in” functionality. Consequently, reasonable minds can disagree +on exactly where the boundaries lie or what principles guide the +decisions. + +This document, which was inspired by [similar efforts from the +community](https://docs.google.com/document/d/1J6yCsPtggsSx_yfqNenb3xxBK22k43c5XZkVQmS38Mk/edit), +aims to clarify the intentions of the Kubernetes’s architecture +SIG. It is currently somewhat aspirational, and is intended to be a +blueprint for ongoing and future development. NIY marks items not yet +implemented as of the lated updated date at the head of this document. + +[Presentation version](https://docs.google.com/presentation/d/1oPZ4rznkBe86O4rPwD2CWgqgMuaSXguIBHIE7Y0TKVc/edit#slide=id.p) + +## System Layers + +Just as Linux has a kernel, core system libraries, and optional +userland tools, Kubernetes also has "layers" of functionality and +tools. An understanding of these layers is important for developers of +Kubernetes functionality to determine which cross-concept dependencies +should be allowed and which should not. + +Kubernetes APIs, concepts, and functionality can be sorted into the +following layers. + + + +### The Nucleus: API and Execution + +Essential API and execution machinery. + +These APIs and functions, implemented by the upstream Kubernetes +codebase, comprise the bare minimum set of features and concepts +needed to build up the higher-order layers of the system. These +pieces are thoroughly specified and documented, and every +containerized application will use them. Developers can safely assume +they are present. + +They should eventually become stable and "boring". However, Linux has +continuously evolved over its 25-year lifetime and major changes, +including NPTL (which required rebuilding all applications) and +features that enable containers (which are changing how all +applications are run), have been added over the past 10 years. It will +take some time for Kubernetes to stabilize, as well. + +#### The API and cluster control plane + +Kubernetes clusters provide a collection of similar REST APIs, exposed +by the Kubernetes [API +server](https://kubernetes.io/docs/admin/kube-apiserver/), supporting +primarily CRUD operations on (mostly) persistent resources. These APIs +serve as the hub of its control plane. + +REST APIs that follow Kubernetes API conventions (path conventions, +standard metadata, …) are automatically able to benefit from shared +API services (authorization, authentication, audit logging) and +generic client code can interact with them (CLI and UI +discoverability, generic status reporting in UIs and CLIs, generic +waiting conventions for orchestration tools, watching, label +selection). + +The lowest layer of the system also needs to support extension +mechanisms necessary to add the functionality provided by the higher +layers. Additionally, this layer must be suitable for use both in +single-purpose clusters and highly tenanted clusters. The nucleus +should provide sufficient flexibility that higher-level APIs could +introduce new scopes (sets of resources) without compromising the +security model of the cluster. + +Kubernetes cannot function without this basic API machinery and semantics, including: + +* [Authentication](https://kubernetes.io/docs/admin/authentication/): + The authentication scheme is a critical function that must be agreed + upon by both the server and clients. The API server supports basic + auth (username/password) (NOTE: We will likely deprecate basic auth + eventually.), X.509 client certificates, OpenID Connect tokens, and + bearer tokens, any of which (but not all) may be disabled. Clients + should support all forms supported by + [kubeconfig](https://kubernetes.io/docs/user-guide/kubeconfig-file/). Third-party + authentication systems may implement the TokenReview API and + configure the authentication webhook to call it, though choice of a + non-standard authentication mechanism may limit the number of usable + clients. + + * The TokenReview API (same schema as the hook) enables external + authentication checks, such as by Kubelet + + * Pod identity is provided by "[service accounts](https://kubernetes.io/docs/user-guide/service-accounts/)" + + * The ServiceAccount API, including default ServiceAccount + secret creation via a controller and injection via an + admission controller. + +* [Authorization](https://kubernetes.io/docs/admin/authorization/): + Third-party authorization systems may implement the + SubjectAccessReview API and configure the authorization webhook to + call it. + + * The SubjectAccessReview (same schema as the hook), + LocalSubjectAccessReview, and SelfSubjectAccessReview APIs + enable external permission checks, such as by Kubelet and other + controllers + +* REST semantics, watch, durability and consistency guarantees, API + versioning, defaulting, and validation + + * NIY: API deficiencies that need to be addressed: + + * [Confusing defaulting + behavior](https://github.com/kubernetes/kubernetes/issues/34292) + * [Lack of + guarantees](https://github.com/kubernetes/kubernetes/issues/30698) + * [Orchestration + support](https://github.com/kubernetes/kubernetes/issues/34363) + * [Support for event-driven + automation](https://github.com/kubernetes/kubernetes/issues/3692) + * [Clean + teardown](https://github.com/kubernetes/kubernetes/issues/4630) + +* NIY: Built-in admission-control semantics, [synchronous + admission-control hooks, and asynchronous resource + initialization](https://github.com/kubernetes/community/pull/132) -- + it needs to be possible for distribution vendors, system + integrators, and cluster administrators to impose additional + policies and automation + +* NIY: API registration and discovery, including API aggregation, to + register additional APIs, to find out which APIs are supported, and + to get the details of supported operations, payloads, and result + schemas + +* NIY: ThirdPartyResource and ThirdPartyResourceData APIs (or their + successors), to support third-party storage and extension APIs + +* NIY: An extensible and HA-compatible replacement for the + /componentstatuses API to determine whether the cluster is fully + turned up and operating correctly: ExternalServiceProvider + (component registration) + +* The Endpoints API (and future evolutions thereof), which is needed + for component registration, self-publication of API server + endpoints, and HA rendezvous, as well as application-layer target + discovery + +* The Namespace API, which is the means of scoping user resources, and + namespace lifecycle (e.g., bulk deletion) + +* The Event API, which is the means of reporting significant + occurrences, such as status changes and errors, and Event garbage + collection + +* NIY: Cascading-deletion garbage collector, finalization, and + orphaning + +* NIY: We need a built-in [add-on + manager](https://github.com/kubernetes/kubernetes/issues/23233) (not + unlike [static pod + manifests](https://kubernetes.io/docs/admin/static-pods/), but at + the cluster level) so that we can automatically add self-hosted + components and dynamic configuration to the cluster, and so we can + factor out functionality from existing components in running + clusters. At its core would be a pull-based declarative reconciler, + as provided by the [current add-on + manager](https://git.k8s.io/kubernetes/cluster/addons/addon-manager) + and as described in the [whitebox app management + doc](https://docs.google.com/document/d/1S3l2F40LCwFKg6WG0srR6056IiZJBwDmDvzHWRffTWk/edit#heading=h.gh6cf96u8mlr). This + would be easier once we have [apply support in the + API](https://github.com/kubernetes/kubernetes/issues/17333). + + * Add-ons should be cluster services that are managed as part of + the cluster and that provide the same degree of + [multi-tenancy](https://docs.google.com/document/d/148Lbe1w1xmUjMx7cIMWTVQmSjJ8qA77HIGCrdB-ugoc/edit) + as that provided by the cluster. + + * They may, but are not required to, run in the kube-system + namespace, but the chosen namespace needs to be chosen such that + it won't conflict with users' namespaces. + +* The API server acts as the gateway to the cluster. By definition, + the API server must be accessible by clients from outside the + cluster, whereas the nodes, and certainly pods, may not be. Clients + authenticate the API server and also use it as a bastion and + proxy/tunnel to nodes and pods (and services), using /proxy and + /portforward APIs. + +* TBD: The + [CertificateSigningRequest](/contributors/design-proposals/cluster-lifecycle/kubelet-tls-bootstrap.md) + API, to enable credential generation, in particular to mint Kubelet + credentials + +Ideally, this nuclear API server would only support the minimum +required APIs, and additional functionality would be added via +[aggregation](/contributors/design-proposals/api-machinery/aggregated-api-servers.md), +hooks, initializers, and other extension mechanisms. + +Note that the centralized asynchronous controllers, such as garbage +collection, are currently run by a separate process, called the +[Controller +Manager](https://kubernetes.io/docs/admin/kube-controller-manager/). + +/healthz and /metrics endpoints may be used for cluster management +mechanisms, but are not considered part of the supported API surface, +and should not be used by clients generally in order to detect cluster +presence. The /version endpoint should be used instead. + +The API server depends on the following external components: + +* Persistent state store (etcd, or equivalent; perhaps multiple + instances) + +The API server may depend on: + +* Certificate authority + +* Identity provider + +* TokenReview API implementer + +* SubjectAccessReview API implementer + +#### Execution + +The most important and most prominent controller in Kubernetes is the +[Kubelet](https://kubernetes.io/docs/admin/kubelet/), which is the +primary implementer of the Pod and Node APIs that drive the container +execution layer. Without these APIs, Kubernetes would just be a +CRUD-oriented REST application framework backed by a key-value store +(and perhaps the API machinery will eventually be spun out as an +independent project). + +Kubernetes executes isolated application containers as its default, +native mode of execution. Kubernetes provides +[Pods](https://kubernetes.io/docs/user-guide/pods/) that can host +multiple containers and storage volumes as its fundamental execution +primitive. + +The Kubelet API surface and semantics include: + +* The Pod API, the Kubernetes execution primitive, including: + + * Pod feasibility-based admission control based on policies in the + Pod API (resource requests, node selector, node/pod affinity and + anti-affinity, taints and tolerations). API admission control + may reject pods or add additional scheduling constraints to + them, but Kubelet is the final arbiter of what pods can and + cannot run on a given node, not the schedulers or DaemonSets. + + * Container and volume semantics and lifecycle + + * Pod IP address allocation (a routable IP address per pod is + required) + + * A mechanism (i.e., ServiceAccount) to tie a Pod to a specific + security scope + + * Volume sources: + + * emptyDir + + * hostPath + + * secret + + * configMap + + * downwardAPI + + * NIY: [Container and image volumes](http://issues.k8s.io/831) + (and deprecate gitRepo) + + * NIY: Claims against local storage, so that complex + templating or separate configs are not needed for dev + vs. prod application manifests + + * flexVolume (which should replace built-in + cloud-provider-specific volumes) + + * Subresources: binding, status, exec, logs, attach, portforward, + proxy + +* NIY: [Checkpointing of API + resources](https://github.com/kubernetes/kubernetes/issues/489) for + availability and bootstrapping + +* Container image and log lifecycles + +* The Secret API, and mechanisms to enable third-party secret + management + +* The ConfigMap API, for [component + configuration](https://groups.google.com/forum/#!searchin/kubernetes-dev/component$20config%7Csort:relevance/kubernetes-dev/wtXaoHOiSfg/QFW5Ca9YBgAJ) + as well as Pod references + +* The Node API, hosts for Pods + + * May only be visible to cluster administrators in some + configurations + +* Node and pod networks and their controllers (route controller) + +* Node inventory, health, and reachability (node controller) + + * Cloud-provider-specific node inventory functions should be split + into a provider-specific controller. + +* Terminated-pod garbage collection + +* Volume controller + + * Cloud-provider-specific attach/detach logic should be split into + a provider-specific controller. Need a way to extract + provider-specific volume sources from the API. + +* The PersistentVolume API + + * NIY: At least backed by local storage, as mentioned above + +* The PersistentVolumeClaim API + +Again, centralized asynchronous functions, such as terminated-pod +garbage collection, are performed by the Controller Manager. + +The Controller Manager and Kubelet currently call out to a "cloud +provider" interface to query information from the infrastructure layer +and/or to manage infrastructure resources. However, [we’re working to +extract those +touchpoints](/contributors/design-proposals/cloud-provider/cloud-provider-refactoring.md) +([issue](https://github.com/kubernetes/kubernetes/issues/2770)) into +external components. The intended model is that unsatisfiable +application/container/OS-level requests (e.g., Pods, +PersistentVolumeClaims) serve as a signal to external “dynamic +provisioning” systems, which would make infrastructure available to +satisfy those requests and represent them in Kubernetes using +infrastructure resources (e.g., Nodes, PersistentVolumes), so that +Kubernetes could bind the requests and infrastructure resources +together. + +The Kubelet depends on the following external components: + +* Image registry + +* Container Runtime Interface implementation + +* Container Network Interface implementation + +* FlexVolume implementations ("CVI" in the diagram) + +And may depend on: + +* NIY: Cloud-provider node plug-in, to provide node identity, + topology, etc. + +* NIY: Third-party secret management system (e.g., Vault) + +* NIY: Credential generation and rotation controller + +Accepted layering violations: + +* [Explicit service links](https://github.com/kubernetes/community/pull/176) + +* The kubernetes service for the API server + +### The Application Layer: Deployment and Routing + +The application management and composition layer, providing +self-healing, scaling, application lifecycle management, service +discovery, load balancing, and routing -- also known as orchestration +and the service fabric. + +These APIs and functions are REQUIRED for any distribution of +Kubernetes. Kubernetes should provide default implementations for +these APIs, but replacements of the implementations of any or all of +these functions are permitted, provided the conformance tests +pass. Without these, most containerized applications will not run, and +few, if any, published examples will work. The vast majority of +containerized applications will use one or more of these. + +Kubernetes’s API provides IaaS-like container-centric primitives and +also lifecycle controllers to support orchestration (self-healing, +scaling, updates, termination) of all major categories of +workloads. These application management, composition, discovery, and +routing APIs and functions include: + +* A default scheduler, which implements the scheduling policies in the + Pod API: resource requests, nodeSelector, node and pod + affinity/anti-affinity, taints and tolerations. The scheduler runs + as a separate process, on or outside the cluster. + +* NIY: A + [rescheduler](/contributors/design-proposals/scheduling/rescheduling.md), + to reactively and proactively delete scheduled pods so that they can + be replaced and rescheduled to other nodes. + +* Continuously running applications: These application types should + all support rollouts (and rollbacks) via declarative updates, + cascading deletion, and orphaning/adoption. Other than DaemonSet, + all should support horizontal scaling. + + * The Deployment API, which orchestrates updates of stateless + applications, including subresources (status, scale, rollback) + + * The ReplicaSet API, for simple fungible/stateless + applications, especially specific versions of Deployment pod + templates, including subresources (status, scale) + + * The DaemonSet API, for cluster services, including subresources + (status) + + * The StatefulSet API, for stateful applications, including + subresources (status, scale) + + * The PodTemplate API, used by DaemonSet and StatefulSet to record change history + +* Terminating batch applications: These should include support for + automatic culling of terminated jobs (NIY). + + * The Job API ([GC + discussion](https://github.com/kubernetes/kubernetes/issues/30243)) + + * The CronJob API + +* Discovery, load balancing, and routing + + * The Service API, including allocation of cluster IPs, repair on + service allocation maps, load balancing via kube-proxy or + equivalent, and automatic Endpoints generation, maintenance, and + deletion for services. NIY: LoadBalancer service support is + OPTIONAL, but conformance tests must pass if it is + supported. If/when they are added, support for [LoadBalancer and + LoadBalancerClaim + APIs](https://github.com/kubernetes/community/pull/275) should + be present if and only if the distribution supports LoadBalancer + services. + + * The Ingress API, including [internal + L7](https://docs.google.com/document/d/1ILXnyU5D5TbVRwmoPnC__YMO9T5lmiA_UiU8HtgSLYk/edit?ts=585421fc) + (NIY) + + * Service DNS. DNS, using the [official Kubernetes + schema](https://git.k8s.io/dns/docs/specification.md), + is required. + +The application layer may depend on: + +* Identity provider (to-cluster identities and/or to-application + identities) + +* NIY: Cloud-provider controller implementation + +* Ingress controller(s) + +* Replacement and/or additional schedulers and/or reschedulers + +* Replacement DNS service + +* Replacement for kube-proxy + +* Replacement and/or + [auxiliary](https://github.com/kubernetes/kubernetes/issues/31571) + workload controllers, especially for extended rollout strategies + +### The Governance Layer: Automation and Policy Enforcement + +Policy enforcement and higher-level automation. + +These APIs and functions should be optional for running applications, +and should be achievable via other solutions. + +Each supported API/function should be applicable to a large fraction +of enterprise operations, security, and/or governance scenarios. + +It needs to be possible to configure and discover default policies for +the cluster (perhaps similar to [Openshift’s new project template +mechanism](https://docs.openshift.org/latest/admin_guide/managing_projects.html#template-for-new-projects), +but supporting multiple policy templates, such as for system +namespaces vs. user ones), to support at least the following use +cases: + +* Is this a: (source: [multi-tenancy working + doc](https://docs.google.com/document/d/1IoINuGz8eR8Awk4o7ePKuYv9wjXZN5nQM_RFlVIhA4c/edit?usp=sharing)) + + * Single tenant / single user cluster + + * Multiple trusted tenant cluster + + * Production vs. dev cluster + + * Highly tenanted playground cluster + + * Segmented cluster for reselling compute / app services to others + +* Do I care about limiting: + + * Resource usage + + * Internal segmentation of nodes + + * End users + + * Admins + + * DoS + +Automation APIs and functions: + +* The Metrics APIs (needed for H/V autoscaling, scheduling TBD) + +* The HorizontalPodAutoscaler API + +* NIY: The vertical pod autoscaling API(s) + +* [Cluster autoscaling and/or node + provisioning](https://git.k8s.io/contrib/cluster-autoscaler) + +* The PodDisruptionBudget API + +* Dynamic volume provisioning, for at least one volume source type + + * The StorageClass API, implemented at least for the default + volume type + +* Dynamic load-balancer provisioning + +* NIY: The + [PodPreset](/contributors/design-proposals/service-catalog/pod-preset.md) + API + +* NIY: The [service + broker/catalog](https://github.com/kubernetes-incubator/service-catalog) + APIs + +* NIY: The + [Template](/contributors/design-proposals/apps/OBSOLETE_templates.md) + and TemplateInstance APIs + +Policy APIs and functions: + +* [Authorization](https://kubernetes.io/docs/admin/authorization/): + The ABAC and RBAC authorization policy schemes. + + * RBAC, if used, is configured using a number of APIs: Role, + RoleBinding, ClusterRole, ClusterRoleBinding + +* The LimitRange API + +* The ResourceQuota API + +* The PodSecurityPolicy API + +* The ImageReview API + +* The NetworkPolicy API + +The management layer may depend on: + +* Network policy enforcement mechanism + +* Replacement and/or additional horizontal and vertical pod + autoscalers + +* [Cluster autoscaler and/or node provisioner](https://git.k8s.io/contrib/cluster-autoscaler) + +* Dynamic volume provisioners + +* Dynamic load-balancer provisioners + +* Metrics monitoring pipeline, or a replacement for it + +* Service brokers + +### The Interface Layer: Libraries and Tools + +These mechanisms are suggested for distributions, and also are +available for download and installation independently by users. They +include commonly used libraries, tools, systems, and UIs developed by +official Kubernetes projects, though other tools may be used to +accomplish the same tasks. They may be used by published examples. + +Commonly used libraries, tools, systems, and UIs developed under some +Kubernetes-owned GitHub org. + +* Kubectl -- We see kubectl as one of many client tools, rather than + as a privileged one. Our aim is to make kubectl thinner, by [moving + commonly used non-trivial functionality into the + API](https://github.com/kubernetes/kubernetes/issues/12143). This is + necessary in order to facilitate correct operation across Kubernetes + releases, to facilitate API extensibility, to preserve the + API-centric Kubernetes ecosystem model, and to simplify other + clients, especially non-Go clients. + +* Client libraries (e.g., client-go, client-python) + +* Cluster federation (API server, controllers, kubefed) + +* Dashboard + +* Helm + +These components may depend on: + +* Kubectl extensions (discoverable via help) + +* Helm extensions (discoverable via help) + +### The Ecosystem + +These things are not really "part of" Kubernetes at all. + +There are a number of areas where we have already defined [clear-cut +boundaries](https://kubernetes.io/docs/whatisk8s#kubernetes-is-not) +for Kubernetes. + +While Kubernetes must offer functionality commonly needed to deploy +and manage containerized applications, as a general rule, we preserve +user choice in areas complementing Kubernetes’s general-purpose +orchestration functionality, especially areas that have their own +competitive landscapes comprised of numerous solutions satisfying +diverse needs and preferences. Kubernetes may provide plug-in APIs for +such solutions, or may expose general-purpose APIs that could be +implemented by multiple backends, or expose APIs that such solutions +can target. Sometimes, the functionality can compose cleanly with +Kubernetes without explicit interfaces. + +Additionally, to be considered part of Kubernetes, a component would +need to follow Kubernetes design conventions. For instance, systems +whose primary interfaces are domain-specific languages (e.g., +[Puppet](https://docs.puppet.com/puppet/4.9/lang_summary.html), [Open +Policy Agent](http://www.openpolicyagent.org/)) aren’t compatible with +the Kubernetes API-centric approach, and are perfectly fine to use +with Kubernetes, but wouldn’t be considered to be part of +Kubernetes. Similarly, solutions designed to support multiple +platforms likely wouldn’t follow Kubernetes API conventions, and +therefore wouldn’t be considered to be part of Kubernetes. + +* Inside container images: Kubernetes is not opinionated about the + contents of container images -- if it lives inside the container + image, it lives outside Kubernetes. This includes, for example, + language-specific application frameworks. + +* On top of Kubernetes + + * Continuous integration and deployment: Kubernetes is + unopinionated in the source-to-image space. It does not deploy + source code and does not build your application. Continuous + Integration (CI) and continuous deployment workflows are areas + where different users and projects have their own requirements + and preferences, so we aim to facilitate layering CI/CD + workflows on Kubernetes but don't dictate how they should work. + + * Application middleware: Kubernetes does not provide application + middleware, such as message queues and SQL databases, as + built-in infrastructure. It may, however, provide + general-purpose mechanisms, such as service-broker integration, + to make it easier to provision, discover, and access such + components. Ideally, such components would just run on + Kubernetes. + + * Logging and monitoring: Kubernetes does not provide logging + aggregation, comprehensive application monitoring, nor telemetry + analysis and alerting systems, though such mechanisms are + essential to production clusters. + + * Data-processing platforms: Spark and Hadoop are well known + examples, but there are [many such + systems](https://hadoopecosystemtable.github.io/). + + * [Application-specific + operators](https://coreos.com/blog/introducing-operators.html): + Kubernetes supports workload management for common categories of + applications, but not for specific applications. + + * Platform as a Service: Kubernetes [provides a + foundation](http://blog.kubernetes.io/2017/02/caas-the-foundation-for-next-gen-paas.html) + for a multitude of focused, opinionated PaaSes, including DIY + ones. + + * Functions as a Service: Similar to PaaS, but FaaS additionally + encroaches into containers and language-specific application + frameworks. + + * [Workflow + orchestration](https://github.com/kubernetes/kubernetes/pull/24781#issuecomment-215914822): + "Workflow" is a very broad, diverse area, with solutions + typically tailored to specific use cases (data-flow graphs, + data-driven processing, deployment pipelines, event-driven + automation, business-process execution, iPaaS) and specific + input and event sources, and often requires arbitrary code to + evaluate conditions, actions, and/or failure handling. + + * [Configuration + DSLs](https://github.com/kubernetes/kubernetes/pull/1007/files): + Domain-specific languages do not facilitate layering + higher-level APIs and tools, they usually have limited + expressibility, testability, familiarity, and documentation, + they promote complex configuration generation, they tend to + compromise interoperability and composability, they complicate + dependency management, and uses often subvert abstraction and + encapsulation. + + * [Kompose](https://github.com/kubernetes-incubator/kompose): + Kompose is a project-supported adaptor tool that facilitates + migration to Kubernetes from Docker Compose and enables simple + use cases, but doesn’t follow Kubernetes conventions and is + based on a manually maintained DSL. + + * [ChatOps](https://github.com/harbur/kubebot): Also adaptor + tools, for the multitude of chat services. + +* Underlying Kubernetes + + * Container runtime: Kubernetes does not provide its own container + runtime, but provides an interface for plugging in the container + runtime of your choice. + + * Image registry: Kubernetes pulls container images to the nodes. + + * Cluster state store: Etcd + + * Network: As with the container runtime, we support an interface + (CNI) that facilitates pluggability. + + * File storage: Local filesystems and network-attached storage. + + * Node management: Kubernetes neither provides nor adopts any + comprehensive machine configuration, maintenance, management, or + self-healing systems, which typically are handled differently in + different public/private clouds, for different operating + systems, for mutable vs. immutable infrastructure, for shops + already using tools outside of their Kubernetes clusters, etc. + + * Cloud provider: IaaS provisioning and management. + + * Cluster creation and management: The community has developed + numerous tools, such as minikube, kubeadm, bootkube, kube-aws, + kops, kargo, kubernetes-anywhere, and so on. As can be seen from + the diversity of tools, there is no one-size-fits-all solution + for cluster deployment and management (e.g., upgrades). There's + a spectrum of possible solutions, each with different + tradeoffs. That said, common building blocks (e.g., [secure + Kubelet + registration](/contributors/design-proposals/cluster-lifecycle/kubelet-tls-bootstrap.md)) + and approaches (in particular, + [self-hosting](/contributors/design-proposals/cluster-lifecycle/self-hosted-kubernetes.md#what-is-self-hosted)) + would reduce the amount of custom orchestration needed in such + tools. + +We would like to see the ecosystem build and/or integrate solutions to +fill these needs. + +Eventually, most Kubernetes development should fall in the ecosystem. + +## Managing the matrix + +Options, Configurable defaults, Extensions, Plug-ins, Add-ons, +Provider-specific functionality, Version skew, Feature discovery, and +Dependency management. + +Kubernetes is not just an open-source toolkit, but is typically +consumed as a running, easy-to-run, or ready-to-run cluster or +service. We would like most users and use cases to be able to use +stock upstream releases. This means Kubernetes needs sufficient +extensibility without rebuilding to handle such use cases. + +While gaps in extensibility are the primary drivers of code forks and +gaps in upstream cluster lifecycle management solutions are currently +the primary drivers of the proliferation of Kubernetes distributions, +the existence of optional features (e.g., alpha APIs, +provider-specific APIs), configurability, pluggability, and +extensibility make the concept inevitable. + +However, to make it possible for users to deploy and manage their +applications and for developers to build Kubernetes extensions on/for +arbitrary Kubernetes clusters, they must be able to make assumptions +about what a cluster or distribution of Kubernetes provides. Where +functionality falls out of these base assumptions, there needs to be a +way to discover what functionality is available and to express +functionality requirements (dependencies) for usage. + +Cluster components, including add-ons, should be registered via the +[component registration +API](https://github.com/kubernetes/kubernetes/issues/18610) and +discovered via /componentstatuses. + +Enabled built-in APIs, aggregated APIs, and registered third-party +resources should be discoverable via the discovery and OpenAPI +(swagger.json) endpoints. As mentioned above, cloud-provider support +for LoadBalancer-type services should be determined by whether the +LoadBalancer API is present. + +Extensions and their options should be registered via FooClass +resources, similar to +[StorageClass](https://git.k8s.io/kubernetes/pkg/apis/storage/v1beta1/types.go#L31), +but with parameter descriptions, types (e.g., integer vs string), +constraints (e.g., range or regexp) for validation, and default +values, with a reference to fooClassName from the extended API. These +APIs should also configure/expose the presence of related features, +such as dynamic volume provisioning (indicated by a non-empty +storageclass.provisioner field), as well as identifying the +responsible +[controller](https://github.com/kubernetes/kubernetes/issues/31571). We +need to add such APIs for at least scheduler classes, ingress +controller classes, flex volume classes, and compute resource classes +(e.g., GPUs, other accelerators). + +Assuming we transitioned existing network-attached volume sources to +flex volumes, this approach would cover volume sources. In the future, +the API should provide only [general-purpose +abstractions](https://docs.google.com/document/d/1QVxD---9tHXYj8c_RayLY9ClrFpqfuejN7p0vtv2kW0/edit#heading=h.mij1ubfelvar), +even if, as with LoadBalancer services, the abstractions are not +implemented in all environments (i.e., the API does not need to cater +to the lowest common denominator). + +NIY: We also need to develop mechanisms for registering and +discovering the following: + +* Admission-control plugins and hooks (including for built-in APIs) + +* Authentication plugins + +* Authorization plugins and hooks + +* Initializers and finalizers + +* [Scheduler + extensions](/contributors/design-proposals/scheduling/scheduler_extender.md) + +* Node labels and [cluster + topology](https://github.com/kubernetes/kubernetes/issues/41442) + (topology classes?) + +NIY: Activation/deactivation of both individual APIs and finer-grain +features could be addressed by the following mechanisms: + +* [The configuration for all components is being converted from + command-line flags to versioned + configuration.](https://github.com/kubernetes/kubernetes/issues/12245) + +* [We intend to store most of that configuration data in + ](https://github.com/kubernetes/kubernetes/issues/1627)[ConfigMap](https://github.com/kubernetes/kubernetes/issues/1627)[s, + to facilitate dynamic reconfiguration, progressive rollouts, and + introspectability.](https://github.com/kubernetes/kubernetes/issues/1627) + +* [Configuration common to all/multiple components should be factored + out into its own configuration + object(s).](https://github.com/kubernetes/kubernetes/issues/19831) + This should include the [feature-gate + mechanism](/contributors/design-proposals/cluster-lifecycle/runtimeconfig.md). + +* An API should be added for semantically meaningful settings, such as + the default length of time to wait before deleting pods on + unresponsive nodes. + +NIY: The problem of [version-skewed +operation](https://github.com/kubernetes/kubernetes/issues/4855), for +features dependent on upgrades of multiple components (including +replicas of the same component in HA clusters), should be addressed +by: + +1. Creating flag gates for all new such features, + +2. Always disabling the features by default in the first minor release + in which they appear, + +3. Providing configuration patches to enable the features, and + +4. Enabling them by default in the next minor release. + +NIY: We additionally need a mechanism to [warn about out of date +nodes](https://github.com/kubernetes/kubernetes/issues/23874), and/or +potentially prevent master upgrades (other than to patch releases) +until/unless the nodes have been upgraded. + +NIY: [Field-level +versioning](https://github.com/kubernetes/kubernetes/issues/34508) +would facilitate solutions to bulk activation of new and/or alpha API +fields, prevention of clobbering of new fields by poorly written +out-of-date clients, and evolution of non-alpha APIs without a +proliferation of full-fledged API definitions. + +The Kubernetes API server silently ignores unsupported resource fields +and query parameters, but not unknown/unregistered APIs (note that +unimplemented/inactive APIs should be disabled). This can facilitate +the reuse of configuration across clusters of multiple releases, but +more often leads to surprises. Kubectl supports optional validation +using the Swagger/OpenAPI specification from the server. Such optional +validation should be [provided by the +server](https://github.com/kubernetes/kubernetes/issues/5889) +(NIY). Additionally, shared resource manifests should specify the +minimum required Kubernetes release, for user convenience, which could +potentially be verified by kubectl and other clients. + +Additionally, unsatisfiable Pod scheduling constraints and +PersistentVolumeClaim criteria silently go unmet, which can useful as +demand signals to automatic provisioners, but also makes the system +more error prone. It should be possible to configure rejection of +unsatisfiable requests, using FooClass-style APIs, as described above +([NIY](https://github.com/kubernetes/kubernetes/issues/17324)). + +The Service Catalog mechanism (NIY) should make it possible to assert +the existence of application-level services, such as S3-compatible +cluster storage. + +## Layering of the system as it relates to security + +In order to properly secure a Kubernetes cluster and enable [safe +extension](https://github.com/kubernetes/kubernetes/issues/17456), a +few fundamental concepts need to be defined and agreed on by the +components of the system. It’s best to think of Kubernetes as a series +of rings from a security perspective, with each layer granting the +successive layer capabilities to act. + +1. One or more data storage systems (etcd) for the nuclear APIs + +2. The nuclear APIs + +3. APIs for highly trusted resources (system policies) + +4. Delegated trust APIs and controllers (users grant access to the API + / controller to perform actions on their behalf) either at the + cluster scope or smaller scopes + +5. Untrusted / scoped APIs and controllers and user workloads that run + at various scopes + +When a lower layer depends on a higher layer, it collapses the +security model and makes defending the system more complicated - an +administrator may *choose* to do so to gain operational simplicity, +but that must be a conscious choice. A simple example is etcd: any +component that can write data to etcd is now root on the entire +cluster, and any actor that can corrupt a highly trusted resource can +almost certainly escalate. It is useful to divide the layers above +into separate sets of machines for each layer of processes (etcd -> +apiservers + controllers -> nuclear security extensions -> delegated +extensions -> user workloads), even if some may be collapsed in +practice. + +If the layers described above define concentric circles, then it +should also be possible for overlapping or independent circles to +exist - for instance, administrators may choose an alternative secret +storage solution that cluster workloads have access to yet the +platform does not implicitly have access to. The point of intersection +for these circles tends to be the machines that run the workloads, and +nodes must have no more privileges than those required for proper +function. + +Finally, adding a new capability via extension at any layer should +follow best practices for communicating the impact of that action. + +When a capability is added to the system via extension, what purpose +does it have? + +* Make the system more secure + +* Enable a new "production quality" API for consumption by everyone in + the cluster + +* Automate a common task across a subset of the cluster + +* Run a hosted workload that offers apis to consumers (spark, a + database, etcd) + +* These fall into three major groups: + + * Required for the cluster (and hence must run close to the core, + and cause operational tradeoffs in the presence of failure) + + * Exposed to all cluster users (must be properly tenanted) + + * Exposed to a subset of cluster users (runs more like traditional + "app" workloads) + +If an administrator can easily be tricked into installing a new +cluster level security rule during extension, then the layering is +compromised and the system is vulnerable. + +## Next Steps + +In addition to completing the technical mechanisms described and/or +implied above, we need to apply the principles in this document to a +set of more focused documents that answer specific practical +questions. Here are some suggested documents and the questions they +answer: + +* **Kubernetes API Conventions For Extension API Developers** + + * Audience: someone planning to build a Kubernetes-like API + extension (TPR or Aggregated API, etc…) + + * Answers Questions: + + * What conventions should I follow? (metadata, status, etc) + + * What integrations do I get from following those conventions? + + * What can I omit and Does + + * Document that answers this question: + +* **Kubernetes API Conventions For CLI and UI Developers** + + * Audience: someone working on kubectl, dashboard, or another CLI + or UI. + + * Answers Questions: + + * How can I show a user which objects subordinate and should + normally be hidden + +* **Required and Optional Behaviors for Kubernetes + Distributions/Services** + + * See also the [certification + issue](https://github.com/kubernetes/community/issues/432) + + * I just implemented a Hosted version of the Kubernetes API. + + * Which API groups, versions and Kinds do I have to expose? + + * Can I provide my own logging, auth, auditing integrations + and so on? + + * Can I call it Kubernetes if it just has pods but no services? + + * I'm packaging up a curated Kubernetes distro and selling it. + + * Which API groups, versions and Kinds do I have to expose in + order to call it Kubernetes. + +* **Assumptions/conventions for application developers** + + * I want to write portable config across on-prem and several clouds. + + * What API types and behaviors should I assume are always + present and fully abstracted. When should I try to detect a + feature (by asking the user what cloud provider they have, + or in the future a feature discovery API). + +* **Kubernetes Security Models** + + * Audience: hosters, distributors, custom cluster builders + diff --git a/contributors/devel/automation.md b/contributors/devel/automation.md index 2744ef84..e46ff9a5 100644 --- a/contributors/devel/automation.md +++ b/contributors/devel/automation.md @@ -11,12 +11,12 @@ processes. In an effort to * reduce load on core developers - * maintain e2e stability + * maintain end-to-end test stability * load test github's label feature -We have added an automated [submit-queue](https://github.com/kubernetes/test-infra/tree/master/mungegithub/submit-queue) +We have added an automated [submit-queue](https://git.k8s.io/test-infra/mungegithub/submit-queue) to the -[github "munger"](https://github.com/kubernetes/test-infra/tree/master/mungegithub) +[github "munger"](https://git.k8s.io/test-infra/mungegithub) for kubernetes. The submit-queue does the following: @@ -35,40 +35,23 @@ The status of the submit-queue is [online.](http://submit-queue.k8s.io/) ### Ready to merge status -The submit-queue lists what it believes are required on the [merge requirements tab](http://submit-queue.k8s.io/#/info) of the info page. That may be more up to date. - -A PR is considered "ready for merging" if it matches the following: - * The PR must have the label "cla: yes" or "cla: human-approved" - * The PR must be mergeable. aka cannot need a rebase - * All of the following github statuses must be green - * Jenkins GCE Node e2e - * Jenkins GCE e2e - * Jenkins unit/integration - * The PR cannot have any prohibited future milestones (such as a v1.5 milestone during v1.4 code freeze) - * The PR must have the "lgtm" label. The "lgtm" label is automatically applied - following a review comment consisting of only "LGTM" (case-insensitive) - * The PR must not have been updated since the "lgtm" label was applied - * The PR must not have the "do-not-merge" label +A PR is considered "ready for merging" by the submit queue if it matches the set +of conditions listed in the [merge requirements tab](http://submit-queue.k8s.io/#/info) +of the info page. +Please visit that page for more details. ### Merge process -Merges _only_ occur when the [critical builds](http://submit-queue.k8s.io/#/e2e) -are passing. We're open to including more builds here, let us know... - -Merges are serialized, so only a single PR is merged at a time, to ensure -against races. - If the PR has the `retest-not-required` label, it is simply merged. If the PR does -not have this label the e2e, unit/integration, and node tests are re-run. If these -tests pass a second time, the PR will be merged as long as the `critical builds` are -green when this PR finishes retesting. +not have this label, the aforementioned required tests are re-run. +If these tests pass a second time, the PR will be merged when this PR finishes retesting. ## Github Munger -We run [github "mungers"](https://github.com/kubernetes/contrib/tree/master/mungegithub). +We run [github "mungers"](https://git.k8s.io/test-infra/mungegithub). -This runs repeatedly over github pulls and issues and runs modular "mungers" -similar to "mungedocs." The mungers include the 'submit-queue' referenced above along +This runs repeatedly over github pulls and issues and runs modular "mungers". +The mungers include the "submit-queue" referenced above along with numerous other functions. See the README in the link above. Please feel free to unleash your creativity on this tool, send us new mungers @@ -79,9 +62,9 @@ that you think will help support the Kubernetes development process. Github Munger will close pull-requests that don't have human activity in the last 90 days. It will warn about this process 60 days before closing the pull-request, and warn again 30 days later. One way to prevent this from -happening is to add the "keep-open" label on the pull-request. +happening is to add the `keep-open` label on the pull-request. -Feel free to re-open and maybe add the "keep-open" label if this happens to a +Feel free to re-open and maybe add the `keep-open` label if this happens to a valid pull-request. It may also be a good opportunity to get more attention by verifying that it is properly assigned and/or mention people that might be interested. Commenting on the pull-request will also keep it open for another 90 @@ -91,25 +74,19 @@ days. We also run a robotic PR builder that attempts to run tests for each PR. -Before a PR from an unknown user is run, the PR builder bot (`k8s-bot`) asks to -a message from a contributor that a PR is "ok to test", the contributor replies -with that message. ("please" is optional, but remember to treat your robots with -kindness...) +Before a PR from an unknown user is run, the PR builder bot (`@k8s-ci-robot`) asks to +a message from a kubernetes member that a PR is safe to test, the member can +reply with the `/ok-to-test` command on a single line to begin CI testing. ## FAQ: #### How can I ask my PR to be tested again for Jenkins failures? PRs should only need to be manually re-tested if you believe there was a flake -during the original test. All flakes should be filed as an -[issue](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Akind%2Fflake). -Once you find or file a flake a contributor (this may be you!) should request -a retest with "@k8s-bot test this issue: #NNNNN", where NNNNN is replaced with -the issue number you found or filed. +during the original test. It would be good to file flakes as an +[issue](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Akind%2Fflake). +`@k8s-ci-robot` will comment to tell you which test(s) failed and how to re-test. +The simplest way is to comment `/retest`. Any pushes of new code to the PR will automatically trigger a new test. No human -interraction is required. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> +interraction is required. Note that if the PR has a `lgtm` label, it will be removed after the pushes. diff --git a/contributors/devel/bazel.md b/contributors/devel/bazel.md index fc99d775..de80b4b2 100644 --- a/contributors/devel/bazel.md +++ b/contributors/devel/bazel.md @@ -1,8 +1,12 @@ -# Build with Bazel +# Build and test with Bazel -Building with Bazel is currently experimental. Automanaged `BUILD` rules have the -tag "automanaged" and are maintained by -[gazel](https://github.com/mikedanese/gazel). Instructions for installing Bazel +Building and testing Kubernetes with Bazel is supported but not yet default. + +Go rules are managed by the [`gazelle`](https://github.com/bazelbuild/rules_go/tree/master/go/tools/gazelle) +tool, with some additional rules managed by the [`kazel`](https://git.k8s.io/repo-infra/kazel) tool. +These tools are called via the `hack/update-bazel.sh` script. + +Instructions for installing Bazel can be found [here](https://www.bazel.io/versions/master/docs/install.html). Several `make` rules have been created for common operations: @@ -20,22 +24,47 @@ tests, run $ bazel test //pkg/kubectl/... ``` +## Planter +If you don't want to install Bazel, you can instead try using the unofficial +[Planter](https://git.k8s.io/test-infra/planter) tool, +which runs Bazel inside a Docker container. + +For example, you can run +```console +$ ../test-infra/planter/planter.sh make bazel-test +$ ../test-infra/planter/planter.sh bazel build //cmd/kubectl +``` + ## Continuous Integration -The [Bazel CI job](http://k8s-testgrid.appspot.com/google-unit#bazel) runs -`make bazel-build`, `make bazel-test`, and (transitively) `make bazel-release`. -A similar job is run on all PRs. +There are several bazel CI jobs: +* [ci-kubernetes-bazel-build](http://k8s-testgrid.appspot.com/google-unit#bazel-build): builds everything + with Bazel +* [ci-kubernetes-bazel-test](http://k8s-testgrid.appspot.com/google-unit#bazel-test): runs unit tests in + with Bazel -Many steps are cached, so the Bazel job usually executes fairly quickly. +Similar jobs are run on all PRs; additionally, several of the e2e jobs use +Bazel-built binaries when launching and testing Kubernetes clusters. ## Known issues [Cross-compilation is not currently supported](https://github.com/bazelbuild/rules_go/issues/70), -so all binaries will be built for the host architecture running Bazel. -Additionally, Go build tags are not supported. This means that builds on macOS may not work. +so all binaries will be built for the host OS and architecture running Bazel. +(For example, you can't currently target linux/amd64 from macOS or linux/s390x +from an amd64 machine.) + +Additionally, native macOS support is still a work in progress. Using Planter is +a possible workaround in the interim. + +[Bazel does not validate build environment](https://github.com/kubernetes/kubernetes/issues/51623), thus make sure that needed +tools and development packages are installed in the system. Bazel builds require presence of `make`, `gcc`, `g++`, `glibc and libstdc++ development headers` and `glibc static development libraries`. Please check your distribution for exact names of the packages. Examples for some commonly used distributions are below: + +| Dependency | Debian/Ubuntu | CentOS | OpenSuSE | +|:---------------------:|-------------------------------|--------------------------------|-----------------------------------------| +| Build essentials | `apt install build-essential` | `yum groupinstall development` | `zypper install -t pattern devel_C_C++` | +| GCC C++ | `apt install g++` | `yum install gcc-c++` | `zypper install gcc-c++` | +| GNU Libc static files | `apt install libc6-dev` | `yum install glibc-static` | `zypper install glibc-devel-static` | -[Binaries produced by Bazel are not statically linked](https://github.com/bazelbuild/rules_go/issues/161), -and they are not currently tagged with version information. ## Updating `BUILD` files @@ -45,10 +74,15 @@ To update `BUILD` files, run: $ ./hack/update-bazel.sh ``` -**NOTE**: `update-bazel.sh` only works if check out directory of Kubernetes is `$GOPATH/src/k8s.io/kubernetes`. +To prevent Go rules from being updated, consult the [gazelle +documentation](https://github.com/bazelbuild/rules_go/tree/master/go/tools/gazelle). + +Note that much like Go files and `gofmt`, BUILD files have standardized, +opinionated style rules, and running `hack/update-bazel.sh` will format them for you. -Only rules which are automanaged will be updated, but all rules will be -auto-formatted. +If you want to auto-format BUILD files in your editor, using something like +[Buildifier](https://github.com/bazelbuild/buildtools/blob/master/buildifier/README.md) +is recommended. Updating the `BUILD` file for a package will be required when: * Files are added to or removed from a package @@ -56,6 +90,6 @@ Updating the `BUILD` file for a package will be required when: * A `BUILD` file has been updated and needs to be reformatted * A new `BUILD` file has been added (parent `BUILD` files will be updated) -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> +## Contacts +For help or discussion, join the [#bazel](https://kubernetes.slack.com/messages/bazel) +channel on Kubernetes Slack. diff --git a/contributors/devel/cherry-picks.md b/contributors/devel/cherry-picks.md index 1a6d621e..232b2a0b 100644 --- a/contributors/devel/cherry-picks.md +++ b/contributors/devel/cherry-picks.md @@ -9,13 +9,13 @@ depending on the point in the release cycle. 1. Cherrypicks are [managed with labels and milestones](pull-requests.md#write-release-notes-if-needed) 1. To get a PR merged to the release branch, first ensure the following labels are on the original **master** branch PR: - * An appropriate milestone (e.g. v1.3) + * The milestone for the branch that you want to cherry-pick to (e.g. v1.7 if you want to backport your change in 1.7 branch) * The `cherrypick-candidate` label 1. If `release-note-none` is set on the master PR, the cherrypick PR will need to set the same label to confirm that no release note is needed. 1. `release-note` labeled PRs generate a release note using the PR title by default OR the release-note block in the PR template if filled in. - * See the [PR template](https://github.com/kubernetes/kubernetes/blob/master/.github/PULL_REQUEST_TEMPLATE.md) for more details. + * See the [PR template](https://git.k8s.io/kubernetes/.github/PULL_REQUEST_TEMPLATE.md) for more details. * PR titles and body comments are mutable and can be modified at any time prior to the release to reflect a release note friendly message. @@ -33,8 +33,8 @@ label and lose the `cherrypick-candidate` label. to the remote branch `upstream/release-3.14`: `hack/cherry_pick_pull.sh upstream/release-3.14 98765` * Your cherrypick PR (targeted to the branch) will immediately get the -`do-not-merge` label. The branch owner will triage PRs targeted to -the branch and label the ones to be merged by applying the `lgtm` +`do-not-merge/cherry-pick-not-approved` label. The branch owner will triage PRs +targeted to the branch and label the ones to be merged by applying the `lgtm` label. There is an [issue](https://github.com/kubernetes/kubernetes/issues/23347) open @@ -56,7 +56,3 @@ status of PRs labeled as `cherrypick-candidate`. considered implicit for all code within cherry-pick pull requests, ***unless there is a large conflict***. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/client-libraries.md b/contributors/devel/client-libraries.md index 6686fc7c..018c929e 100644 --- a/contributors/devel/client-libraries.md +++ b/contributors/devel/client-libraries.md @@ -1,7 +1,3 @@ ## Kubernetes API client libraries This document has been moved to https://kubernetes.io/docs/reference/client-libraries/. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/coding-conventions.md b/contributors/devel/coding-conventions.md index 04f58613..17c890a9 100644 --- a/contributors/devel/coding-conventions.md +++ b/contributors/devel/coding-conventions.md @@ -3,7 +3,6 @@ Updated: 5/3/2016 **Table of Contents** -<!-- BEGIN MUNGE: GENERATED_TOC --> - [Coding Conventions](#coding-conventions) - [Code conventions](#code-conventions) @@ -11,7 +10,6 @@ Updated: 5/3/2016 - [Directory and file conventions](#directory-and-file-conventions) - [Coding advice](#coding-advice) -<!-- END MUNGE: GENERATED_TOC --> ## Code conventions @@ -138,8 +136,3 @@ using the system](https://kubernetes.io/docs/user-guide/config-best-practices/) - Go - [Go landmines](https://gist.github.com/lavalamp/4bd23295a9f32706a48f) - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/collab.md b/contributors/devel/collab.md index 4cc24ac1..43416051 100644 --- a/contributors/devel/collab.md +++ b/contributors/devel/collab.md @@ -45,7 +45,3 @@ Maintainers can assign reviews to other maintainers, when appropriate. The assignee becomes the shepherd for that PR and is responsible for merging the PR once they are satisfied with it or else closing it. The assignee might request reviews from non-maintainers. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/community-expectations.md b/contributors/devel/community-expectations.md index b5271197..45200bc1 100644 --- a/contributors/devel/community-expectations.md +++ b/contributors/devel/community-expectations.md @@ -79,9 +79,3 @@ Many thanks in advance to everyone who contributes their time and effort to making Kubernetes both a successful system as well as a successful community. The strength of our software shines in the strengths of each individual community member. Thanks! - - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/component-config-conventions.md b/contributors/devel/component-config-conventions.md new file mode 100644 index 00000000..b4a5a09d --- /dev/null +++ b/contributors/devel/component-config-conventions.md @@ -0,0 +1,221 @@ +# Component Configuration Conventions + +# Objective + +This document concerns the configuration of Kubernetes system components (as +opposed to the configuration of user workloads running on Kubernetes). +Component configuration is a major operational burden for operators of +Kubernetes clusters. To date, much literature has been written on and much +effort expended to improve component configuration. Despite this, the state of +component configuration remains dissonant. This document attempts to aggregate +that literature and propose a set of guidelines that component owners can +follow to improve consistency across the project. + +# Background + +Currently, component configuration is primarily driven through command line +flags. Command line driven configuration poses certain problems which are +discussed below. Attempts to improve component configuration as a whole have +been slow to make progress and have petered out (ref componentconfig api group, +configmap driven config issues). Some component owners have made use case +specific improvements on a per-need basis. Various comments in issues recommend +subsets of best design practice but no coherent, complete story exists. + +## Pain Points of Current Configuration + +Flag based configuration has poor qualities such as: + +1. Flags exist in a flat namespace, hampering the ability to organize them and expose them in helpful documentation. --help becomes useless as a reference as the number of knobs grows. It's impossible to distinguish useful knobs from cruft. +1. Flags can't easily have different values for different instances of a class. To adjust the resync period in the informers of O(n) controllers requires O(n) different flags in a global namespace. +1. Changing a process's command line necessitates a binary restart. This negatively impacts availability. +1. Flags are unsuitable for passing confidential configuration. The command line of a process is available to unprivileged process running in the host pid namespace. +1. Flags are a public API but are unversioned and unversionable. +1. Many arguments against using global variables apply to flags. + +Configuration in general has poor qualities such as: + +1. Configuration changes have the same forward/backward compatibility requirements as releases but rollout/rollback of configuration largely untested. Examples of configuration changes that might break a cluster: kubelet CNI plugin, etcd storage version. +1. Configuration options often exist only to test a specific feature where the default is reasonable for all real use cases. Examples: many sync periods. +1. Configuration options often exist to defer a "hard" design decision and to pay forward the "TODO(someonelse): think critically". +1. Configuration options are often used to workaround deficiencies of the API. For example `--register-with-labels` and `--register-with-taints` could be solved with a node initializer, if initializers existed. +1. Configuration options often exist to take testing shortcuts. There is a mentality that because a feature is opt-in, it can be released as a flag without robust testing. +1. Configuration accumulates new knobs, knobs accumulate new behaviors, knobs are forgotten and bitrot reducing code quality over time. +1. Number of configuration options is inversely proportional to test coverage. The size of the configuration state space grows >O(2^n) with the number of configuration bits. A handful of states in that space are ever tested. +1. Configuration options hamper troubleshooting efforts. On github, users frequently file tickets from environments that are neither consistent nor reproducible. + +## Types Of Configuration + +Configuration can only come from three sources: + +1. Command line flags. +1. API types serialized and stored on disk. +1. API types serialized and stored in the kubernetes API. + +Configuration options can be partitioned along certain lines. To name a few +important partitions: + +1. Bootstrap: This is configuration that is required before the component can contact the API. Examples include the kubeconfig and the filepath to the kubeconfig. +1. Dynamic vs Static: Dynamic config is config that is expected to change as part of normal operations such as a scheduler configuration or a node entering maintenance mode. Static config is config that is unlikely to change over subsequent deployments and even releases of a component. +1. Shared vs Per-Instance: Per-Instance configuration is configuration whose value is unique to the instance that the node runs on (e.g. Kubelet's `--hostname-override`). +1. Feature Gates: Feature gates are configuration options that enable a feature that has been deemed unsafe to enable by default. +1. Request context dependent: Request context dependent config is config that should probably be scoped to an attribute of the request (such as the user). We do a pretty good job of keeping these out of config and in policy objects (e.g. Quota, RBAC) but we could do more (e.g. rate limits). +1. Environment information: This is configuration that is available through downwards and OS APIs, e.g. node name, pod name, number of cpus, IP address. + +# Requirements + +Desired qualities of a configuration solution: + +1. Secure: We need to control who can change configuration. We need to control who can read sensitive configuration. +1. Manageable: We need to control which instances of a component uses which configuration, especially when those instances differ in version. +1. Reliable: Configuration pushes should just work. If they fail, they should fail early in the rollout, rollback config if possible, and alert noisily. +1. Recoverable: We need to be able to update (e.g. rollback) configuration when a component is down. +1. Monitorable: Both humans and computers need to monitor configuration; humans through json interfaces like /configz, computers through interfaces like prometheus /streamz. Confidential configuration needs to be accounted for, but can also be useful to monitor in an unredacted or partially redacted (i.e. hashed) form. +1. Verifiable: We need to be able to verify that a configuration is good. We need to verify the integrity of the received configuration and we need to validate that the encoded configuration state is sensible. +1. Auditable: We need to be able to trace the origin of a configuration change. +1. Accountable: We need to correlate a configuration push with its impact to the system. We need to be able to do this at the time of the push and later when analyzing logs. +1. Available: We should avoid high frequency configuration updates that require service disruption. We need to take into account system component SLA. +1. Scalable: We need to support distributing configuration to O(10,000) components at our current supported scalability limits. +1. Consistent: There should exist conventions that hold across components. +1. Composable: We should favor composition of configuration sources over layering/templating/inheritance. +1. Normalized: Redundant specification of configuration data should be avoided. +1. Testable: We need to be able to test the system under many different configurations. We also need to test configuration changes, both dynamic changes and those that require process restarts. +1. Maintainable: We need to push back on ever increasing cyclomatic complexity in our codebase. Each if statement and function argument added to support a configuration option negatively impacts the maintainability of our code. +1. Evolvable: We need to be able to extend our configuration API like we extend our other user facing APIs. We need to hold our configuration API to the same SLA and deprecation policy of public facing APIs. (e.g. [dynamic admission control](https://github.com/kubernetes/community/pull/611) and [hooks](https://github.com/kubernetes/kubernetes/issues/3585)) + +These don't need to be implemented immediately but are good to keep in mind. At +some point these should be ranked by priority and implemented. + +# Two Part Solution: + +## Part 1: Don't Make It Configuration + +The most effective way to reduce the operational burden of configuration is to +minimize the amount of configuration. When adding a configuration option, ask +whether alternatives might be a better fit. + +1. Policy objects: Create first class Kubernetes objects to encompass how the system should behave. These are especially useful for request context dependent configuration. We do this already in places such as RBAC and ResourceQuota but we could do more such as rate limiting. We should never hardcode groups or usermaps in configuration. +1. API features: Use (or implement) functionality of the API (e.g. think through and implement initializers instead of --register-with-label). Allowing for extension in the right places is a better way to give users control. +1. Feature discovery: Write components that introspect the existing API to decide whether to enable a feature or not. E.g. controller-manager should start an app controller if the app API is available, kubelet should enable zram if zram is set in the node spec. +1. Downwards API: Use the APIs that the OS and pod environment expose directly before opting to pass in new configuration options. +1. const's: If you don't know whether tweaking a value will be useful, make the value const. Only give it a configuration option once there becomes a need to tweak the value at runtime. +1. Autotuning: Build systems that incorporate feedback and do the best thing under the given circumstances. This makes the system more robust. (e.g. prefer congestion control, load shedding, backoff rather than explicit limiting). +1. Avoid feature flags: Turn on features when they are tested and ready for production. Don't use feature flags as a fallback for poorly tested code. +1. Configuration profiles: Instead of allowing individual configuration options to be modified, try to encompass a broader desire as a configuration profile. For example: instead of enabling individual alpha features, have an EnableAlpha option that enables all. Instead of allowing individual controller knobs to be modified, have a TestMode option that sets a broad number of parameters to be suitable for tests. + +## Part 2: Component Configuration + +### Versioning Configuration + +We create configuration API groups per component that live in the source tree of +the component. Each component has its own API group for configuration. +Components will use the same API machinery that we use for other API groups. +Configuration API serialization doesn't have the same performance requirements +as other APIs so much of the codegen can be avoided (e.g. ugorji, generated +conversions) and we can instead fallback to the reflection based implementations +where they exist. + +Configuration API groups for component config should be named according to the +scheme `<component>.config.k8s.io`. The `.config.k8s.io` suffix serves to +disambiguate types of config API groups from served APIs. + +### Retrieving Configuration + +The primary mechanism for retrieving static configuration should be +deserialization from files. For the majority of components (with the possible +exception of the kubelet, see +[here](https://github.com/kubernetes/kubernetes/pull/29459)), these files will +be source from the configmap API and managed by the kubelet. Reliability of +this mechanism is predicated on kubelet checkpointing of pod dependencies. + + +### Structuring Configuration + +Group related options into distinct objects and subobjects. Instead of writing: + + +```yaml +kind: KubeProxyConfiguration +apiVersion: kubeproxy.config.k8s.io/v1beta3 +ipTablesSyncPeriod: 2 +ipTablesConntrackHashSize: 2 +ipTablesConntrackTableSize: 2 +``` + +Write: + +```yaml +kind: KubeProxyConfiguration +apiVersion: kubeproxy.config.k8s.io/v1beta3 +ipTables: + syncPeriod: 2 + conntrack: + hashSize: 2 + tableSize: 2 +``` + +We should avoid passing around full configuration options to deeply constructed +modules. For example, instead of calling NewSomethingController in the +controller-manager with the full controller-manager config, group relevant +config into a subobject and only pass the subobject. We should expose the +smallest possible necessary configuration to the SomethingController. + + +### Handling Different Types Of Configuration + +Above in "Type Of Configuration" we introduce a few ways to partition +configuration options. Environment information, request context depending +configuration, feature gates, and static configuration should be avoided if at +all possible using a configuration alternative. We should maintain separate +objects along these partitions and consider retrieving these configurations +from separate source (i.e. files). For example: kubeconfig (which falls into +the bootstrapping category) should not be part of the main config option (nor +should the filepath to the kubeconfig), per-instance config should be stored +separately from shared config. This allows for composition and obviates the +need for layering/templating solutions. + + +### In-Process Representation Of Configuration + +We should separate structs for flags, serializable config, and runtime config. + +1. Structs for flags should have enough information for the process startup to retrieve its full configuration. Examples include: path the kubeconfig, path to configuration file, namespace and name of configmap to use for configuration. +1. Structs for serializable configuration: This struct contains the full set of options in a serializable form (e.g. to represent an ip address instead of `net.IP`, use `string`). This is the struct that is versioned and serialized to disk using API machinery. +1. Structs for runtime: This struct holds data in the most appropriate format for execution. This field can hold non-serializable types (e.g. have a `kubeClient` field instead of a `kubeConfig` field, store ip addresses as `net.IP`). + +The flag struct is transformed into the configuration struct which is +transformed into the runtime struct. + + +### Migrating Away From Flags + +Migrating to component configuration can happen incrementally (per component). +By versioning each component's API group separately, we can allow each API +group to advance to beta and GA independently. APIs should be approved by +component owners and reviewers familiar with the component configuration +conventions. We can incentivize operators to migrate away from flags by making +new configuration options only available through the component configuration +APIs. + +# Caveats + +Proposed are not laws but guidelines and as such we've favored completeness +over consistency. There will thus be need for exceptions. + +1. Components (especially those that are not self hosted such as the kubelet) will require custom rollout strategies of new config. +1. Pod checkpointing by kubelet would allow this strategy to be simpler to make reliable. + + +# Miscellaneous Consideration + +1. **This document takes intentionally a very zealous stance against configuration.** Often configuration alternatives are not possible in Kubernetes as they are in proprietary software because Kubernetes has to run in diverse environments, with diverse users, managed by diverse operators. +1. More frequent releases of kubernetes would make "skipping the config knob" more enticing because fixing a bad guess at a const wouldn't take O(4 months) best case to rollout. Factoring in our support for old versions, it takes closer to a year. +1. Self-hosting resolves much of the distribution issue (except for maybe the Kubelet) but reliability is predicated on to-be-implemented features such as kubelet checkpointing of pod dependencies and sound operational practices such as incremental rollout of new configuration using Deployments/DaemonSets. +1. Validating config is hard. Fatal logs lead to crash loops and error logs are ignored. Both options are suboptimal. +1. Configuration needs to be updatable when components are down. +1. Naming style guide: + 1. No negatives, e.g. prefer --enable-foo over --disable-foo + 1. Use the active voice +1. We should actually enforce deprecation. Can we have a test that fails when a comment exists beyond its deadline to be removed? See [#44248](https://github.com/kubernetes/kubernetes/issues/44248) +1. Use different implementations of the same interface rather than if statements to toggle features. This makes deprecation and deletion easy, improving maintainability. +1. How does the proposed solution meet the requirements? Which desired qualities are missed? +1. Configuration changes should trigger predictable and reproducible actions. From a given system state and a given component configuration, we should be able to simulate the actions that the system will take. diff --git a/contributors/devel/container-runtime-interface.md b/contributors/devel/container-runtime-interface.md index 6a6a3159..29e96ed2 100644 --- a/contributors/devel/container-runtime-interface.md +++ b/contributors/devel/container-runtime-interface.md @@ -3,9 +3,9 @@ ## What is CRI? CRI (_Container Runtime Interface_) consists of a -[protobuf API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/v1alpha1/runtime/api.proto), +[protobuf API](https://git.k8s.io/kubernetes/pkg/kubelet/apis/cri/v1alpha1/runtime/api.proto), specifications/requirements (to-be-added), -and [libraries](https://github.com/kubernetes/kubernetes/tree/master/pkg/kubelet/server/streaming) +and [libraries](https://git.k8s.io/kubernetes/pkg/kubelet/server/streaming) for container runtimes to integrate with kubelet on a node. CRI is currently in Alpha. In the future, we plan to add more developer tools such as the CRI validation @@ -59,8 +59,8 @@ Below is a mixed list of CRI specifications/requirements, design docs and proposals. We are working on adding more documentation for the API. - [Original proposal](https://github.com/kubernetes/kubernetes/blob/release-1.5/docs/proposals/container-runtime-interface-v1.md) - - [Networking](https://github.com/kubernetes/community/blob/master/contributors/devel/kubelet-cri-networking.md) - - [Container metrics](https://github.com/kubernetes/community/blob/master/contributors/devel/cri-container-stats.md) + - [Networking](/contributors/devel/kubelet-cri-networking.md) + - [Container metrics](/contributors/devel/cri-container-stats.md) - [Exec/attach/port-forward streaming requests](https://docs.google.com/document/d/1OE_QoInPlVCK9rMAx9aybRmgFiVjHpJCHI9LrfdNM_s/edit?usp=sharing) - [Container stdout/stderr logs](https://github.com/kubernetes/kubernetes/blob/release-1.5/docs/proposals/kubelet-cri-logging.md) @@ -134,7 +134,3 @@ default in Kubelet**. - Email: sig-node (kubernetes-sig-node@googlegroups.com) - Slack: https://kubernetes.slack.com/messages/sig-node - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/contributor-cheatsheet.md b/contributors/devel/contributor-cheatsheet.md new file mode 100644 index 00000000..5ec5a3f3 --- /dev/null +++ b/contributors/devel/contributor-cheatsheet.md @@ -0,0 +1,42 @@ +# Kubernetes Cheat Sheet + +A list of common resources when contributing to Kubernetes. + +| Repo | PRs | Issues | Notes | +| ---- | --- | ------ | ----- | +| [Kubernetes](https://github.com/kubernetes/kubernetes) | [PRs](https://github.com/kubernetes/kubernetes/pulls) | [Issues](https://github.com/kubernetes/kubernetes/issues) | [Meeting Notes](http://bit.ly/kubenotes) +| [Community](https://github.com/kubernetes/community) | [PRs](https://github.com/kubernetes/community/pulls) | [Issues](https://github.com/kubernetes/community/issues) | +| [Docs](https://github.com/kubernetes/website) | [PRs](https://github.com/kubernetes/website/pulls) | [Issues](https://github.com/kubernetes/website/issues) + +## Workflow + +- [Gubernator Dashboard - k8s.reviews](https://k8s-gubernator.appspot.com/pr) +- [reviewable.kubernetes.io](https://reviewable.kubernetes.io/reviews#-) +- [Submit Queue](https://submit-queue.k8s.io) +- [Bot commands](https://git.k8s.io/test-infra/commands.md) +- [Release Buckets](http://gcsweb.k8s.io/gcs/kubernetes-release/) +- Developer Guide + - [Cherry Picking Guide](/contributors/devel/cherry-picks.md) - [Queue](http://cherrypick.k8s.io/#/queue) + +## SIGs and Working Groups + +- [Master SIG list](/sig-list.md#master-sig-list) + +## Community + +- [Calendar](https://calendar.google.com/calendar/embed?src=cgnt364vd8s86hr2phapfjc6uk%40group.calendar.google.com) +- [kubernetes-dev](https://groups.google.com/forum/#!forum/kubernetes-dev) +- [kubernetes-users](https://groups.google.com/forum/#!forum/kubernetes-users) +- [Slack channels](http://slack.k8s.io/) +- [StackOverflow](https://stackoverflow.com/questions/tagged/kubernetes) +- [YouTube Channel](https://www.youtube.com/c/KubernetesCommunity/) + +## Tests + +- [Current Test Status](https://prow.k8s.io/) +- [Aggregated Failures](https://storage.googleapis.com/k8s-gubernator/triage/index.html) +- [Test Grid](https://k8s-testgrid.appspot.com/) + +## Other + +- [Developer Statistics](https://devstats.k8s.io) diff --git a/contributors/devel/controllers.md b/contributors/devel/controllers.md index b8addd94..50dada02 100644 --- a/contributors/devel/controllers.md +++ b/contributors/devel/controllers.md @@ -32,7 +32,7 @@ When you're writing controllers, there are few guidelines that will help make su 1. Use `SharedInformers`. `SharedInformers` provide hooks to receive notifications of adds, updates, and deletes for a particular resource. They also provide convenience functions for accessing shared caches and determining when a cache is primed. - Use the factory methods down in https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/informers/factory.go to ensure that you are sharing the same instance of the cache as everyone else. + Use the factory methods down in https://git.k8s.io/kubernetes/staging/src/k8s.io/client-go/informers/factory.go to ensure that you are sharing the same instance of the cache as everyone else. This saves us connections against the API server, duplicate serialization costs server-side, duplicate deserialization costs controller-side, and duplicate caching costs controller-side. @@ -62,7 +62,7 @@ When you're writing controllers, there are few guidelines that will help make su This lets clients know that the controller has processed a resource. Make sure that your controller is the main controller that is responsible for that resource, otherwise if you need to communicate observation via your own controller, you will need to create a different kind of ObservedGeneration in the Status of the resource. -1. Consider using owner references for resources that result in the creation of other resources (eg. a ReplicaSet results in creating Pods). Thus you ensure that children resources are going to be garbage-collected once a resource managed by your controller is deleted. For more information on owner references, read more [here](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/controller-ref.md). +1. Consider using owner references for resources that result in the creation of other resources (eg. a ReplicaSet results in creating Pods). Thus you ensure that children resources are going to be garbage-collected once a resource managed by your controller is deleted. For more information on owner references, read more [here](/contributors/design-proposals/api-machinery/controller-ref.md). Pay special attention in the way you are doing adoption. You shouldn't adopt children for a resource when either the parent or the children are marked for deletion. If you are using a cache for your resources, you will likely need to bypass it with a direct API read in case you observe that an owner reference has been updated for one of the children. Thus, you ensure your controller is not racing with the garbage collector. @@ -74,14 +74,49 @@ Overall, your controller should look something like this: ```go type Controller struct { - // podLister is secondary cache of pods which is used for object lookups - podLister cache.StoreToPodLister + // pods gives cached access to pods. + pods informers.PodLister + podsSynced cache.InformerSynced // queue is where incoming work is placed to de-dup and to allow "easy" // rate limited requeues on errors queue workqueue.RateLimitingInterface } +func NewController(pods informers.PodInformer) *Controller { + c := &Controller{ + pods: pods.Lister(), + podsSynced pods.Informer().HasSynced, + queue: workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "controller-name"), + } + + // register event handlers to fill the queue with pod creations, updates and deletions + pods.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ + AddFunc: func(obj interface{}) { + key, err := cache.MetaNamespaceKeyFunc(obj) + if err == nil { + c.queue.Add(key) + } + }, + UpdateFunc: func(old interface{}, new interface{}) { + key, err := cache.MetaNamespaceKeyFunc(new) + if err == nil { + c.queue.Add(key) + } + }, + DeleteFunc: func(obj interface{}) { + // IndexerInformer uses a delta nodeQueue, therefore for deletes we have to use this + // key function. + key, err := cache.DeletionHandlingMetaNamespaceKeyFunc(obj) + if err == nil { + c.queue.Add(key) + } + }, + },) + + return c +} + func (c *Controller) Run(threadiness int, stopCh chan struct{}) { // don't let panics crash the process defer utilruntime.HandleCrash() @@ -91,7 +126,7 @@ func (c *Controller) Run(threadiness int, stopCh chan struct{}) { glog.Infof("Starting <NAME> controller") // wait for your secondary caches to fill before starting your work - if !framework.WaitForCacheSync(stopCh, c.podStoreSynced) { + if !cache.WaitForCacheSync(stopCh, c.podsSynced) { return } @@ -153,10 +188,4 @@ func (c *Controller) processNextWorkItem() bool { return true } - ``` - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/cri-container-stats.md b/contributors/devel/cri-container-stats.md index b44cfdbf..c1176f05 100644 --- a/contributors/devel/cri-container-stats.md +++ b/contributors/devel/cri-container-stats.md @@ -1,7 +1,7 @@ # Container Runtime Interface: Container Metrics [Container runtime interface -(CRI)](https://github.com/kubernetes/community/blob/master/contributors/devel/container-runtime-interface.md) +(CRI)](/contributors/devel/container-runtime-interface.md) provides an abstraction for container runtimes to integrate with Kubernetes. CRI expects the runtime to provide resource usage statistics for the containers. @@ -12,7 +12,7 @@ Historically Kubelet relied on the [cAdvisor](https://github.com/google/cadvisor library, an open-source project hosted in a separate repository, to retrieve container metrics such as CPU and memory usage. These metrics are then aggregated and exposed through Kubelet's [Summary -API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/stats/v1alpha1/types.go) +API](https://git.k8s.io/kubernetes/pkg/kubelet/apis/stats/v1alpha1/types.go) for the monitoring pipeline (and other components) to consume. Any container runtime (e.g., Docker and Rkt) integrated with Kubernetes needed to add a corresponding package in cAdvisor to support tracking container and image file @@ -23,9 +23,9 @@ progression to augment CRI to serve container metrics to eliminate a separate integration point. *See the [core metrics design -proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/core-metrics-pipeline.md) +proposal](/contributors/design-proposals/instrumentation/core-metrics-pipeline.md) for more information on metrics exposed by Kubelet, and [monitoring -architecture](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/monitoring_architecture.md) +architecture](/contributors/design-proposals/instrumentation/monitoring_architecture.md) for the evolving monitoring pipeline in Kubernetes.* # Container Metrics diff --git a/contributors/devel/development.md b/contributors/devel/development.md index 7cd8ee26..374e10b2 100644 --- a/contributors/devel/development.md +++ b/contributors/devel/development.md @@ -132,7 +132,9 @@ development environment, please [set one up](http://golang.org/doc/code.html). | 1.0 - 1.2 | 1.4.2 | | 1.3, 1.4 | 1.6 | | 1.5, 1.6 | 1.7 - 1.7.5 | -| 1.7+ | 1.8.1 | +| 1.7 | 1.8.1 | +| 1.8 | 1.8.3 | +| 1.9+ | 1.9.1 | Ensure your GOPATH and PATH have been configured in accordance with the Go environment instructions. @@ -144,32 +146,18 @@ images. - The image for cross compiling in [build/build-image/cross]. The `VERSION` file and `Dockerfile`. -- Update [dockerized-e2e-runner.sh] to run a kubekins-e2e with the desired Go - version. This requires pushing the [e2e][e2e-image] and [test][test-image] - images that are `FROM` the desired Go version. +- Update the desired Go version in Dockerfile for the [e2e][e2e-image] and [test][test-image]. + This requires pushing the [e2e][e2e-image] and [test][test-image] images that are `FROM` the desired Go version. - The cross tag `KUBE_BUILD_IMAGE_CROSS_TAG` in [build/common.sh]. #### Dependency management -Kubernetes build/test scripts use [`godep`](https://github.com/tools/godep) to -manage dependencies. - -```sh -go get -u github.com/tools/godep -``` - -The Godep version that Kubernetes is using is listed in `Godep/Godep.json` (in -the kubernetes repo root). See what version you are running with this command: - -```sh -godep version -``` - -Developers planning to manage dependencies in the `vendor/` tree may want to -explore alternative environment setups. See [using godep to manage -dependencies](godep.md). +Kubernetes uses [`godep`](https://github.com/tools/godep) to manage +dependencies. +Developers who need to manage dependencies in the `vendor/` tree should read +the docs on [using godep to manage dependencies](godep.md). ## Build with Bazel/Gazel @@ -178,231 +166,15 @@ Building with Bazel is currently experimental. For more information, see [Build with Bazel]. +## GitHub workflow -## Workflow - - - -### 1 Fork in the cloud - -1. Visit https://github.com/kubernetes/kubernetes -2. Click `Fork` button (top right) to establish a cloud-based fork. - -### 2 Clone fork to local storage - -Per Go's [workspace instructions][go-workspace], place Kubernetes' code on your -`GOPATH` using the following cloning procedure. - -Define a local working directory: - -```sh -# If your GOPATH has multiple paths, pick -# just one and use it instead of $GOPATH here. -# You must follow exactly this pattern, -# neither `$GOPATH/src/github.com/${your github profile name/` -# nor any other pattern will work. -working_dir=$GOPATH/src/k8s.io -``` - -> If you already do Go development on github, the `k8s.io` directory -> will be a sibling to your existing `github.com` directory. - -Set `user` to match your github profile name: - -```sh -user={your github profile name} -``` - -Both `$working_dir` and `$user` are mentioned in the figure above. - -Create your clone: - -```sh -mkdir -p $working_dir -cd $working_dir -git clone https://github.com/$user/kubernetes.git -# or: git clone git@github.com:$user/kubernetes.git - -cd $working_dir/kubernetes -git remote add upstream https://github.com/kubernetes/kubernetes.git -# or: git remote add upstream git@github.com:kubernetes/kubernetes.git - -# Never push to upstream master -git remote set-url --push upstream no_push - -# Confirm that your remotes make sense: -git remote -v -``` - -### 3 Branch - -Get your local master up to date: - -```sh -cd $working_dir/kubernetes -git fetch upstream -git checkout master -git rebase upstream/master -``` - -Branch from it: -```sh -git checkout -b myfeature -``` - -Then edit code on the `myfeature` branch. - -#### Build - -*Note:* If you are using `CDPATH`, you must either start it with a leading colon, or unset the variable. The make rules and scripts to build require the current directory to come first on the CD search path in order to properly navigate between directories. - -```sh -cd $working_dir/kubernetes -make -``` - -To remove the limit on the number of errors the Go compiler reports (default -limit is 10 errors): -```sh -make GOGCFLAGS="-e" -``` - -To build with optimizations disabled (enables use of source debug tools): - -```sh -make GOGCFLAGS="-N -l" -``` - -To build binaries for all platforms: - -```sh -make cross -``` - -#### Install etcd - -```sh -hack/install-etcd.sh -``` - -#### Test - -```sh -cd $working_dir/kubernetes - -# Run all the presubmission verification. Then, run a specific update script (hack/update-*.sh) -# for each failed verification. For example: -# hack/update-gofmt.sh (to make sure all files are correctly formatted, usually needed when you add new files) -# hack/update-bazel.sh (to update bazel build related files, usually needed when you add or remove imports) -make verify - -# Alternatively, run all update scripts to avoid fixing verification failures one by one. -make update - -# Run every unit test -make test - -# Run package tests verbosely -make test WHAT=./pkg/api/helper GOFLAGS=-v - -# Run integration tests, requires etcd -# For more info, visit https://github.com/kubernetes/community/blob/master/contributors/devel/testing.md#integration-tests -make test-integration - -# Run e2e tests by building test binaries, turn up a test cluster, run all tests, and tear the cluster down -# Equivalent to: go run hack/e2e.go -- -v --build --up --test --down -# Note: running all e2e tests takes a LONG time! To run specific e2e tests, visit: -# https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md#building-kubernetes-and-running-the-tests -make test-e2e -``` - -See the [testing guide](testing.md) and [end-to-end tests](e2e-tests.md) -for additional information and scenarios. - -Run `make help` for additional information on these make targets. - -### 4 Keep your branch in sync - -```sh -# While on your myfeature branch -git fetch upstream -git rebase upstream/master -``` - -### 5 Commit - -Commit your changes. - -```sh -git commit -``` -Likely you go back and edit/build/test some more then `commit --amend` -in a few cycles. - -### 6 Push - -When ready to review (or just to establish an offsite backup or your work), -push your branch to your fork on `github.com`: - -```sh -git push -f ${your_remote_name} myfeature -``` - -### 7 Create a pull request - -1. Visit your fork at https://github.com/$user/kubernetes -2. Click the `Compare & Pull Request` button next to your `myfeature` branch. -3. Check out the pull request [process](pull-requests.md) for more details. - -_If you have upstream write access_, please refrain from using the GitHub UI for -creating PRs, because GitHub will create the PR branch inside the main -repository rather than inside your fork. - -#### Get a code review - -Once your pull request has been opened it will be assigned to one or more -reviewers. Those reviewers will do a thorough code review, looking for -correctness, bugs, opportunities for improvement, documentation and comments, -and style. - -Commit changes made in response to review comments to the same branch on your -fork. - -Very small PRs are easy to review. Very large PRs are very difficult to review. -At the assigned reviewer's discretion, a PR may be switched to use -[Reviewable](https://reviewable.k8s.io) instead. Once a PR is switched to -Reviewable, please ONLY send or reply to comments through Reviewable. Mixing -code review tools can be very confusing. - -See [Faster Reviews](faster_reviews.md) for some thoughts on how to streamline -the review process. - - -#### Squash and Merge - -Upon merge (by either you or your reviewer), all commits left on the review -branch should represent meaningful milestones or units of work. Use commits to -add clarity to the development and review process. - -Before merging a PR, squash any _fix review feedback_, _typo_, and _rebased_ -sorts of commits. - -It is not imperative that every commit in a PR compile and pass tests -independently, but it is worth striving for. - -For mass automated fixups (e.g. automated doc formatting), use one or more -commits for the changes to tooling and a final commit to apply the fixup en -masse. This makes reviews easier. +To check out code to work on, please refer to [this guide](/contributors/guide/github-workflow.md). -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> [OS X GNU tools]: https://www.topbug.net/blog/2013/04/14/install-and-use-gnu-command-line-tools-in-mac-os-x -[build/build-image/cross]: https://github.com/kubernetes/kubernetes/blob/master/build/build-image/cross -[build/common.sh]: https://github.com/kubernetes/kubernetes/blob/master/build/common.sh -[dockerized-e2e-runner.sh]: https://github.com/kubernetes/test-infra/blob/master/jenkins/dockerized-e2e-runner.sh -[e2e-image]: https://github.com/kubernetes/test-infra/tree/master/jenkins/e2e-image +[build/build-image/cross]: https://git.k8s.io/kubernetes/build/build-image/cross +[build/common.sh]: https://git.k8s.io/kubernetes/build/common.sh +[e2e-image]: https://git.k8s.io/test-infra/jenkins/e2e-image [etcd-latest]: https://coreos.com/etcd/docs/latest [etcd-install]: testing.md#install-etcd-dependency <!-- https://github.com/coreos/etcd/releases --> @@ -411,5 +183,5 @@ masse. This makes reviews easier. [kubectl user guide]: https://kubernetes.io/docs/user-guide/kubectl [kubernetes.io]: https://kubernetes.io [mercurial]: http://mercurial.selenic.com/wiki/Download -[test-image]: https://github.com/kubernetes/test-infra/tree/master/jenkins/test-image +[test-image]: https://git.k8s.io/test-infra/jenkins/test-image [Build with Bazel]: bazel.md diff --git a/contributors/devel/e2e-node-tests.md b/contributors/devel/e2e-node-tests.md index 8bbf2dd7..4f3327cb 100644 --- a/contributors/devel/e2e-node-tests.md +++ b/contributors/devel/e2e-node-tests.md @@ -137,7 +137,7 @@ make test-e2e-node REMOTE=true IMAGE_PROJECT="<name-of-project-with-images>" IMA ``` Setting up your own host image may require additional steps such as installing etcd or docker. See -[setup_host.sh](https://github.com/kubernetes/kubernetes/tree/master/test/e2e_node/environment/setup_host.sh) for common steps to setup hosts to run node tests. +[setup_host.sh](https://git.k8s.io/kubernetes/test/e2e_node/environment/setup_host.sh) for common steps to setup hosts to run node tests. ## Create instances using a different instance name prefix @@ -210,10 +210,10 @@ make test-e2e-node TEST_ARGS='--kubelet-flags="--network-plugin= --network-plugi ## Additional QoS Cgroups Hierarchy level testing -For testing with the QoS Cgroup Hierarchy enabled, you can pass --experimental-cgroups-per-qos flag as an argument into Ginkgo using TEST_ARGS +For testing with the QoS Cgroup Hierarchy enabled, you can pass --cgroups-per-qos flag as an argument into Ginkgo using TEST_ARGS ```sh -make test_e2e_node TEST_ARGS="--experimental-cgroups-per-qos=true" +make test_e2e_node TEST_ARGS="--cgroups-per-qos=true" ``` # Notes on tests run by the Kubernetes project during pre-, post- submit. @@ -223,11 +223,7 @@ the bottom of the comments section. To re-run just the node e2e tests from the `@k8s-bot node e2e test this issue: #<Flake-Issue-Number or IGNORE>` and **include a link to the test failure logs if caused by a flake.** -The PR builder runs tests against the images listed in [jenkins-pull.properties](https://github.com/kubernetes/kubernetes/tree/master/test/e2e_node/jenkins/jenkins-pull.properties) +The PR builder runs tests against the images listed in [jenkins-pull.properties](https://git.k8s.io/kubernetes/test/e2e_node/jenkins/jenkins-pull.properties) -The post submit tests run against the images listed in [jenkins-ci.properties](https://github.com/kubernetes/kubernetes/tree/master/test/e2e_node/jenkins/jenkins-ci.properties) +The post submit tests run against the images listed in [jenkins-ci.properties](https://git.k8s.io/kubernetes/test/e2e_node/jenkins/jenkins-ci.properties) - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/e2e-tests.md b/contributors/devel/e2e-tests.md index 8ccd77f1..cf4127d0 100644 --- a/contributors/devel/e2e-tests.md +++ b/contributors/devel/e2e-tests.md @@ -1,13 +1,10 @@ # End-to-End Testing in Kubernetes -Updated: 5/3/2016 - **Table of Contents** -<!-- BEGIN MUNGE: GENERATED_TOC --> - [End-to-End Testing in Kubernetes](#end-to-end-testing-in-kubernetes) - [Overview](#overview) - - [Building and Running the Tests](#building-and-running-the-tests) + - [Building Kubernetes and Running the Tests](#building-kubernetes-and-running-the-tests) - [Cleaning up](#cleaning-up) - [Advanced testing](#advanced-testing) - [Installing/updating kubetest](#installingupdating-kubetest) @@ -40,7 +37,6 @@ Updated: 5/3/2016 - [Performance Evaluation](#performance-evaluation) - [One More Thing](#one-more-thing) -<!-- END MUNGE: GENERATED_TOC --> ## Overview @@ -150,7 +146,7 @@ go run hack/e2e.go -- -v --down The logic in `e2e.go` moved out of the main kubernetes repo to test-infra. The remaining code in `hack/e2e.go` installs `kubetest` and sends it flags. -It now lives in [kubernetes/test-infra/kubetest](https://github.com/kubernetes/test-infra/tree/master/kubetest). +It now lives in [kubernetes/test-infra/kubetest](https://git.k8s.io/test-infra/kubetest). By default `hack/e2e.go` updates and installs `kubetest` once per day. Control the updater behavior with the `--get` and `--old` flags: The `--` flag separates updater and kubetest flags (kubetest flags on the right). @@ -229,10 +225,6 @@ when a failure occurs --kubeconfig="": Path to kubeconfig containing embedded authinfo. ---prom-push-gateway="": The URL to prometheus gateway, so that metrics can be -pushed during e2es and scraped by prometheus. Typically something like -127.0.0.1:9091. - --provider="": The name of the Kubernetes provider (gce, gke, local, vagrant, etc.) @@ -342,7 +334,7 @@ Next, specify the docker repository where your ci images will be pushed. * Push the federation container images ```sh - $ build/push-federation-images.sh + $ federation/develop/push-federation-images.sh ``` #### Deploy federation control plane @@ -393,8 +385,8 @@ to gather logs. This script requires that the cluster provider supports ssh. Assuming it does, running: -``` -cluster/log-dump.sh <directory> +```sh +$ federation/cluster/log-dump.sh <directory> ``` will ssh to the master and all nodes and download a variety of useful logs to @@ -454,7 +446,7 @@ similarly enough to older versions. The general strategy is to cover the follow same version (e.g. a cluster upgraded to v1.3 passes the same v1.3 tests as a newly-created v1.3 cluster). -[hack/e2e-runner.sh](https://github.com/kubernetes/test-infra/blob/master/jenkins/e2e-image/e2e-runner.sh) is +[hack/e2e-runner.sh](https://git.k8s.io/test-infra/jenkins/e2e-image/e2e-runner.sh) is the authoritative source on how to run version-skewed tests, but below is a quick-and-dirty tutorial. @@ -577,7 +569,7 @@ breaking changes, it does *not* block the merge-queue, and thus should run in some separate test suites owned by the feature owner(s) (see [Continuous Integration](#continuous-integration) below). -Every test should be owned by a [SIG](https://github.com/kubernetes/community/blob/master/sig-list.md), +Every test should be owned by a [SIG](/sig-list.md), and have a corresponding `[sig-<name>]` label. ### Viper configuration and hierarchichal test parameters. @@ -590,7 +582,7 @@ To use viper, rather than flags, to configure your tests: - Just add "e2e.json" to the current directory you are in, and define parameters in it... i.e. `"kubeconfig":"/tmp/x"`. -Note that advanced testing parameters, and hierarchichally defined parameters, are only defined in viper, to see what they are, you can dive into [TestContextType](https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/test_context.go). +Note that advanced testing parameters, and hierarchichally defined parameters, are only defined in viper, to see what they are, you can dive into [TestContextType](https://git.k8s.io/kubernetes/test/e2e/framework/test_context.go). In time, it is our intent to add or autogenerate a sample viper configuration that includes all e2e parameters, to ship with kubernetes. @@ -664,7 +656,7 @@ A quick overview of how we run e2e CI on Kubernetes. We run a battery of `e2e` tests against `HEAD` of the master branch on a continuous basis, and block merges via the [submit queue](http://submit-queue.k8s.io/) on a subset of those tests if they fail (the -subset is defined in the [munger config](https://github.com/kubernetes/test-infra/tree/master/mungegithub/mungers/submit-queue.go) +subset is defined in the [munger config](https://git.k8s.io/test-infra/mungegithub/mungers/submit-queue.go) via the `jenkins-jobs` flag; note we also block on `kubernetes-build` and `kubernetes-test-go` jobs for build and unit and integration tests). @@ -721,7 +713,7 @@ the existing suite as a guide. TODO(#20357): Create a self-documented example which has been disabled, but can be copied to create new tests and outlines the capabilities and libraries used. -When writing a test, consult #kinds_of_tests above to determine how your test +When writing a test, consult #kinds-of-tests above to determine how your test should be marked, (e.g. `[Slow]`, `[Serial]`; remember, by default we assume a test can run in parallel with other tests!). @@ -740,7 +732,7 @@ label, and will be incorporated into our core suites. If tests are not expected to pass by default, (e.g. they require a special environment such as added quota,) they should remain with the `[Feature:.+]` label, and the suites that run them should be incorporated into the -[munger config](https://github.com/kubernetes/test-infra/tree/master/mungegithub/mungers/submit-queue.go) +[munger config](https://git.k8s.io/test-infra/mungegithub/mungers/submit-queue.go) via the `jenkins-jobs` flag. Occasionally, we'll want to add tests to better exercise features that are @@ -809,9 +801,3 @@ metrics that kubernetes provides. You should also know the [testing conventions](coding-conventions.md#testing-conventions). **HAPPY TESTING!** - - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/faster_reviews.md b/contributors/devel/faster_reviews.md index 50223f68..37de43cb 100644 --- a/contributors/devel/faster_reviews.md +++ b/contributors/devel/faster_reviews.md @@ -1,7 +1,3 @@ # How to get faster PR reviews Merged with the [pull requests doc](pull-requests.md#best-practices-for-faster-reviews). - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> - []() - <!-- END MUNGE: GENERATED_ANALYTICS -->
\ No newline at end of file diff --git a/contributors/devel/flaky-tests.md b/contributors/devel/flaky-tests.md index 9656bd5f..f06bb078 100644 --- a/contributors/devel/flaky-tests.md +++ b/contributors/devel/flaky-tests.md @@ -35,7 +35,7 @@ discoverable from the issue. link is nice but strictly optional: not only does it expire more quickly, it's not accessible to non-Googlers. -## Finding filed flaky test cases +## Finding failed flaky test cases Find flaky tests issues on GitHub under the [kind/flake issue label][flake]. There are significant numbers of flaky tests reported on a regular basis and P2 @@ -188,7 +188,3 @@ exited -1, since that's what happens when you stop the replication controller. Happy flake hunting! - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/flexvolume.md b/contributors/devel/flexvolume.md index 41d7e54c..52d42ccf 100644 --- a/contributors/devel/flexvolume.md +++ b/contributors/devel/flexvolume.md @@ -1,16 +1,23 @@ # Flexvolume -Flexvolume enables users to write their own drivers and add support for their volumes in Kubernetes. Vendor drivers should be installed in the volume plugin path on every Kubelet node and on master node(s) if "--enable-controller-attach-detach" Kubelet option is enabled. +Flexvolume enables users to write their own drivers and add support for their volumes in Kubernetes. Vendor drivers should be installed in the volume plugin path on every Kubelet node and on master node(s) if `--enable-controller-attach-detach` Kubelet option is enabled. -*Note: Flexvolume is an alpha feature and is most likely to change in future* +Flexvolume is a GA feature from Kubernetes 1.8 release onwards. ## Prerequisites -Install the vendor driver on all nodes (also on master nodes if "--enable-controller-attach-detach" Kubelet option is enabled) in the plugin path. Path for installing the plugin: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/\<vendor~driver\>/\<driver\> +Install the vendor driver on all nodes (also on master nodes if "--enable-controller-attach-detach" Kubelet option is enabled) in the plugin path. Path for installing the plugin: `<plugindir>/<vendor~driver>/<driver>`. The default plugin directory is `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/`. It can be changed in kubelet via the `--volume-plugin-dir` flag, and in controller manager via the `--flex-volume-plugin-dir` flag. -For example to add a 'cifs' driver, by vendor 'foo' install the driver at: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/\<foo~cifs\>/cifs +For example to add a `cifs` driver, by vendor `foo` install the driver at: `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/foo~cifs/cifs` -The vendor and driver names must match flexVolume.driver in the volume spec, with '~' replaced with '/'. For example, if flexVolume.driver is set to "foo/cifs", then the vendor is "foo", and driver is "cifs". +The vendor and driver names must match flexVolume.driver in the volume spec, with '~' replaced with '/'. For example, if `flexVolume.driver` is set to `foo/cifs`, then the vendor is `foo`, and driver is `cifs`. + +## Dynamic Plugin Discovery +Beginning in v1.8, Flexvolume supports the ability to detect drivers on the fly. Instead of requiring drivers to exist at system initialization time or having to restart kubelet or controller manager, drivers can be installed, upgraded/downgraded, and uninstalled while the system is running. +For more information, please refer to the [design document](/contributors/design-proposals/storage/flexvolume-deployment.md). + +## Automated Plugin Installation/Upgrade +One possible way to install and upgrade your Flexvolume drivers is by using a DaemonSet. See [Recommended Driver Deployment Method](/contributors/design-proposals/storage/flexvolume-deployment.md#recommended-driver-deployment-method) for details. ## Plugin details The plugin expects the following call-outs are implemented for the backend drivers. Some call-outs are optional. Call-outs are invoked from the Kubelet & the Controller manager nodes. @@ -27,13 +34,6 @@ See [Driver output](#driver-output) for the capabilities map format. <driver executable> init ``` -#### Get volume name: -Get a cluster wide unique volume name for the volume. Called from both Kubelet & Controller manager. - -``` -<driver executable> getvolumename <json options> -``` - #### Attach: Attach the volume specified by the given spec on the given host. On success, returns the device path where the device is attached on the node. Nodename param is only valid/relevant if "--enable-controller-attach-detach" Kubelet option is enabled. Called from both Kubelet & Controller manager. @@ -50,7 +50,7 @@ Detach the volume from the Kubelet node. Nodename param is only valid/relevant i ``` #### Wait for attach: -Wait for the volume to be attached on the remote node. On success, the path to the device is returned. Called from both Kubelet & Controller manager. +Wait for the volume to be attached on the remote node. On success, the path to the device is returned. Called from both Kubelet & Controller manager. The timeout should be 10m (based on https://git.k8s.io/kubernetes/pkg/kubelet/volumemanager/volume_manager.go#L88 ) ``` <driver executable> waitforattach <mount device> <json options> @@ -131,11 +131,8 @@ Note: Secrets are passed only to "mount/unmount" call-outs. See [nginx.yaml] & [nginx-nfs.yaml] for a quick example on how to use Flexvolume in a pod. -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> -[lvm]: https://github.com/kubernetes/kubernetes/blob/master/examples/volumes/flexvolume/lvm -[nfs]: https://github.com/kubernetes/kubernetes/blob/master/examples/volumes/flexvolume/nfs -[nginx.yaml]: https://github.com/kubernetes/kubernetes/blob/master/examples/volumes/flexvolume/nginx.yaml -[nginx-nfs.yaml]: https://github.com/kubernetes/kubernetes/blob/master/examples/volumes/flexvolume/nginx-nfs.yaml +[lvm]: https://git.k8s.io/kubernetes/examples/volumes/flexvolume/lvm +[nfs]: https://git.k8s.io/kubernetes/examples/volumes/flexvolume/nfs +[nginx.yaml]: https://git.k8s.io/kubernetes/examples/volumes/flexvolume/nginx.yaml +[nginx-nfs.yaml]: https://git.k8s.io/kubernetes/examples/volumes/flexvolume/nginx-nfs.yaml diff --git a/contributors/devel/generating-clientset.md b/contributors/devel/generating-clientset.md index 9590dbc0..7a47aeb8 100644 --- a/contributors/devel/generating-clientset.md +++ b/contributors/devel/generating-clientset.md @@ -1,12 +1,29 @@ # Generation and release cycle of clientset -Client-gen is an automatic tool that generates [clientset](../design-proposals/client-package-structure.md#high-level-client-sets) based on API types. This doc introduces the use the client-gen, and the release cycle of the generated clientsets. +Client-gen is an automatic tool that generates [clientset](../design-proposals/api-machinery/client-package-structure.md#high-level-client-sets) based on API types. This doc introduces the use the client-gen, and the release cycle of the generated clientsets. ## Using client-gen The workflow includes three steps: -**1.** Marking API types with tags: in `pkg/apis/${GROUP}/${VERSION}/types.go`, mark the types (e.g., Pods) that you want to generate clients for with the `// +genclient=true` tag. If the resource associated with the type is not namespace scoped (e.g., PersistentVolume), you need to append the `nonNamespaced=true` tag as well. +**1.** Marking API types with tags: in `pkg/apis/${GROUP}/${VERSION}/types.go`, mark the types (e.g., Pods) that you want to generate clients for with the `// +genclient` tag. If the resource associated with the type is not namespace scoped (e.g., PersistentVolume), you need to append the `// +genclient:nonNamespaced` tag as well. + +The following `// +genclient` are supported: + +- `// +genclient` - generate default client verb functions (*create*, *update*, *delete*, *get*, *list*, *update*, *patch*, *watch* and depending on the existence of `.Status` field in the type the client is generated for also *updateStatus*). +- `// +genclient:nonNamespaced` - all verb functions are generated without namespace. +- `// +genclient:onlyVerbs=create,get` - only listed verb functions will be generated. +- `// +genclient:skipVerbs=watch` - all default client verb functions will be generated **except** *watch* verb. +- `// +genclient:noStatus` - skip generation of *updateStatus* verb even thought the `.Status` field exists. + +In some cases you want to generate non-standard verbs (eg. for sub-resources). To do that you can use the following generator tag: + +- `// +genclient:method=Scale,verb=update,subresource=scale,input=k8s.io/api/extensions/v1beta1.Scale,result=k8s.io/api/extensions/v1beta1.Scale` - in this case a new function `Scale(string, *v1beta.Scale) *v1beta.Scale` will be added to the default client and the body of the function will be based on the *update* verb. The optional *subresource* argument will make the generated client function use subresource `scale`. Using the optional *input* and *result* arguments you can override the default type with a custom type. If the import path is not given, the generator will assume the type exists in the same package. + +In addition, the following optional tags influence the client generation: + +- `// +groupName=policy.authorization.k8s.io` – used in the fake client as the full group name (defaults to the package name), +- `// +groupGoName=AuthorizationPolicy` – a CamelCase Golang identifier to de-conflict groups with non-unique prefixes like `policy.authorization.k8s.io` and `policy.k8s.io`. These would lead to two `Policy()` methods in the clientset otherwise (defaults to the upper-case first segement of the group name). **2a.** If you are developing in the k8s.io/kubernetes repository, you just need to run hack/update-codegen.sh. @@ -16,7 +33,7 @@ The workflow includes three steps: $ client-gen --input="api/v1,extensions/v1beta1" --clientset-name="my_release" ``` -**3.** ***Adding expansion methods***: client-gen only generates the common methods, such as CRUD. You can manually add additional methods through the expansion interface. For example, this [file](https://github.com/kubernetes/kubernetes/tree/master/pkg/client/clientset_generated/clientset/typed/core/v1/pod_expansion.go) adds additional methods to Pod's client. As a convention, we put the expansion interface and its methods in file ${TYPE}_expansion.go. In most cases, you don't want to remove existing expansion files. So to make life easier, instead of creating a new clientset from scratch, ***you can copy and rename an existing clientset (so that all the expansion files are copied)***, and then run client-gen. +**3.** ***Adding expansion methods***: client-gen only generates the common methods, such as CRUD. You can manually add additional methods through the expansion interface. For example, this [file](https://git.k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/typed/core/internalversion/pod_expansion.go) adds additional methods to Pod's client. As a convention, we put the expansion interface and its methods in file ${TYPE}_expansion.go. In most cases, you don't want to remove existing expansion files. So to make life easier, instead of creating a new clientset from scratch, ***you can copy and rename an existing clientset (so that all the expansion files are copied)***, and then run client-gen. ## Output of client-gen @@ -26,13 +43,8 @@ $ client-gen --input="api/v1,extensions/v1beta1" --clientset-name="my_release" ## Released clientsets -If you are contributing code to k8s.io/kubernetes, try to use the generated clientset [here](https://github.com/kubernetes/kubernetes/tree/master/pkg/client/clientset_generated/clientset). +If you are contributing code to k8s.io/kubernetes, try to use the generated clientset [here](https://git.k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset). If you need a stable Go client to build your own project, please refer to the [client-go repository](https://github.com/kubernetes/client-go). We are migrating k8s.io/kubernetes to use client-go as well, see issue [#35159](https://github.com/kubernetes/kubernetes/issues/35159). - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/getting-builds.md b/contributors/devel/getting-builds.md index 86563390..0ae7031b 100644 --- a/contributors/devel/getting-builds.md +++ b/contributors/devel/getting-builds.md @@ -46,7 +46,3 @@ Example installation: $ curl -sSL https://storage.googleapis.com/pub/gsutil.tar.gz | sudo tar -xz -C /usr/local/src $ sudo ln -s /usr/local/src/gsutil/gsutil /usr/bin/gsutil ``` - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/go-code.md b/contributors/devel/go-code.md index 2af055f4..80ffa9d5 100644 --- a/contributors/devel/go-code.md +++ b/contributors/devel/go-code.md @@ -1,7 +1,7 @@ # Kubernetes Go Tools and Tips -Kubernetes is one of the largest open source Go projects, so good tooling a solid understanding of -Go is critical to Kubernetes development. This document provides a collection of resources, tools +Kubernetes is one of the largest open source Go projects. Good tooling and a solid understanding of +Go is critical to Kubernetes development. This document provides a collection of resources, tools, and tips that our developers have found useful. ## Recommended Reading @@ -16,17 +16,13 @@ and tips that our developers have found useful. ## Recommended Tools -- [godep](https://github.com/tools/godep) - Used for Kubernetes dependency management. See also [Kubernetes godep and dependency management](development.md#godep-and-dependency-management) +- [godep](https://github.com/tools/godep) - Used for Kubernetes dependency management. See also [Kubernetes godep and dependency management](development.md#godep-and-dependency-management). - [Go Version Manager](https://github.com/moovweb/gvm) - A handy tool for managing Go versions. - [godepq](https://github.com/google/godepq) - A tool for analyzing go import trees. ## Go Tips -- [Godoc bookmarklet](https://gist.github.com/timstclair/c891fb8aeb24d663026371d91dcdb3fc) - navigate from a github page to the corresponding godoc page. +- [Godoc bookmarklet](https://gist.github.com/timstclair/c891fb8aeb24d663026371d91dcdb3fc) - Navigate from a github page to the corresponding godoc page. - Consider making a separate Go tree for each project, which can make overlapping dependency management much easier. Remember to set the `$GOPATH` correctly! Consider [scripting](https://gist.github.com/timstclair/17ca792a20e0d83b06dddef7d77b1ea0) this. -- Emacs users - setup [go-mode](https://github.com/dominikh/go-mode.el) +- Emacs users - setup [go-mode](https://github.com/dominikh/go-mode.el). - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/godep.md b/contributors/devel/godep.md index 3b9ca7fd..c997a1ca 100644 --- a/contributors/devel/godep.md +++ b/contributors/devel/godep.md @@ -1,130 +1,189 @@ # Using godep to manage dependencies This document is intended to show a way for managing `vendor/` tree dependencies -in Kubernetes. If you are not planning on managing `vendor` dependencies go here -[Godep dependency management](development.md#godep-dependency-management). +in Kubernetes. If you do not need to manage vendored dependencies, you probably +do not need to read this. -## Alternate GOPATH for installing and using godep +## Background -There are many ways to build and host Go binaries. Here is one way to get -utilities like `godep` installed: +As a tool, `godep` leaves much to be desired. It builds on `go get`, and adds +the ability to pin dependencies to exact git version. The `go get` tool itself +doesn't have any concept of versions, and tends to blow up if it finds a git +repo synced to anything but `master`, but that is exactly the state that +`godep` leaves repos. This is a recipe for frustration when people try to use +the tools. -Create a new GOPATH just for your go tools and install godep: +This doc will focus on predictability and reproducibility. + +## Theory of operation + +The `go` toolchain assumes a global workspace that hosts all of your Go code. + +The `godep` tool operates by first "restoring" dependencies into your `$GOPATH`. +This reads the `Godeps.json` file, downloads all of the dependencies from the +internet, and syncs them to the specified revisions. You can then make +changes - sync to different revisions or edit Kubernetes code to use new +dependencies (and satisfy them with `go get`). When ready, you tell `godep` to +"save" everything, which it does by walking the Kubernetes code, finding all +required dependencies, copying them from `$GOPATH` into the `vendor/` directory, +and rewriting `Godeps.json`. + +This does not work well, when combined with a global Go workspace. Instead, we +will set up a private workspace for this process. + +The Kubernetes build process uses this same technique, and offers a tool called +`run-in-gopath.sh` which sets up and switches to a local, private workspace, +including setting up `$GOPATH` and `$PATH`. If you wrap commands with this +tool, they will use the private workspace, which will not conflict with other +projects and is easily cleaned up and recreated. + +To see this in action, you can run an interactive shell in this environment: ```sh -export GOPATH=$HOME/go-tools -mkdir -p $GOPATH -go get -u github.com/tools/godep +# Run a shell, but don't run your own shell initializations. +hack/run-in-gopath.sh bash --norc --noprofile ``` -Add this $GOPATH/bin to your path. Typically you'd add this to your ~/.profile: +## Restoring deps + +To extract and download dependencies into `$GOPATH` we provide a script: +`hack/godep-restore.sh`. If you run this tool, it will restore into your own +`$GOPATH`. If you wrap it in `run-in-gopath.sh` it will restore into your +`_output/` directory. ```sh -export GOPATH=$HOME/go-tools -export PATH=$PATH:$GOPATH/bin +hack/run-in-gopath.sh hack/godep-restore.sh ``` -## Using godep +This script will try to optimize what it needs to download, and if it seems the +dependencies are all present already, it will return very quickly. + +If there's every any doubt about the correctness of your dependencies, you can +simply `make clean` or `rm -rf _output`, and run it again. -Here's a quick walkthrough of one way to use godeps to add or update a -Kubernetes dependency into `vendor/`. For more details, please see the -instructions in [godep's documentation](https://github.com/tools/godep). +Now you should have a clean copy of all of the Kubernetes dependencies. -1) Devote a directory to this endeavor: +## Making changes -_Devoting a separate directory is not strictly required, but it is helpful to -separate dependency updates from other changes._ +The most common things people need to do with deps are add and update them. +These are similar but different. + +### Adding a dep + +For the sake of examples, consider that we have discovered a wonderful Go +library at `example.com/go/frob`. The first thing you need to do is get that +code into your workspace: ```sh -export KPATH=$HOME/code/kubernetes -mkdir -p $KPATH/src/k8s.io -cd $KPATH/src/k8s.io -git clone https://github.com/$YOUR_GITHUB_USERNAME/kubernetes.git # assumes your fork is 'kubernetes' -# Or copy your existing local repo here. IMPORTANT: making a symlink doesn't work. +hack/run-in-gopath.sh go get -d example.com/go/frob ``` -2) Set up your GOPATH. +This will fetch, but not compile (omit the `-d` if you want to compile it now), +the library into your private `$GOPATH`. It will pull whatever the default +revision of that library is, typically the `master` branch for git repositories. +If this is not the revision you need, you can change it, for example to +`v1.0.0`: + +```sh +hack/run-in-gopath.sh bash -c 'git -C $GOPATH/src/example.com/go/frob checkout v1.0.0' +``` + +Now that the code is present, you can start to use it in Kubernetes code. +Because it is in your private workspace's `$GOPATH`, it might not be part of +your own `$GOPATH`, so tools like `goimports` might not find it. This is an +unfortunate side-effect of this process. You can either add the whole private +workspace to your own `$GOPATH` or you can `go get` the library into your own +`$GOPATH` until it is properly vendored into Kubernetes. + +Another possible complication is a dep that uses `gopdep` itself. In that case, +you need to restore its dependencies, too: ```sh -# This will *not* let your local builds see packages that exist elsewhere on your system. -export GOPATH=$KPATH +hack/run-in-gopath.sh bash -c 'cd $GOPATH/src/example.com/go/frob && godep restore' ``` -3) Populate your new GOPATH. +If the transitive deps collide with Kubernetes deps, you may have to manually +resolve things. This is where the ability to run a shell in this environment +comes in handy: ```sh -cd $KPATH/src/k8s.io/kubernetes -./hack/godep-restore.sh +hack/run-in-gopath.sh bash --norc --noprofile ``` -4) Next, you can either add a new dependency or update an existing one. +### Updating a dep -To add a new dependency is simple (if a bit slow): +Sometimes we already have a dep, but the version of it is wrong. Because of the +way that `godep` and `go get` interact (badly) it's generally easiest to hit it +with a big hammer: ```sh -cd $KPATH/src/k8s.io/kubernetes -DEP=example.com/path/to/dependency -godep get $DEP/... -# Now change code in Kubernetes to use the dependency. -./hack/godep-save.sh +hack/run-in-gopath.sh bash -c 'rm -rf $GOPATH/src/example.com/go/frob' +hack/run-in-gopath.sh go get -d example.com/go/frob +hack/run-in-gopath.sh bash -c 'git -C $GOPATH/src/example.com/go/frob checkout v2.0.0' ``` -To update an existing dependency is a bit more complicated. Godep has an -`update` command, but none of us can figure out how to actually make it work. -Instead, this procedure seems to work reliably: +This will remove the code, re-fetch it, and sync to your desired version. + +### Removing a dep + +This happens almost for free. If you edit Kubernetes code and remove the last +use of a given dependency, you only need to restore and save the deps, and the +`godep` tool will figure out that you don't need that dep any more: + +## Saving deps + +Now that you have made your changes - adding, updating, or removing the use of a +dep - you need to rebuild the dependency database and make changes to the +`vendor/` directory. ```sh -cd $KPATH/src/k8s.io/kubernetes -DEP=example.com/path/to/dependency -# NB: For the next step, $DEP is assumed be the repo root. If it is actually a -# subdir of the repo, use the repo root here. This is required to keep godep -# from getting angry because `godep restore` left the tree in a "detached head" -# state. -rm -rf $KPATH/src/$DEP # repo root -godep get $DEP/... -# Change code in Kubernetes, if necessary. -rm -rf Godeps -rm -rf vendor -./hack/godep-save.sh -# Regenerate removed BUILD, licenses. -touch vendor/BUILD -./hack/update-bazel.sh -./hack/update-godep-licenses.sh -# If you haven't followed this doc step-by-step and haven't created a dedicated GOPATH, -# make sure there is no client-go or other staging repo in $GOPATH before running the next command. -./hack/update-staging-godeps.sh -git checkout -- $(git status -s | grep "^ D" | awk '{print $2}' | grep ^Godeps) +hack/run-in-gopath.sh hack/godep-save.sh ``` -_If `go get -u path/to/dependency` fails with compilation errors, instead try -`go get -d -u path/to/dependency` to fetch the dependencies without compiling -them. This is unusual, but has been observed._ +This will run through all of the primary targets for the Kubernetes project, +calculate which deps are needed, and rebuild the database. It will also +regenerate other metadata files which the project needs, such as BUILD files and +the LICENSE database. -After all of this is done, `git status` should show you what files have been -modified and added/removed. Make sure to `git add` and `git rm` them. It is -commonly advised to make one `git commit` which includes just the dependency -update and Godeps files, and another `git commit` that includes changes to -Kubernetes code to use the new/updated dependency. These commits can go into a -single pull request. +## Saving deps in staging repos + +Kubernetes stores some code in a directory called `staging` which is handled +specially, and is not covered by the above. If you modified any code under +staging, or if you changed a dependency of code under staging (even +transitively), you'll also need to update deps there: -5) Before sending your PR, it's a good idea to sanity check that your -Godeps.json file and the contents of `vendor/ `are ok by running `hack/verify-godeps.sh` +```sh +./hack/update-staging-godeps.sh +``` -_If `hack/verify-godeps.sh` fails after a `godep update`, it is possible that a -transitive dependency was added or removed but not updated by godeps. It then -may be necessary to perform a `hack/godep-save.sh` to pick up the transitive -dependency changes._ +## Sanity checking -It is sometimes expedient to manually fix the /Godeps/Godeps.json file to -minimize the changes. However without great care this can lead to failures -with `hack/verify-godeps.sh`. This must pass for every PR. +After all of this is done, `git status` should show you what files have been +modified and added/removed. Make sure to sanity-check them with `git diff`, and +to `git add` and `git rm` them, as needed. It is commonly advised to make one +`git commit` which includes just the dependencies and Godeps files, and +another `git commit` that includes changes to Kubernetes code to use (or stop +using) the new/updated/removed dependency. These commits can go into a single +pull request. -6) If you updated the Godeps, please also update `Godeps/LICENSES` by running -`hack/update-godep-licenses.sh`. +Before sending your PR, it's a good idea to sanity check that your +Godeps.json file and the contents of `vendor/ `are ok: + +```sh +hack/run-in-gopath.sh hack/verify-godeps.sh +``` +All this script will do is a restore, followed by a save, and then look for +changes. If you followed the above instructions, it should be clean. If it is +not, you get to figure out why. +## Manual updates +It is sometimes expedient to manually fix the `Godeps.json` file to +minimize the changes. However, without great care this can lead to failures +with the verifier scripts. The kubernetes codebase does "interesting things" +with symlinks between `vendor/` and `staging/` to allow multiple Go import +paths to coexist in the same git repo. -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> +The verifiers, including `hack/verify-godeps.sh` *must* pass for every pull +request. diff --git a/contributors/devel/gubernator.md b/contributors/devel/gubernator.md index 96155f8d..2a25ddd7 100644 --- a/contributors/devel/gubernator.md +++ b/contributors/devel/gubernator.md @@ -2,7 +2,6 @@ *This document is oriented at developers who want to use Gubernator to debug while developing for Kubernetes.* -<!-- BEGIN MUNGE: GENERATED_TOC --> - [Gubernator](#gubernator) - [What is Gubernator?](#what-is-gubernator) @@ -12,7 +11,6 @@ - [Gubernator for Local Tests](#gubernator-for-local-tests) - [Future Work](#future-work) -<!-- END MUNGE: GENERATED_TOC --> ## What is Gubernator? @@ -115,7 +113,7 @@ k8s-gubernator.appspot.com/build/yourusername-g8r-logs/logs/e2e-node/timestamp Gubernator provides a framework for debugging failures and introduces useful features. There is still a lot of room for more features and growth to make the debugging process more efficient. -How to contribute (see https://github.com/kubernetes/test-infra/blob/master/gubernator/README.md) +How to contribute (see https://git.k8s.io/test-infra/gubernator/README.md) * Extend GUBERNATOR flag to all local tests @@ -136,7 +134,3 @@ How to contribute (see https://github.com/kubernetes/test-infra/blob/master/gube * Improve UI * Have separate folders of logs in rows instead of in one long column * Improve interface for adding additional features (maybe instead of textbox and checkbox, have chips) - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/help-wanted.md b/contributors/devel/help-wanted.md new file mode 100644 index 00000000..eb1038a4 --- /dev/null +++ b/contributors/devel/help-wanted.md @@ -0,0 +1,13 @@ +# Help Wanted + +Items marked with the `help wanted` label need to ensure that they are: + +- Sufficiently actionable: clear description of what should be done +- Tractable for new/casual contributors: there is documentation how that type of change should be made +- Goldilocks priority: Not too high that a core contributor should do it, but not too low that it isn't useful enough for a core contributor to spend time to review it, answer questions, help get it into a release, etc. +- Up to date: Often these issues become obsolete and have already been done, are no longer desirable, no longer make sense, change in priority, change in difficulty, etc. + +Related commands: + +- `/help` : adds the `help wanted` label to an issue +- `/remove-help` : removes the `help wanted` label from an issue
\ No newline at end of file diff --git a/contributors/devel/how-to-doc.md b/contributors/devel/how-to-doc.md index 891969d7..14be18b7 100644 --- a/contributors/devel/how-to-doc.md +++ b/contributors/devel/how-to-doc.md @@ -1,205 +1,5 @@ # Document Conventions -Updated: 11/3/2015 +Updated: 11/3/2017 -*This document is oriented at users and developers who want to write documents -for Kubernetes.* - -**Table of Contents** -<!-- BEGIN MUNGE: GENERATED_TOC --> - -- [Document Conventions](#document-conventions) - - [General Concepts](#general-concepts) - - [How to Get a Table of Contents](#how-to-get-a-table-of-contents) - - [How to Write Links](#how-to-write-links) - - [How to Include an Example](#how-to-include-an-example) - - [Misc.](#misc) - - [Code formatting](#code-formatting) - - [Syntax Highlighting](#syntax-highlighting) - - [Headings](#headings) - - [What Are Mungers?](#what-are-mungers) - - [Auto-added Mungers](#auto-added-mungers) - - [Generate Analytics](#generate-analytics) -- [Generated documentation](#generated-documentation) - -<!-- END MUNGE: GENERATED_TOC --> - -## General Concepts - -Each document needs to be munged to ensure its format is correct, links are -valid, etc. To munge a document, simply run `hack/update-munge-docs.sh`. We -verify that all documents have been munged using `hack/verify-munge-docs.sh`. -The scripts for munging documents are called mungers, see the -[mungers section](#what-are-mungers) below if you're curious about how mungers -are implemented or if you want to write one. - -## How to Get a Table of Contents - -Instead of writing table of contents by hand, insert the following code in your -md file: - -``` -<!-- BEGIN MUNGE: GENERATED_TOC --> -<!-- END MUNGE: GENERATED_TOC --> -``` - -After running `hack/update-munge-docs.sh`, you'll see a table of contents -generated for you, layered based on the headings. - -## How to Write Links - -It's important to follow the rules when writing links. It helps us correctly -versionize documents for each release. - -Use inline links instead of urls at all times. When you add internal links to -`docs/` or `examples/`, use relative links; otherwise, use -`http://releases.k8s.io/HEAD/<path/to/link>`. For example, avoid using: - -``` -[GCE](https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/gce.md) # note that it's under docs/ -[Kubernetes package](../../pkg/) # note that it's under pkg/ -http://kubernetes.io/ # external link -``` - -Instead, use: - -``` -[GCE](../getting-started-guides/gce.md) # note that it's under docs/ -[Kubernetes package](http://releases.k8s.io/HEAD/pkg/) # note that it's under pkg/ -[Kubernetes](http://kubernetes.io/) # external link -``` - -The above example generates the following links: -[GCE](../getting-started-guides/gce.md), -[Kubernetes package](http://releases.k8s.io/HEAD/pkg/), and -[Kubernetes](http://kubernetes.io/). - -## How to Include an Example - -While writing examples, you may want to show the content of certain example -files (e.g. [pod.yaml](../../test/fixtures/doc-yaml/user-guide/pod.yaml)). In this case, insert the -following code in the md file: - -``` -<!-- BEGIN MUNGE: EXAMPLE path/to/file --> -<!-- END MUNGE: EXAMPLE path/to/file --> -``` - -Note that you should replace `path/to/file` with the relative path to the -example file. Then `hack/update-munge-docs.sh` will generate a code block with -the content of the specified file, and a link to download it. This way, you save -the time to do the copy-and-paste; what's better, the content won't become -out-of-date every time you update the example file. - -For example, the following: - -``` -<!-- BEGIN MUNGE: EXAMPLE ../../test/fixtures/doc-yaml/user-guide/pod.yaml --> -<!-- END MUNGE: EXAMPLE ../../test/fixtures/doc-yaml/user-guide/pod.yaml --> -``` - -generates the following after `hack/update-munge-docs.sh`: - -<!-- BEGIN MUNGE: EXAMPLE ../../test/fixtures/doc-yaml/user-guide/pod.yaml --> - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: nginx - labels: - app: nginx -spec: - containers: - - name: nginx - image: nginx - ports: - - containerPort: 80 -``` - -[Download example](../../test/fixtures/doc-yaml/user-guide/pod.yaml?raw=true) -<!-- END MUNGE: EXAMPLE ../../test/fixtures/doc-yaml/user-guide/pod.yaml --> - -## Misc. - -### Code formatting - -Wrap a span of code with single backticks (`` ` ``). To format multiple lines of -code as its own code block, use triple backticks (```` ``` ````). - -### Syntax Highlighting - -Adding syntax highlighting to code blocks improves readability. To do so, in -your fenced block, add an optional language identifier. Some useful identifier -includes `yaml`, `console` (for console output), and `sh` (for shell quote -format). Note that in a console output, put `$ ` at the beginning of each -command and put nothing at the beginning of the output. Here's an example of -console code block: - -``` -```console - -$ kubectl create -f test/fixtures/doc-yaml/user-guide/pod.yaml -pod "foo" created - -``` -``` - -which renders as: - -```console -$ kubectl create -f test/fixtures/doc-yaml/user-guide/pod.yaml -pod "foo" created -``` - -### Headings - -Add a single `#` before the document title to create a title heading, and add -`##` to the next level of section title, and so on. Note that the number of `#` -will determine the size of the heading. - -## What Are Mungers? - -Mungers are like gofmt for md docs which we use to format documents. To use it, -simply place - -``` -<!-- BEGIN MUNGE: xxxx --> -<!-- END MUNGE: xxxx --> -``` - -in your md files. Note that xxxx is the placeholder for a specific munger. -Appropriate content will be generated and inserted between two brackets after -you run `hack/update-munge-docs.sh`. See -[munger document](http://releases.k8s.io/HEAD/cmd/mungedocs/) for more details. - -## Auto-added Mungers - -After running `hack/update-munge-docs.sh`, you may see some code / mungers in -your md file that are auto-added. You don't have to add them manually. It's -recommended to just read this section as a reference instead of messing up with -the following mungers. - -### Generate Analytics - -ANALYTICS munger inserts a Google Anaylytics link for this page. - -``` -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -<!-- END MUNGE: GENERATED_ANALYTICS --> -``` - -# Generated documentation - -Some documents can be generated automatically. Run `hack/generate-docs.sh` to -populate your repository with these generated documents, and a list of the files -it generates is placed in `.generated_docs`. To reduce merge conflicts, we do -not want to check these documents in; however, to make the link checker in the -munger happy, we check in a placeholder. `hack/update-generated-docs.sh` puts a -placeholder in the location where each generated document would go, and -`hack/verify-generated-docs.sh` verifies that the placeholder is in place. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> +*Users and developers who want to write documents for Kubernetes can get started [here](https://github.com/kubernetes/website).* diff --git a/contributors/devel/instrumentation.md b/contributors/devel/instrumentation.md index d54bb45a..dfa21c69 100644 --- a/contributors/devel/instrumentation.md +++ b/contributors/devel/instrumentation.md @@ -213,8 +213,3 @@ metric could look as follows: kube_pod_restarts and on(namespace, pod) kube_pod_info{uuid=”ABC”} ``` - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/issues.md b/contributors/devel/issues.md index ecebc7ae..575eddff 100644 --- a/contributors/devel/issues.md +++ b/contributors/devel/issues.md @@ -33,27 +33,73 @@ for other github repositories related to Kubernetes is TBD. Most people can leave comments and open issues. They don't have the ability to set labels, change milestones and close other peoples issues. For that we use a bot to manage labelling and triaging. The bot has a set of -[commands and permissions](https://github.com/kubernetes/test-infra/blob/master/commands.md) +[commands and permissions](https://git.k8s.io/test-infra/commands.md) and this document will cover the basic ones. ## Determine if it’s a support request Sometimes users ask for support requests in issues; these are usually requests from people who need help configuring some aspect of Kubernetes. These should be -directed to our [support structures](https://github.com/kubernetes/community/blob/master/contributors/devel/on-call-user-support.md) and then closed. Also, if the issue is clearly abandoned or in -the wrong place, it should be closed. Keep in mind that only issue reporter, -assignees and component organization members can close issue. If you do not -have such privilege, just comment your findings. Otherwise, first `/assign` -issue to yourself and then `/close`. +directed to our support structures (see below) and then closed. Also, if the issue +is clearly abandoned or in the wrong place, it should be closed. Keep in mind that +only issue reporter, assignees and component organization members can close issue. +If you do not have such privilege, just comment your findings. Otherwise, first +`/assign` issue to yourself and then `/close`. + +### Support Structures + +Support requests should be directed to the following: + +* [User documentation](https://kubernetes.io/docs/) and +[troubleshooting guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/) + +* [Stack Overflow](http://stackoverflow.com/questions/tagged/kubernetes) and +[ServerFault](http://serverfault.com/questions/tagged/kubernetes) + +* [Slack](https://kubernetes.slack.com) ([registration](http://slack.k8s.io)) + * Check out the [Slack Archive](http://kubernetes.slackarchive.io/) first. + +* [Email/Groups](https://groups.google.com/forum/#!forum/kubernetes-users) + +### User support response example + +If you see support questions on kubernetes-dev@googlegroups.com or issues asking for +support try to redirect them to Stack Overflow. Example response: + +```code +Please re-post your question to [Stack Overflow] +(http://stackoverflow.com/questions/tagged/kubernetes). + +We are trying to consolidate the channels to which questions for help/support +are posted so that we can improve our efficiency in responding to your requests, +and to make it easier for you to find answers to frequently asked questions and +how to address common use cases. + +We regularly see messages posted in multiple forums, with the full response +thread only in one place or, worse, spread across multiple forums. Also, the +large volume of support issues on github is making it difficult for us to use +issues to identify real bugs. + +Members of the Kubernetes community use Stack Overflow to field support +requests. Before posting a new question, please search Stack Overflow for answers +to similar questions, and also familiarize yourself with: + + * [user documentation](http://kubernetes.io/docs/) + * [troubleshooting guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/) + +Again, thanks for using Kubernetes. + +The Kubernetes Team +``` ## Find the right SIG(s) -Components are divided among [Special Interest Groups (SIGs)](https://github.com/kubernetes/community/blob/master/sig-list.md). Find a proper SIG for the ownership of the issue using the bot: +Components are divided among [Special Interest Groups (SIGs)](/sig-list.md). Find a proper SIG for the ownership of the issue using the bot: * Typing `/sig network` in a comment should add the sig/network label, for example. * Multiword SIGs use dashes, for example `/sig cluster-lifecycle`. -Keep in mind that these commands must be on its own and at the front of the +Keep in mind that these commands must be on its own line and at the front of the comment. ## Validate if the issue is bug @@ -158,7 +204,3 @@ issues are work we would merge into the release if it gets done, but we wouldn't block the release on it. A few days before release, we will probably move all `priority/important-soon` and `priority/important-longterm` bugs out of that milestone in bulk. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/kubectl-conventions.md b/contributors/devel/kubectl-conventions.md index 90a77633..5b009657 100644 --- a/contributors/devel/kubectl-conventions.md +++ b/contributors/devel/kubectl-conventions.md @@ -3,7 +3,6 @@ Updated: 3/23/2017 **Table of Contents** -<!-- BEGIN MUNGE: GENERATED_TOC --> - [Kubectl Conventions](#kubectl-conventions) - [Principles](#principles) @@ -15,9 +14,9 @@ Updated: 3/23/2017 - [Documentation conventions](#documentation-conventions) - [kubectl Factory conventions](#kubectl-Factory-conventions) - [Command implementation conventions](#command-implementation-conventions) + - [Exit code conventions](#exit-code-conventions) - [Generators](#generators) -<!-- END MUNGE: GENERATED_TOC --> ## Principles @@ -173,7 +172,7 @@ generation, etc., and display the output * `--output-version=...`: Convert the output to a different API group/version * `--short`: Output a compact summary of normal output; the format is subject -to change and is optimizied for reading not parsing. +to change and is optimized for reading not parsing. * `--validate`: Validate the resource schema @@ -256,10 +255,8 @@ input, output, commonly used flags, etc. * Example should contain examples * A comment should precede each example command. Comment should start with - an uppercase letter and terminate with a period - * Start commands with `$` - - + an uppercase letter + * Command examples should not include a `$` prefix * Use "FILENAME" for filenames @@ -375,7 +372,22 @@ and as noted in [command conventions](#command-conventions), ideally that logic should exist server-side so any client could take advantage of it. Notice that this is not a mandatory structure and not every command is implemented this way, but this is a nice convention so try to be compliant with it. As an example, -have a look at how [kubectl logs](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/logs.go) is implemented. +have a look at how [kubectl logs](https://git.k8s.io/kubernetes/pkg/kubectl/cmd/logs.go) is implemented. + +## Exit code conventions + +Generally, for all the command exit code, result of `zero` means success and `non-zero` means errors. + +For idempotent ("make-it-so") commands, we should return `zero` when success even if no changes were provided, user can request treating "make-it-so" as "already-so" via flag `--error-unchanged` to make it return `non-zero` exit code. + +For non-idempotent ("already-so") commands, we should return `non-zero` by default, user can request treating "already-so" as "make-it-so" via flag `--ignore-unchanged` to make it return `zero` exit code. + + +| Exit Code Number | Meaning | Enable | +| :--- | :--- | :--- | +| 0 | Command exited success | By default, By flag `--ignore-unchanged` | +| 1 | Command exited for general errors | By default | +| 3 | Command was successful, but the user requested a distinct exit code when no change was made | By flag `--error-unchanged`| ## Generators @@ -444,7 +456,3 @@ method which configures the generated namespace that callers of the generator * `--dry-run` should output the resource that would be created, without creating it. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/kubemark-guide.md b/contributors/devel/kubemark-guide.md index cb699a88..2c404424 100755..100644 --- a/contributors/devel/kubemark-guide.md +++ b/contributors/devel/kubemark-guide.md @@ -254,7 +254,3 @@ We currently plan to extend kubemark along the following directions: to logs. - Create a Dashboard that lets easy viewing and comparison of these metrics across tests. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/local-cluster/OWNERS b/contributors/devel/local-cluster/OWNERS deleted file mode 100644 index 1f917ede..00000000 --- a/contributors/devel/local-cluster/OWNERS +++ /dev/null @@ -1,12 +0,0 @@ -reviewers: - - xilabao - - michelleN - - jianglingxia - - ruebenramirez - - davidxia -approvers: - - xilabao - - michelleN - - jianglingxia - - ruebenramirez - - davidxia diff --git a/contributors/devel/local-cluster/docker.md b/contributors/devel/local-cluster/docker.md deleted file mode 100644 index b1d9f95a..00000000 --- a/contributors/devel/local-cluster/docker.md +++ /dev/null @@ -1,268 +0,0 @@ -**Stop. This guide has been superseded by [Minikube](https://github.com/kubernetes/minikube) which is the recommended method of running Kubernetes on your local machine.** - - -The following instructions show you how to set up a simple, single node Kubernetes cluster using Docker. - -Here's a diagram of what the final result will look like: - - - -## Prerequisites - -**Note: These steps have not been tested with the [Docker For Mac or Docker For Windows beta programs](https://blog.docker.com/2016/03/docker-for-mac-windows-beta/).** - -1. You need to have Docker version >= "1.10" installed on the machine. -2. Enable mount propagation. Hyperkube is running in a container which has to mount volumes for other containers, for example in case of persistent storage. The required steps depend on the init system. - - - In case of **systemd**, change MountFlags in the Docker unit file to shared. - - ```shell - DOCKER_CONF=$(systemctl cat docker | head -1 | awk '{print $2}') - sed -i.bak 's/^\(MountFlags=\).*/\1shared/' $DOCKER_CONF - systemctl daemon-reload - systemctl restart docker - ``` - - **Otherwise**, manually set the mount point used by Hyperkube to be shared: - - ```shell - mkdir -p /var/lib/kubelet - mount --bind /var/lib/kubelet /var/lib/kubelet - mount --make-shared /var/lib/kubelet - ``` - - -### Run it - -1. Decide which Kubernetes version to use. Set the `${K8S_VERSION}` variable to a version of Kubernetes >= "v1.2.0". - - - If you'd like to use the current **stable** version of Kubernetes, run the following: - - ```sh - export K8S_VERSION=$(curl -sS https://storage.googleapis.com/kubernetes-release/release/stable.txt) - ``` - - and for the **latest** available version (including unstable releases): - - ```sh - export K8S_VERSION=$(curl -sS https://storage.googleapis.com/kubernetes-release/release/latest.txt) - ``` - -2. Start Hyperkube - - ```shell - export ARCH=amd64 - docker run -d \ - --volume=/sys:/sys:rw \ - --volume=/var/lib/docker/:/var/lib/docker:rw \ - --volume=/var/lib/kubelet/:/var/lib/kubelet:rw,shared \ - --volume=/var/run:/var/run:rw \ - --net=host \ - --pid=host \ - --privileged \ - --name=kubelet \ - gcr.io/google_containers/hyperkube-${ARCH}:${K8S_VERSION} \ - /hyperkube kubelet \ - --hostname-override=127.0.0.1 \ - --api-servers=http://localhost:8080 \ - --kubeconfig=/etc/kubernetes/manifests \ - --cluster-dns=10.0.0.10 \ - --cluster-domain=cluster.local \ - --allow-privileged --v=2 - ``` - - > Note that `--cluster-dns` and `--cluster-domain` is used to deploy dns, feel free to discard them if dns is not needed. - - > If you would like to mount an external device as a volume, add `--volume=/dev:/dev` to the command above. It may however, cause some problems described in [#18230](https://github.com/kubernetes/kubernetes/issues/18230) - - > Architectures other than `amd64` are experimental and sometimes unstable, but feel free to try them out! Valid values: `arm`, `arm64` and `ppc64le`. ARM is available with Kubernetes version `v1.3.0-alpha.2` and higher. ARM 64-bit and PowerPC 64 little-endian are available with `v1.3.0-alpha.3` and higher. Track progress on multi-arch support [here](https://github.com/kubernetes/kubernetes/issues/17981) - - > If you are behind a proxy, you need to pass the proxy setup to curl in the containers to pull the certificates. Create a .curlrc under /root folder (because the containers are running as root) with the following line: - - ``` - proxy = <your_proxy_server>:<port> - ``` - - This actually runs the kubelet, which in turn runs a [pod](http://kubernetes.io/docs/user-guide/pods/) that contains the other master components. - - ** **SECURITY WARNING** ** services exposed via Kubernetes using Hyperkube are available on the host node's public network interface / IP address. Because of this, this guide is not suitable for any host node/server that is directly internet accessible. Refer to [#21735](https://github.com/kubernetes/kubernetes/issues/21735) for additional info. - -### Download `kubectl` - -At this point you should have a running Kubernetes cluster. You can test it out -by downloading the kubectl binary for `${K8S_VERSION}` (in this example: `{{page.version}}.0`). - - -Downloads: - - - `linux/amd64`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/amd64/kubectl - - `linux/386`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/386/kubectl - - `linux/arm`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/arm/kubectl - - `linux/arm64`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/arm64/kubectl - - `linux/ppc64le`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/ppc64le/kubectl - - `OS X/amd64`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/darwin/amd64/kubectl - - `OS X/386`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/darwin/386/kubectl - - `windows/amd64`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/windows/amd64/kubectl.exe - - `windows/386`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/windows/386/kubectl.exe - -The generic download path is: - -``` -http://storage.googleapis.com/kubernetes-release/release/${K8S_VERSION}/bin/${GOOS}/${GOARCH}/${K8S_BINARY} -``` - -An example install with `linux/amd64`: - -``` -curl -sSL "https://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/amd64/kubectl" > /usr/bin/kubectl -chmod +x /usr/bin/kubectl -``` - -On OS X, to make the API server accessible locally, setup a ssh tunnel. - -```shell -docker-machine ssh `docker-machine active` -N -L 8080:localhost:8080 -``` - -Setting up a ssh tunnel is applicable to remote docker hosts as well. - -(Optional) Create kubernetes cluster configuration: - -```shell -kubectl config set-cluster test-doc --server=http://localhost:8080 -kubectl config set-context test-doc --cluster=test-doc -kubectl config use-context test-doc -``` - -### Test it out - -List the nodes in your cluster by running: - -```shell -kubectl get nodes -``` - -This should print: - -```shell -NAME STATUS AGE -127.0.0.1 Ready 1h -``` - -### Run an application - -```shell -kubectl run nginx --image=nginx --port=80 -``` - -Now run `docker ps` you should see nginx running. You may need to wait a few minutes for the image to get pulled. - -### Expose it as a service - -```shell -kubectl expose deployment nginx --port=80 -``` - -Run the following command to obtain the cluster local IP of this service we just created: - -```shell -ip=$(kubectl get svc nginx --template={{.spec.clusterIP}}) -echo $ip -``` - -Hit the webserver with this IP: - -```shell -curl $ip -``` - -On OS X, since docker is running inside a VM, run the following command instead: - -```shell -docker-machine ssh `docker-machine active` curl $ip -``` - -## Deploy a DNS - -Read [documentation for manually deploying a DNS](https://git.k8s.io/examples/staging/cluster-dns) for instructions. - -### Turning down your cluster - -1. Delete the nginx service and deployment: - -If you plan on re-creating your nginx deployment and service you will need to clean it up. - -```shell -kubectl delete service,deployments nginx -``` - -2. Delete all the containers including the kubelet: - -```shell -docker rm -f kubelet -docker rm -f `docker ps | grep k8s | awk '{print $1}'` -``` - -3. Cleanup the filesystem: - -On OS X, first ssh into the docker VM: - -```shell -docker-machine ssh `docker-machine active` -``` - -```shell -grep /var/lib/kubelet /proc/mounts | awk '{print $2}' | sudo xargs -n1 umount -sudo rm -rf /var/lib/kubelet -``` - -### Troubleshooting - -#### Node is in `NotReady` state - -If you see your node as `NotReady` it's possible that your OS does not have memcg enabled. - -1. Your kernel should support memory accounting. Ensure that the -following configs are turned on in your linux kernel: - -```shell -CONFIG_RESOURCE_COUNTERS=y -CONFIG_MEMCG=y -``` - -2. Enable the memory accounting in the kernel, at boot, as command line -parameters as follows: - -```shell -GRUB_CMDLINE_LINUX="cgroup_enable=memory=1" -``` - -NOTE: The above is specifically for GRUB2. -You can check the command line parameters passed to your kernel by looking at the -output of /proc/cmdline: - -```shell -$ cat /proc/cmdline -BOOT_IMAGE=/boot/vmlinuz-3.18.4-aufs root=/dev/sda5 ro cgroup_enable=memory=1 -``` - -## Support Level - - -IaaS Provider | Config. Mgmt | OS | Networking | Conforms | Support Level --------------------- | ------------ | ------ | ---------- | ---------| ---------------------------- -Docker Single Node | custom | N/A | local | | Project ([@brendandburns](https://github.com/brendandburns)) - - - -## Further reading - -Please see the [Kubernetes docs](http://kubernetes.io/docs) for more details on administering -and using a Kubernetes cluster. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/local-cluster/k8s-singlenode-docker.png b/contributors/devel/local-cluster/k8s-singlenode-docker.png Binary files differdeleted file mode 100644 index 5ebf8126..00000000 --- a/contributors/devel/local-cluster/k8s-singlenode-docker.png +++ /dev/null diff --git a/contributors/devel/local-cluster/local.md b/contributors/devel/local-cluster/local.md deleted file mode 100644 index fd538c81..00000000 --- a/contributors/devel/local-cluster/local.md +++ /dev/null @@ -1,129 +0,0 @@ -**Stop. This guide has been superseded by [Minikube](https://github.com/kubernetes/minikube) which is the recommended method of running Kubernetes on your local machine.** - -### Requirements - -#### Linux - -Not running Linux? Consider running Linux in a local virtual machine with [vagrant](https://www.vagrantup.com/), or on a cloud provider like Google Compute Engine - -#### Docker - -At least [Docker](https://docs.docker.com/installation/#installation) -1.8.3+. Ensure the Docker daemon is running and can be contacted (try `docker -ps`). Some of the Kubernetes components need to run as root, which normally -works fine with docker. - -#### etcd - -You need an [etcd](https://github.com/coreos/etcd/releases) in your path, please make sure it is installed and in your ``$PATH``. - -#### go - -You need [go](https://golang.org/doc/install) at least 1.8.3 in your path, please make sure it is installed and in your ``$PATH``. - -### Starting the cluster - -First, you need to [download Kubernetes](http://kubernetes.io/docs/getting-started-guides/binary_release/). Then open a separate tab of your terminal -and run the following (since one needs sudo access to start/stop Kubernetes daemons, it is easier to run the entire script as root): - -```shell -cd kubernetes -hack/local-up-cluster.sh -``` -If you want to enable RBAC on local cluster, you can run the following: -```shell -ENABLE_RBAC=true hack/local-up-cluster.sh -``` - -This will build and start a lightweight local cluster, consisting of a master -and a single node. Type Control-C to shut it down. - -You can use the cluster/kubectl.sh script to interact with the local cluster. hack/local-up-cluster.sh will -print the commands to run to point kubectl at the local cluster. - - -### Running a container - -Your cluster is running, and you want to start running containers! - -You can now use any of the cluster/kubectl.sh commands to interact with your local setup. - -```shell -export KUBERNETES_PROVIDER=local -cluster/kubectl.sh get pods -cluster/kubectl.sh get services -cluster/kubectl.sh get deployments -cluster/kubectl.sh run my-nginx --image=nginx --replicas=2 --port=80 - -## begin wait for provision to complete, you can monitor the docker pull by opening a new terminal - sudo docker images - ## you should see it pulling the nginx image, once the above command returns it - sudo docker ps - ## you should see your container running! - exit -## end wait - -## create a service for nginx, which serves on port 80 -cluster/kubectl.sh expose deployment my-nginx --port=80 --name=my-nginx - -## introspect Kubernetes! -cluster/kubectl.sh get pods -cluster/kubectl.sh get services -cluster/kubectl.sh get deployments - -## Test the nginx service with the IP/port from "get services" command -curl http://10.X.X.X:80/ -``` - -### Running a user defined pod - -Note the difference between a [container](http://kubernetes.io/docs/user-guide/containers/) -and a [pod](http://kubernetes.io/docs/user-guide/pods/). Since you only asked for the former, Kubernetes will create a wrapper pod for you. -However you cannot view the nginx start page on localhost. To verify that nginx is running you need to run `curl` within the docker container (try `docker exec`). - -You can control the specifications of a pod via a user defined manifest, and reach nginx through your browser on the port specified therein: - -```shell -cluster/kubectl.sh create -f test/fixtures/doc-yaml/user-guide/pod.yaml -``` - -Congratulations! - -### FAQs - -#### I cannot reach service IPs on the network. - -Some firewall software that uses iptables may not interact well with -kubernetes. If you have trouble around networking, try disabling any -firewall or other iptables-using systems, first. Also, you can check -if SELinux is blocking anything by running a command such as `journalctl --since yesterday | grep avc`. - -By default the IP range for service cluster IPs is 10.0.*.* - depending on your -docker installation, this may conflict with IPs for containers. If you find -containers running with IPs in this range, edit hack/local-cluster-up.sh and -change the service-cluster-ip-range flag to something else. - -#### I changed Kubernetes code, how do I run it? - -```shell -cd kubernetes -hack/build-go.sh -hack/local-up-cluster.sh -``` - -#### kubectl claims to start a container but `get pods` and `docker ps` don't show it. - -One or more of the Kubernetes daemons might've crashed. Tail the [logs](http://kubernetes.io/docs/admin/cluster-troubleshooting/#looking-at-logs) of each in /tmp. - -```shell -$ ls /tmp/kube*.log -$ tail -f /tmp/kube-apiserver.log -``` - -#### The pods fail to connect to the services by host names - -The local-up-cluster.sh script doesn't start a DNS service. Similar situation can be found [here](https://issue.k8s.io/6667). You can start a manually. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/local-cluster/vagrant.md b/contributors/devel/local-cluster/vagrant.md deleted file mode 100644 index 0f0fe91c..00000000 --- a/contributors/devel/local-cluster/vagrant.md +++ /dev/null @@ -1,397 +0,0 @@ -Running Kubernetes with Vagrant (and VirtualBox) is an easy way to run/test/develop on your local machine (Linux, Mac OS X). - -### Prerequisites - -1. Install latest version >= 1.7.4 of [Vagrant](http://www.vagrantup.com/downloads.html) -2. Install one of: - 1. The latest version of [Virtual Box](https://www.virtualbox.org/wiki/Downloads) - 2. [VMWare Fusion](https://www.vmware.com/products/fusion/) version 5 or greater as well as the appropriate [Vagrant VMWare Fusion provider](https://www.vagrantup.com/vmware) - 3. [VMWare Workstation](https://www.vmware.com/products/workstation/) version 9 or greater as well as the [Vagrant VMWare Workstation provider](https://www.vagrantup.com/vmware) - 4. [Parallels Desktop](https://www.parallels.com/products/desktop/) version 9 or greater as well as the [Vagrant Parallels provider](https://parallels.github.io/vagrant-parallels/) - 5. libvirt with KVM and enable support of hardware virtualisation. [Vagrant-libvirt](https://github.com/pradels/vagrant-libvirt). For fedora provided official rpm, and possible to use `yum install vagrant-libvirt` - -### Setup - -Setting up a cluster is as simple as running: - -```sh -export KUBERNETES_PROVIDER=vagrant -curl -sS https://get.k8s.io | bash -``` - -Alternatively, you can download [Kubernetes release](https://github.com/kubernetes/kubernetes/releases) and extract the archive. To start your local cluster, open a shell and run: - -```sh -cd kubernetes - -export KUBERNETES_PROVIDER=vagrant -./cluster/kube-up.sh -``` - -The `KUBERNETES_PROVIDER` environment variable tells all of the various cluster management scripts which variant to use. If you forget to set this, the assumption is you are running on Google Compute Engine. - -By default, the Vagrant setup will create a single master VM (called kubernetes-master) and one node (called kubernetes-node-1). Each VM will take 1 GB, so make sure you have at least 2GB to 4GB of free memory (plus appropriate free disk space). - -If you'd like more than one node, set the `NUM_NODES` environment variable to the number you want: - -```sh -export NUM_NODES=3 -``` - -Vagrant will provision each machine in the cluster with all the necessary components to run Kubernetes. The initial setup can take a few minutes to complete on each machine. - -If you installed more than one Vagrant provider, Kubernetes will usually pick the appropriate one. However, you can override which one Kubernetes will use by setting the [`VAGRANT_DEFAULT_PROVIDER`](https://docs.vagrantup.com/v2/providers/default.html) environment variable: - -```sh -export VAGRANT_DEFAULT_PROVIDER=parallels -export KUBERNETES_PROVIDER=vagrant -./cluster/kube-up.sh -``` - -By default, each VM in the cluster is running Fedora. - -To access the master or any node: - -```sh -vagrant ssh master -vagrant ssh node-1 -``` - -If you are running more than one node, you can access the others by: - -```sh -vagrant ssh node-2 -vagrant ssh node-3 -``` - -Each node in the cluster installs the docker daemon and the kubelet. - -The master node instantiates the Kubernetes master components as pods on the machine. - -To view the service status and/or logs on the kubernetes-master: - -```console -[vagrant@kubernetes-master ~] $ vagrant ssh master -[vagrant@kubernetes-master ~] $ sudo su - -[root@kubernetes-master ~] $ systemctl status kubelet -[root@kubernetes-master ~] $ journalctl -ru kubelet - -[root@kubernetes-master ~] $ systemctl status docker -[root@kubernetes-master ~] $ journalctl -ru docker - -[root@kubernetes-master ~] $ tail -f /var/log/kube-apiserver.log -[root@kubernetes-master ~] $ tail -f /var/log/kube-controller-manager.log -[root@kubernetes-master ~] $ tail -f /var/log/kube-scheduler.log -``` - -To view the services on any of the nodes: - -```console -[vagrant@kubernetes-master ~] $ vagrant ssh node-1 -[vagrant@kubernetes-master ~] $ sudo su - -[root@kubernetes-master ~] $ systemctl status kubelet -[root@kubernetes-master ~] $ journalctl -ru kubelet - -[root@kubernetes-master ~] $ systemctl status docker -[root@kubernetes-master ~] $ journalctl -ru docker -``` - -### Interacting with your Kubernetes cluster with Vagrant. - -With your Kubernetes cluster up, you can manage the nodes in your cluster with the regular Vagrant commands. - -To push updates to new Kubernetes code after making source changes: - -```sh -./cluster/kube-push.sh -``` - -To stop and then restart the cluster: - -```sh -vagrant halt -./cluster/kube-up.sh -``` - -To destroy the cluster: - -```sh -vagrant destroy -``` - -Once your Vagrant machines are up and provisioned, the first thing to do is to check that you can use the `kubectl.sh` script. - -You may need to build the binaries first, you can do this with `make` - -```console -$ ./cluster/kubectl.sh get nodes - -NAME LABELS -10.245.1.4 <none> -10.245.1.5 <none> -10.245.1.3 <none> -``` - -### Authenticating with your master - -When using the vagrant provider in Kubernetes, the `cluster/kubectl.sh` script will cache your credentials in a `~/.kubernetes_vagrant_auth` file so you will not be prompted for them in the future. - -```sh -cat ~/.kubernetes_vagrant_auth -``` - -```json -{ "User": "vagrant", - "Password": "vagrant", - "CAFile": "/home/k8s_user/.kubernetes.vagrant.ca.crt", - "CertFile": "/home/k8s_user/.kubecfg.vagrant.crt", - "KeyFile": "/home/k8s_user/.kubecfg.vagrant.key" -} -``` - -You should now be set to use the `cluster/kubectl.sh` script. For example try to list the nodes that you have started with: - -```sh -./cluster/kubectl.sh get nodes -``` - -### Running containers - -Your cluster is running, you can list the nodes in your cluster: - -```sh -$ ./cluster/kubectl.sh get nodes - -NAME LABELS -10.245.2.4 <none> -10.245.2.3 <none> -10.245.2.2 <none> -``` - -Now start running some containers! - -You can now use any of the `cluster/kube-*.sh` commands to interact with your VM machines. -Before starting a container there will be no pods, services and replication controllers. - -```sh -$ ./cluster/kubectl.sh get pods -NAME READY STATUS RESTARTS AGE - -$ ./cluster/kubectl.sh get services -NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE - -$ ./cluster/kubectl.sh get replicationcontrollers -CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS -``` - -Start a container running nginx with a replication controller and three replicas - -```sh -$ ./cluster/kubectl.sh run my-nginx --image=nginx --replicas=3 --port=80 -``` - -When listing the pods, you will see that three containers have been started and are in Waiting state: - -```sh -$ ./cluster/kubectl.sh get pods -NAME READY STATUS RESTARTS AGE -my-nginx-5kq0g 0/1 Pending 0 10s -my-nginx-gr3hh 0/1 Pending 0 10s -my-nginx-xql4j 0/1 Pending 0 10s -``` - -You need to wait for the provisioning to complete, you can monitor the nodes by doing: - -```sh -$ vagrant ssh node-1 -c 'sudo docker images' -kubernetes-node-1: - REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE - <none> <none> 96864a7d2df3 26 hours ago 204.4 MB - google/cadvisor latest e0575e677c50 13 days ago 12.64 MB - kubernetes/pause latest 6c4579af347b 8 weeks ago 239.8 kB -``` - -Once the docker image for nginx has been downloaded, the container will start and you can list it: - -```sh -$ vagrant ssh node-1 -c 'sudo docker ps' -kubernetes-node-1: - CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES - dbe79bf6e25b nginx:latest "nginx" 21 seconds ago Up 19 seconds k8s--mynginx.8c5b8a3a--7813c8bd_-_3ffe_-_11e4_-_9036_-_0800279696e1.etcd--7813c8bd_-_3ffe_-_11e4_-_9036_-_0800279696e1--fcfa837f - fa0e29c94501 kubernetes/pause:latest "/pause" 8 minutes ago Up 8 minutes 0.0.0.0:8080->80/tcp k8s--net.a90e7ce4--7813c8bd_-_3ffe_-_11e4_-_9036_-_0800279696e1.etcd--7813c8bd_-_3ffe_-_11e4_-_9036_-_0800279696e1--baf5b21b - aa2ee3ed844a google/cadvisor:latest "/usr/bin/cadvisor" 38 minutes ago Up 38 minutes k8s--cadvisor.9e90d182--cadvisor_-_agent.file--4626b3a2 - 65a3a926f357 kubernetes/pause:latest "/pause" 39 minutes ago Up 39 minutes 0.0.0.0:4194->8080/tcp k8s--net.c5ba7f0e--cadvisor_-_agent.file--342fd561 -``` - -Going back to listing the pods, services and replicationcontrollers, you now have: - -```sh -$ ./cluster/kubectl.sh get pods -NAME READY STATUS RESTARTS AGE -my-nginx-5kq0g 1/1 Running 0 1m -my-nginx-gr3hh 1/1 Running 0 1m -my-nginx-xql4j 1/1 Running 0 1m - -$ ./cluster/kubectl.sh get services -NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE - -$ ./cluster/kubectl.sh get replicationcontrollers -CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGE -my-nginx my-nginx nginx run=my-nginx 3 1m -``` - -We did not start any services, hence there are none listed. But we see three replicas displayed properly. - -Learn about [running your first containers](http://kubernetes.io/docs/user-guide/simple-nginx/) application to learn how to create a service. - -You can already play with scaling the replicas with: - -```sh -$ ./cluster/kubectl.sh scale rc my-nginx --replicas=2 -$ ./cluster/kubectl.sh get pods -NAME READY STATUS RESTARTS AGE -my-nginx-5kq0g 1/1 Running 0 2m -my-nginx-gr3hh 1/1 Running 0 2m -``` - -Congratulations! - -## Troubleshooting - -#### I keep downloading the same (large) box all the time! - -By default the Vagrantfile will download the box from S3. You can change this (and cache the box locally) by providing a name and an alternate URL when calling `kube-up.sh` - -```sh -export KUBERNETES_BOX_NAME=choose_your_own_name_for_your_kuber_box -export KUBERNETES_BOX_URL=path_of_your_kuber_box -export KUBERNETES_PROVIDER=vagrant -./cluster/kube-up.sh -``` - -#### I am getting timeouts when trying to curl the master from my host! - -During provision of the cluster, you may see the following message: - -```sh -Validating node-1 -............. -Waiting for each node to be registered with cloud provider -error: couldn't read version from server: Get https://10.245.1.2/api: dial tcp 10.245.1.2:443: i/o timeout -``` - -Some users have reported VPNs may prevent traffic from being routed to the host machine into the virtual machine network. - -To debug, first verify that the master is binding to the proper IP address: - -```sh -$ vagrant ssh master -$ ifconfig | grep eth1 -C 2 -eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.245.1.2 netmask - 255.255.255.0 broadcast 10.245.1.255 -``` - -Then verify that your host machine has a network connection to a bridge that can serve that address: - -```sh -$ ifconfig | grep 10.245.1 -C 2 - -vboxnet5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 - inet 10.245.1.1 netmask 255.255.255.0 broadcast 10.245.1.255 - inet6 fe80::800:27ff:fe00:5 prefixlen 64 scopeid 0x20<link> - ether 0a:00:27:00:00:05 txqueuelen 1000 (Ethernet) -``` - -If you do not see a response on your host machine, you will most likely need to connect your host to the virtual network created by the virtualization provider. - -If you do see a network, but are still unable to ping the machine, check if your VPN is blocking the request. - -#### I just created the cluster, but I am getting authorization errors! - -You probably have an incorrect ~/.kubernetes_vagrant_auth file for the cluster you are attempting to contact. - -```sh -rm ~/.kubernetes_vagrant_auth -``` - -After using kubectl.sh make sure that the correct credentials are set: - -```sh -cat ~/.kubernetes_vagrant_auth -``` - -```json -{ - "User": "vagrant", - "Password": "vagrant" -} -``` - -#### I just created the cluster, but I do not see my container running! - -If this is your first time creating the cluster, the kubelet on each node schedules a number of docker pull requests to fetch prerequisite images. This can take some time and as a result may delay your initial pod getting provisioned. - -#### I have brought Vagrant up but the nodes cannot validate! - -Log on to one of the nodes (`vagrant ssh node-1`) and inspect the salt minion log (`sudo cat /var/log/salt/minion`). - -#### I want to change the number of nodes! - -You can control the number of nodes that are instantiated via the environment variable `NUM_NODES` on your host machine. If you plan to work with replicas, we strongly encourage you to work with enough nodes to satisfy your largest intended replica size. If you do not plan to work with replicas, you can save some system resources by running with a single node. You do this, by setting `NUM_NODES` to 1 like so: - -```sh -export NUM_NODES=1 -``` - -#### I want my VMs to have more memory! - -You can control the memory allotted to virtual machines with the `KUBERNETES_MEMORY` environment variable. -Just set it to the number of megabytes you would like the machines to have. For example: - -```sh -export KUBERNETES_MEMORY=2048 -``` - -If you need more granular control, you can set the amount of memory for the master and nodes independently. For example: - -```sh -export KUBERNETES_MASTER_MEMORY=1536 -export KUBERNETES_NODE_MEMORY=2048 -``` - -#### I want to set proxy settings for my Kubernetes cluster boot strapping! - -If you are behind a proxy, you need to install vagrant proxy plugin and set the proxy settings by - -```sh -vagrant plugin install vagrant-proxyconf -export VAGRANT_HTTP_PROXY=http://username:password@proxyaddr:proxyport -export VAGRANT_HTTPS_PROXY=https://username:password@proxyaddr:proxyport -``` - -Optionally you can specify addresses to not proxy, for example - -```sh -export VAGRANT_NO_PROXY=127.0.0.1 -``` - -If you are using sudo to make kubernetes build for example make quick-release, you need run `sudo -E make quick-release` to pass the environment variables. - -#### I ran vagrant suspend and nothing works! - -`vagrant suspend` seems to mess up the network. This is not supported at this time. - -#### I want vagrant to sync folders via nfs! - -You can ensure that vagrant uses nfs to sync folders with virtual machines by setting the KUBERNETES_VAGRANT_USE_NFS environment variable to 'true'. nfs is faster than virtualbox or vmware's 'shared folders' and does not require guest additions. See the [vagrant docs](http://docs.vagrantup.com/v2/synced-folders/nfs.html) for details on configuring nfs on the host. This setting will have no effect on the libvirt provider, which uses nfs by default. For example: - -```sh -export KUBERNETES_VAGRANT_USE_NFS=true -``` - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/logging.md b/contributors/devel/logging.md index 1241ee7f..12a719de 100644 --- a/contributors/devel/logging.md +++ b/contributors/devel/logging.md @@ -29,8 +29,3 @@ The following conventions for the glog levels to use. As per the comments, the practical default level is V(2). Developers and QE environments may wish to run at V(3) or V(4). If you wish to change the log level, you can pass in `-v=X` where X is the desired maximum level to log. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/mesos-style.md b/contributors/devel/mesos-style.md index 53ee139e..fbded425 100644 --- a/contributors/devel/mesos-style.md +++ b/contributors/devel/mesos-style.md @@ -61,7 +61,7 @@ machine manages the collection during their lifetimes Out-of-the-box Kubernetes has *workload-specific* abstractions (ReplicaSet, Job, DaemonSet, etc.) and corresponding controllers, and in the future may have -[workload-specific schedulers](../design-proposals/multiple-schedulers.md), +[workload-specific schedulers](../design-proposals/scheduling/multiple-schedulers.md), e.g. different schedulers for long-running services vs. short-running batch. But these abstractions, controllers, and schedulers are not *application-specific*. @@ -210,9 +210,3 @@ Mesos is described [here](https://www.usenix.org/legacy/event/nsdi11/tech/full_p Omega is described [here](http://research.google.com/pubs/pub41684.html). Borg is described [here](http://research.google.com/pubs/pub43438.html). - - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/node-performance-testing.md b/contributors/devel/node-performance-testing.md index 4e9e7c1a..d43737a8 100644 --- a/contributors/devel/node-performance-testing.md +++ b/contributors/devel/node-performance-testing.md @@ -26,7 +26,7 @@ Heapster will hide the performance cost of serving those stats in the Kubelet. Disabling addons is simple. Just ssh into the Kubernetes master and move the addon from `/etc/kubernetes/addons/` to a backup location. More details -[here](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/). +[here](https://git.k8s.io/kubernetes/cluster/addons/). ### Which / how many pods? @@ -57,7 +57,7 @@ sampling. ## E2E Performance Test There is an end-to-end test for collecting overall resource usage of node -components: [kubelet_perf.go](https://github.com/kubernetes/kubernetes/tree/master/test/e2e/kubelet_perf.go). To +components: [kubelet_perf.go](https://git.k8s.io/kubernetes/test/e2e/node/kubelet_perf.go). To run the test, simply make sure you have an e2e cluster running (`go run hack/e2e.go -- -up`) and [set up](#cluster-set-up) correctly. @@ -119,8 +119,3 @@ More details on benchmarking [here](https://golang.org/pkg/testing/). - (yujuhong) Measuring memory usage - Add section on monitoring kubelet metrics (e.g. with prometheus) - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/on-call-build-cop.md b/contributors/devel/on-call-build-cop.md deleted file mode 100644 index e83fd8d0..00000000 --- a/contributors/devel/on-call-build-cop.md +++ /dev/null @@ -1,50 +0,0 @@ -# Kubernetes BuildCop Workflow - -June 2017 - -## Objective - -This document describes the responsibilities and the workflow of a person assuming the buildcop role. -The current buildcop can be found [here](https://storage.googleapis.com/kubernetes-jenkins/oncall.html). - -## Prerequisites for build-copping - -- Ensure you have admin access to [http://github.com/kubernetes/kubernetes](http://github.com/kubernetes/kubernetes) - - Check your membership in the GitHub team: [kubernetes-build-cops](https://github.com/orgs/kubernetes/teams/kubernetes-build-cops/members). - If you are not a member contact one of the team maintainers to get yourself added to it. - - Test your admin access by e.g. adding a label to an issue. -- You must communicate any concerns/actions via the **#sig-release** slack channel to ensure that -the release team has context on the current state of the submit queue. -- You must attend the release burndown meeting to provide an update on the current state of the submit-queue - -## Responsibilities - -The build-cop's primary responsibility is to ensure that automatic merges are happening at a -**reasonable** rate. This may include performing merging of test flake PRs when the pre-submits -are failing repeatedly. The buildcop must be familiar with the -[queue labels](https://submit-queue.k8s.io/#/info) and apply them as necessary to critical fixes. -The priority labels are defunct and no longer respected by the submit-queue. As of June 2017, -the merge rate is ~30 PRs per day if there are that many PRs in the queue. The previous -responsibilities of this role included classification of incoming issues, but that is no -longer a part of the mandate. - -## Workflow - -1. Check the Prow batch dashboard: [https://prow.k8s.io/?type=batch](https://prow.k8s.io/?type=batch) -to ensure that batch jobs are running regularly. It's okay to see occasional flakes. Do not worry -about manually re-running individual tests, since Prow will rerun them. -2. If there are post-submit blocking jobs (see [link](https://submit-queue.k8s.io/#/e2e)), ensure -that those builds are green and allowing merges to occur. -3. If several batch merges are failing, file an issue for that job and describe the possible -causes for the failure. Debug if possible, else triage and assign to a particular SIG, and -@-mention the maintainers. For example, see: -[#47135](https://github.com/kubernetes/kubernetes/issues/47135) -4. Communicate the actions to **#sig-release** via slack and ensure that the issue is being worked on. -5. If the issue is not worked on for several hours, please escalate to the release team. - The release team members can be found via the [features](https://github.com/kubernetes/features) repo. - For example, the Kubernetes 1.7 release team members are listed [here](https://github.com/kubernetes/features/blob/master/release-1.7/release_team.md). - Notify the release manager/release team members via GitHub mentions and slack. -6. When the SIG member sends a fix, manually merge if necessary, after verifying that pre-submits pass, -or use the 'retest-not-required' label with the appropriate 'queue/*' label to ensure merge of the -flake fix. -7. Issue an update to the **#sig-release** channel on the merge rate and the PR that was used to fix the queue. diff --git a/contributors/devel/on-call-federation-build-cop.md b/contributors/devel/on-call-federation-build-cop.md index 892bf21c..708c854a 100644 --- a/contributors/devel/on-call-federation-build-cop.md +++ b/contributors/devel/on-call-federation-build-cop.md @@ -1,45 +1,48 @@ # Federation Buildcop Guide and Playbook -Federation runs two classes of tests: CI and Presubmits. +Federation runs two classes of tests: CI and Pre-submits. ## CI * These tests run on the HEADs of master and release branches (starting - from Kubernetes v1.6). + from Kubernetes v1.7). * As a result, they run on code that's already merged. * As the name suggests, they run continuously. Currently, they are - configured to run - [at least once every 30 minutes](https://github.com/kubernetes/test-infra/blob/22c38cfb64137086373e1b89d5e7d98766560747/prow/config.yaml#L3686). -* Federation CI tests run as - [periodic jobs on prow](https://github.com/kubernetes/test-infra/blob/22c38cfb64137086373e1b89d5e7d98766560747/prow/config.yaml#L3686). + configured to run at least once every 30 minutes. +* Federation CI tests run as periodic jobs on prow. * CI jobs always run sequentially. In other words, no single CI job can have two instances of the job running at the same time. +* Latest build results can be viewed in [testgrid](https://k8s-testgrid.appspot.com/sig-multicluster) ### Configuration -Configuration steps are described in https://github.com/kubernetes/test-infra/blob/0c56d2c9d32307c0a0f8fece85ef6919389e77fd/jenkins/README.md#how-to-work-with-jenkins-jobs +Configuration steps are described in https://github.com/kubernetes/test-infra#create-a-new-job. +Federation CI e2e job names are as below: +* master branch - `ci-federation-e2e-gce` and `ci-federation-e2e-gce-serial` +* 1.8 release branch - `ci-kubernetes-e2e-gce-federation-release-1-8` +* 1.7 release branch - `ci-kubernetes-e2e-gce-federation-release-1-7` -The configuration of CI tests are stored in: +Search for the above job names in various configuration files as below: -* Jenkins config: https://github.com/kubernetes/test-infra/blob/0c56d2c9d32307c0a0f8fece85ef6919389e77fd/jenkins/job-configs/kubernetes-jenkins/bootstrap-ci.yaml -* Test job/bootstrap config: https://github.com/kubernetes/test-infra/blob/0c56d2c9d32307c0a0f8fece85ef6919389e77fd/jobs/config.json -* Test grid config: https://github.com/kubernetes/test-infra/blob/0c56d2c9d32307c0a0f8fece85ef6919389e77fd/testgrid/config/config.yaml -* Job specific config: https://github.com/kubernetes/test-infra/tree/0c56d2c9d32307c0a0f8fece85ef6919389e77fd/jobs +* Prow config: https://git.k8s.io/test-infra/prow/config.yaml +* Test job/bootstrap config: https://git.k8s.io/test-infra/jobs/config.json +* Test grid config: https://git.k8s.io/test-infra/testgrid/config/config.yaml +* Job specific config: https://git.k8s.io/test-infra/jobs/env ### Results -Results of all the federation CI tests, including the soak tests, are -listed in the corresponding tabs on the Cluster Federation page in the -testgrid. -https://k8s-testgrid.appspot.com/sig-federation +Results of all the federation CI tests are listed in the corresponding +tabs on the Cluster Federation page in the testgrid. +https://k8s-testgrid.appspot.com/sig-multicluster ### Playbook #### Triggering a new run -Please ping someone who has access to the Jenkins UI/dashboard and ask -them to login and click the "Build Now" link on the Jenkins page -corresponding to the CI job you want to manually start. +Please ping someone who has access to the prow project and ask +them to click the `rerun` button from, for example +http://prow.k8s.io/?type=periodic&job=ci-federation-e2e-gce, +and execute the kubectl command. #### Quota cleanup @@ -49,115 +52,54 @@ delete button corresponding to those leaked resources on Google Cloud Console. -## Presubmit - -* We only have one presubmit test, but it is configured very - differently than the CI tests. -* The presubmit test is currently configured to run on the master - branch and any release branch that's 1.7 or newer. -* Federation presubmit infrastructure is composed of two separate test - jobs: - * Deploy job: This job runs in the background and recycles federated - clusters every time it runs. Although this job supports federation - presubmit tests, it is configured as a CI/Soak job. More on - configuration later. Since recycling federated clusters is an - expensive operation, we do not want to run this often. Hence, this - job is configured to run once every 24 hours, around midnight - Pacific time. - * Test job: This is the job that runs federation presubmit tests on - every PR in the core repository, i.e. - [kubernetes/kubernetes](https://github.com/kubernetes/kubernetes). - These jobs can run in parallel on the PRs in the repository. - -### Two-jobs setup - -The deploy job runs once every 24 hours at around midnight Pacific -time. It is configured to turn up and tear down 3 federated clusters. -It starts out by downloading the latest Kubernetes release built from -[kubernetes/kubernetes](https://github.com/kubernetes/kubernetes) -master. It then tears down the existing federated clusters and turns -up new ones. As the clusters are created, their kubeconfigs are -written to a local kubeconfig file where the job runs. Once all the -clusters are successfully turned up, the local kubeconfig is then -copied to a pre-configured GCS bucket. Any existing kubeconfig in the -bucket will be overwritten. - -The test job on the other hand starts by copying the latest kubeconfig -from the pre-configured GCS bucket. It uses this kubeconfig to deploy -a new federation control plane on one of the clusters in the -kubeconfig. It then joins all the clusters in the kubeconfig, including -the host cluster where federation control plane is deployed, as members -to the newly created federation control plane. The test job then runs -the federation presubmit tests on this control plane and tears down the -control plane in the end. - -Since federated clusters are recycled only once every 24 hours, all -presubmit runs in that period share the federated clusters. And since -there could be multiple presubmit tests running in parallel, each -instance of the test gets its own namespace where it deploys the -federation control plane. These federation control planes deployed in -separate namespaces are independent of each other and do not interfere -with other federation control planes in any way. +## Pre-submit -### Configuration - -The two jobs are configured differently. - -#### Deploy job - -The deploy job is configured as a CI/Soak job in Jenkins. -Configuration steps are described in https://github.com/kubernetes/test-infra/blob/0c56d2c9d32307c0a0f8fece85ef6919389e77fd/jenkins/README.md#how-to-work-with-jenkins-jobs - -The configuration of the deploy job is stored in: +* The pre-submit test is currently configured to run on the master + branch and any release branch that's 1.9 or newer. +* Multiple pre-submit jobs could be running in parallel(one per pr). +* Latest build results can be viewed in [testgrid](https://k8s-testgrid.appspot.com/presubmits-federation) +* We have following pre-submit jobs in federation + * bazel-test - Runs all the bazel test targets in federation. + * e2e-gce - Runs federation e2e tests on gce. + * verify - Runs federation unit, integration tests and few verify scripts. -* Jenkins config: https://github.com/kubernetes/test-infra/blob/0c56d2c9d32307c0a0f8fece85ef6919389e77fd/jenkins/job-configs/kubernetes-jenkins/bootstrap-ci-soak.yaml#L76 -* Test job/bootstrap config: https://github.com/kubernetes/test-infra/blob/0c56d2c9d32307c0a0f8fece85ef6919389e77fd/jobs/config.json#L3996 -* Test grid config: https://github.com/kubernetes/test-infra/blob/0c56d2c9d32307c0a0f8fece85ef6919389e77fd/testgrid/config/config.yaml#L152 -* Job specific config: https://github.com/kubernetes/test-infra/blob/0c56d2c9d32307c0a0f8fece85ef6919389e77fd/jobs/ci-kubernetes-pull-gce-federation-deploy.env - -#### Test job +### Configuration -The test job is -[configured in prow](https://github.com/kubernetes/test-infra/blob/35ceb37e999bb0589218708262634951b79dfe05/prow/config.yaml#L236), -but it runs in Jenkins mode. The configuration steps are described in -https://github.com/kubernetes/test-infra/blob/0c56d2c9d32307c0a0f8fece85ef6919389e77fd/README.md#create-a-new-job +Configuration steps are described in https://github.com/kubernetes/test-infra#create-a-new-job. +Federation pre-submit jobs have following names. +* bazel-test - `pull-federation-bazel-test` +* verify - `pull-federation-verify` +* e2e-gce - `pull-federation-e2e-gce` -The configuration of the test job is stored in: +Search for the above job names in various configuration files as below: -* Prow config: https://github.com/kubernetes/test-infra/blob/0c56d2c9d32307c0a0f8fece85ef6919389e77fd/prow/config.yaml#L244 -* Test job/bootstrap config: https://github.com/kubernetes/test-infra/blob/0c56d2c9d32307c0a0f8fece85ef6919389e77fd/jobs/config.json#L4691 -* Job specific config: https://github.com/kubernetes/test-infra/blob/0c56d2c9d32307c0a0f8fece85ef6919389e77fd/jobs/pull-kubernetes-federation-e2e-gce.env +* Prow config: https://git.k8s.io/test-infra/prow/config.yaml +* Test job/bootstrap config: https://git.k8s.io/test-infra/jobs/config.json +* Test grid config: https://git.k8s.io/test-infra/testgrid/config/config.yaml +* Job specific config: https://git.k8s.io/test-infra/jobs/env ### Results Aggregated results are available on the Gubernator dashboard page for -the federation presubmit tests. +the federation pre-submit tests. -https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-federation-e2e-gce +https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-federation-e2e-gce ### Metrics -We track the flakiness metrics of all the presubmit jobs and +We track the flakiness metrics of all the pre-submit jobs and individual tests that run against PRs in -[kubernetes/kubernetes](https://github.com/kubernetes/kubernetes). +[kubernetes/federation](https://github.com/kubernetes/federation). -* The metrics that we track are documented in https://github.com/kubernetes/test-infra/blob/0c56d2c9d32307c0a0f8fece85ef6919389e77fd/metrics/README.md#metrics. -* Job-level metrics are available in - [http://storage.googleapis.com/k8s-metrics/job-flakes-latest.json](). -* As of this writing, federation presubmits have a [success rate of - 93.4%](http://storage.googleapis.com/k8s-metrics/job-flakes-latest.json). +* The metrics that we track are documented in https://git.k8s.io/test-infra/metrics/README.md#metrics. +* Job-level metrics are available in http://storage.googleapis.com/k8s-metrics/job-flakes-latest.json. ### Playbook -#### Triggering a new deploy job run - -Please ping someone who has access to the Jenkins UI/dashboard and ask -them to login and click the "Build Now" link on the Jenkins page -corresponding to the CI job you want to manually start. - -#### Triggering a new test run +#### Triggering a new run -Use the `/test` command on the PR to retrigger the test. The exact -incantation is: `/test pull-kubernetes-federation-e2e-gce` +Use the `/test` command on the PR to re-trigger the test. The exact +incantation is: `/test pull-federation-e2e-gce` #### Quota cleanup diff --git a/contributors/devel/on-call-rotations.md b/contributors/devel/on-call-rotations.md deleted file mode 100644 index a6535e82..00000000 --- a/contributors/devel/on-call-rotations.md +++ /dev/null @@ -1,43 +0,0 @@ -## Kubernetes On-Call Rotations - -### Kubernetes "first responder" rotations - -Kubernetes has generated a lot of public traffic: email, pull-requests, bugs, -etc. So much traffic that it's becoming impossible to keep up with it all! This -is a fantastic problem to have. In order to be sure that SOMEONE, but not -EVERYONE on the team is paying attention to public traffic, we have instituted -two "first responder" rotations, listed below. Please read this page before -proceeding to the pages linked below, which are specific to each rotation. - -Please also read our [notes on OSS collaboration](collab.md), particularly the -bits about hours. Specifically, each rotation is expected to be active primarily -during work hours, less so off hours. - -During regular workday work hours of your shift, your primary responsibility is -to monitor the traffic sources specific to your rotation. You can check traffic -in the evenings if you feel so inclined, but it is not expected to be as highly -focused as work hours. For weekends, you should check traffic very occasionally -(e.g. once or twice a day). Again, it is not expected to be as highly focused as -workdays. It is assumed that over time, everyone will get weekday and weekend -shifts, so the workload will balance out. - -If you can not serve your shift, and you know this ahead of time, it is your -responsibility to find someone to cover and to change the rotation. If you have -an emergency, your responsibilities fall on the primary of the other rotation, -who acts as your secondary. If you need help to cover all of the tasks, partners -with oncall rotations (e.g., -[Redhat](https://github.com/orgs/kubernetes/teams/rh-oncall)). - -If you are not on duty you DO NOT need to do these things. You are free to focus -on "real work". - -Note that Kubernetes will occasionally enter code slush/freeze, prior to -milestones. When it does, there might be changes in the instructions (assigning -milestones, for instance). - -* [Github and Build Cop Rotation](on-call-build-cop.md) -* [User Support Rotation](on-call-user-support.md) - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/on-call-user-support.md b/contributors/devel/on-call-user-support.md deleted file mode 100644 index 0b3eb1bb..00000000 --- a/contributors/devel/on-call-user-support.md +++ /dev/null @@ -1,83 +0,0 @@ -## Kubernetes "User Support" Rotation - -### Traffic sources and responsibilities - -* [Stack Overflow](http://stackoverflow.com/questions/tagged/kubernetes) and -[ServerFault](http://serverfault.com/questions/tagged/google-kubernetes): -Respond to any thread that has no responses and is more than 6 hours old (over -time we will lengthen this timeout to allow community responses). If you are not -equipped to respond, it is your job to redirect to someone who can. - - * [Query for unanswered Kubernetes Stack Overflow questions](http://stackoverflow.com/search?tab=newest&q=%5bkubernetes%5d%20answers%3a0) - * [Query for unanswered Kubernetes ServerFault questions](https://serverfault.com/search?tab=newest&q=%5bgoogle-kubernetes%5d%20answers%3a0) - * Direct poorly formulated questions to [Stack Overflow's tips about how to ask](http://stackoverflow.com/help/how-to-ask) - * Direct off-topic questions to [Stack Overflow's policy](http://stackoverflow.com/help/on-topic) - -* [Slack](https://kubernetes.slack.com) ([registration](http://slack.k8s.io)): -Your job is to be on Slack, watching for questions and answering or redirecting -as needed, such as to a SIG-specific channel. Please especially watch -`#kubernetes-users` and `#kubernetes-novice`. Also check out the -[Slack Archive](http://kubernetes.slackarchive.io/). - -* [Email/Groups](https://groups.google.com/forum/#!forum/kubernetes-users): -Respond to any thread that has no responses and is more than 6 hours old (over -time we will lengthen this timeout to allow community responses). If you are not -equipped to respond, it is your job to redirect to someone who can. - -* on slack: Respond to questions that -don't get answers. - -In general, try to direct support questions to: - -1. Documentation, such as the [user documentation](https://kubernetes.io/docs/) and -[troubleshooting guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/) - -2. Stack Overflow - -#### User support response example - -If you see questions on kubernetes-dev@googlegroups.com, try to redirect them -to Stack Overflow. Example response: - -```code -Please re-post your question to [Stack Overflow] -(http://stackoverflow.com/questions/tagged/kubernetes). - -We are trying to consolidate the channels to which questions for help/support -are posted so that we can improve our efficiency in responding to your requests, -and to make it easier for you to find answers to frequently asked questions and -how to address common use cases. - -We regularly see messages posted in multiple forums, with the full response -thread only in one place or, worse, spread across multiple forums. Also, the -large volume of support issues on github is making it difficult for us to use -issues to identify real bugs. - -The Kubernetes team scans Stack Overflow on a regular basis, and will try to -ensure your questions don't go unanswered. - -Before posting a new question, please search Stack Overflow for answers to -similar questions, and also familiarize yourself with: - - * [user documentation](http://kubernetes.io/docs/) - * [troubleshooting guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/) - -Again, thanks for using Kubernetes. - -The Kubernetes Team -``` - -If you answer a question (in any of the above forums) that you think might be -useful for someone else in the future, please send a PR or file an issue in -[kubernetes.github.io](https://github.com/kubernetes/kubernetes.github.io). - -### Contact information - -[@k8s-support-oncall](https://github.com/k8s-support-oncall) will reach the -current person on call. - - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/owners.md b/contributors/devel/owners.md index aea8579e..489cf309 100644 --- a/contributors/devel/owners.md +++ b/contributors/devel/owners.md @@ -16,7 +16,7 @@ of OWNERS files ## OWNERS spec The [mungegithub gitrepos -feature](https://github.com/kubernetes/test-infra/blob/master/mungegithub/features/repo-updates.go) +feature](https://git.k8s.io/test-infra/mungegithub/features/repo-updates.go) is the main consumer of OWNERS files. If this page is out of date, look there. Each directory that contains a unit of independent code or content may also contain an OWNERS file. @@ -72,7 +72,7 @@ GitHub usernames and aliases listed in OWNERS files are case-insensitive. ## Code Review Process This is a simplified description of our [full PR testing and merge -workflow](https://github.com/kubernetes/community/blob/master/contributors/devel/pull-requests.md#the-testing-and-merge-workflow) +workflow](/contributors/devel/pull-requests.md#the-testing-and-merge-workflow) that conveniently forgets about the existence of tests, to focus solely on the roles driven by OWNERS files. @@ -158,13 +158,13 @@ is the state of today. ## Implementation -### [`mungegithub`](https://github.com/kubernetes/test-infra/tree/master/mungegithub) +### [`mungegithub`](https://git.k8s.io/test-infra/mungegithub) Mungegithub polls GitHub, and "munges" things it finds, including issues and pull requests. It is stateful, in that restarting it means it loses track of which things it has munged at what time. - [feature: - gitrepos](https://github.com/kubernetes/test-infra/blob/master/mungegithub/features/repo-updates.go) + gitrepos](https://git.k8s.io/test-infra/mungegithub/features/repo-updates.go) - responsible for parsing OWNERS and OWNERS_ALIAS files - if its `use-reviewers` flag is set to false, **approvers** will also be **reviewers** - if its `enable-md-yaml` flag is set, `.md` files will also be parsed to see if they have @@ -172,14 +172,14 @@ stateful, in that restarting it means it loses track of which things it has mung [kubernetes.github.io](https://github.com/kubernetes/kubernetes.github.io/)) - used by other mungers to get the set of **reviewers** or **approvers** for a given path - [munger: - blunderbuss](https://github.com/kubernetes/test-infra/blob/master/mungegithub/mungers/blunderbuss.go) + blunderbuss](https://git.k8s.io/test-infra/mungegithub/mungers/blunderbuss.go) - responsible for determining **reviewers** and assigning to them - chooses from people in the deepest/closest OWNERS files to the code being changed - weights its choice based on the magnitude of lines changed for each file - randomly chooses to ensure the same people aren't chosen every time - if its `blunderbuss-number-assignees` flag is unset, it will default to 2 assignees - [munger: - approval-handler](https://github.com/kubernetes/test-infra/blob/master/mungegithub/mungers/approval-handler.go) + approval-handler](https://git.k8s.io/test-infra/mungegithub/mungers/approval-handler.go) - responsible for adding the `approved` label once an **approver** for each of the required OWNERS files has `/approve`'d - responsible for commenting as required OWNERS files are satisfied @@ -187,19 +187,19 @@ stateful, in that restarting it means it loses track of which things it has mung - [full description of the algorithm](https://github.com/kubernetes/test-infra/blob/6f5df70c29528db89d07106a8156411068518cbc/mungegithub/mungers/approval-handler.go#L99-L111) - [munger: - submit-queue](https://github.com/kubernetes/test-infra/blob/master/mungegithub/mungers/submit-queue.go) + submit-queue](https://git.k8s.io/test-infra/mungegithub/mungers/submit-queue.go) - responsible for merging PR's - responsible for updating a GitHub status check explaining why a PR can't be merged (eg: a missing `lgtm` or `approved` label) -### [`prow`](https://github.com/kubernetes/test-infra/tree/master/prow) +### [`prow`](https://git.k8s.io/test-infra/prow) Prow receives events from GitHub, and reacts to them. It is effectively stateless. -- [plugin: lgtm](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/lgtm) +- [plugin: lgtm](https://git.k8s.io/test-infra/prow/plugins/lgtm) - responsible for adding the `lgtm` label when a **reviewer** comments `/lgtm` on a PR - the **PR author** may not `/lgtm` their own PR -- [plugin: assign](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/assign) +- [plugin: assign](https://git.k8s.io/test-infra/prow/plugins/assign) - responsible for assigning GitHub users in response to `/assign` comments on a PR - responsible for unassigning GitHub users in response to `/unassign` comments on a PR diff --git a/contributors/devel/profiling.md b/contributors/devel/profiling.md index f50537f1..abe0ba78 100644 --- a/contributors/devel/profiling.md +++ b/contributors/devel/profiling.md @@ -40,7 +40,3 @@ to get 30 sec. CPU profile. To enable contention profiling you need to add line `rt.SetBlockProfileRate(1)` in addition to `m.mux.HandleFunc(...)` added before (`rt` stands for `runtime` in `master.go`). This enables 'debug/pprof/block' subpage, which can be used as an input to `go tool pprof`. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/pull-requests.md b/contributors/devel/pull-requests.md index 8381ceca..50e457a2 100644 --- a/contributors/devel/pull-requests.md +++ b/contributors/devel/pull-requests.md @@ -2,14 +2,13 @@ This doc explains the process and best practices for submitting a PR to the [Kubernetes project](https://github.com/kubernetes/kubernetes). It should serve as a reference for all contributors, and be useful especially to new and infrequent submitters. -<!-- BEGIN MUNGE: GENERATED_TOC --> -- [Pull Request Process](#pull-request-process) - [Before You Submit a PR](#before-you-submit-a-pr) * [Run Local Verifications](#run-local-verifications) * [Sign the CLA](#sign-the-cla) - [The PR Submit Process](#the-pr-submit-process) * [Write Release Notes if Needed](#write-release-notes-if-needed) * [The Testing and Merge Workflow](#the-testing-and-merge-workflow) + * [Marking Unfinished Pull Requests](#marking-unfinished-pull-requests) * [Comment Commands Reference](#comment-commands-reference) * [Automation](#automation) * [How the e2e Tests Work](#how-the-e2e-tests-work) @@ -17,20 +16,20 @@ This doc explains the process and best practices for submitting a PR to the [Kub - [Why is my PR not getting reviewed?](#why-is-my-pr-not-getting-reviewed) - [Best Practices for Faster Reviews](#best-practices-for-faster-reviews) * [0. Familiarize yourself with project conventions](#0-familiarize-yourself-with-project-conventions) - * [1. Is the feature wanted? Make a Design Doc or Sketch PR](#1-is-the-feature-wanted--make-a-design-doc-or-sketch-pr) - * [2. Smaller Is Better: Small Commits, Small PRs](#2-smaller-is-better--small-commits--small-prs) + * [1. Is the feature wanted? Make a Design Doc or Sketch PR](#1-is-the-feature-wanted-make-a-design-doc-or-sketch-pr) + * [2. Smaller Is Better: Small Commits, Small PRs](#2-smaller-is-better-small-commits-small-prs) * [3. Open a Different PR for Fixes and Generic Features](#3-open-a-different-pr-for-fixes-and-generic-features) * [4. Comments Matter](#4-comments-matter) * [5. Test](#5-test) * [6. Squashing and Commit Titles](#6-squashing-and-commit-titles) - * [7. KISS, YAGNI, MVP, etc.](#7-kiss--yagni--mvp--etc) - * [8. It's OK to Push Back](#8-it-s-ok-to-push-back) - * [Common Sense and Courtesy](#common-sense-and-courtesy) -<!-- END MUNGE: GENERATED_TOC --> + * [7. KISS, YAGNI, MVP, etc.](#7-kiss-yagni-mvp-etc) + * [8. It's OK to Push Back](#8-its-ok-to-push-back) + * [9. Common Sense and Courtesy](#9-common-sense-and-courtesy) + # Before You Submit a PR -This guide is for contributors who already have a PR to submit. If you're looking for information on setting up your developer environment and creating code to contribute to Kubernetes, see the [development guide](development.md). +This guide is for contributors who already have a PR to submit. If you're looking for information on setting up your developer environment and creating code to contribute to Kubernetes, see the [development guide](development.md). **Make sure your PR adheres to our best practices. These include following project conventions, making small PRs, and commenting thoroughly. Please read the more detailed section on [Best Practices for Faster Reviews](#best-practices-for-faster-reviews) at the end of this doc.** @@ -45,9 +44,10 @@ pass or fail of continuous integration. ## Sign the CLA -You must sign the CLA before your first contribution. [Read more about the CLA.](https://github.com/kubernetes/community/blob/master/CLA.md) +You must sign the CLA before your first contribution. [Read more about the CLA.](/CLA.md) -If you haven't signed the Contributor License Agreement (CLA) before making a PR, the `k8s-ci-robot` will leave a comment with instructions on how to sign the CLA. +If you haven't signed the Contributor License Agreement (CLA) before making a PR, +the `@k8s-ci-robot` will leave a comment with instructions on how to sign the CLA. # The PR Submit Process @@ -57,9 +57,9 @@ Merging a PR requires the following steps to be completed before the PR will be - Make the PR - Release notes - do one of the following: - Add notes in the release notes block, or - - Update the `release-note-label-needed` label + - Update the release note label - Pass all e2e tests -- Get a `LGTM` from a reviewer +- Get a `/lgtm` from a reviewer - Get approval from an owner If your PR meets all of the steps above, it will enter the submit queue to be merged. When it is next in line to be merged, the e2e tests will run a second time. If all tests pass, the PR will be merged automatically. @@ -68,9 +68,7 @@ If your PR meets all of the steps above, it will enter the submit queue to be me Release notes are required for any PR with user-visible changes, such as bug-fixes, feature additions, and output format changes. -If you don't add release notes in the PR template the `release-note-label-needed` label is added to your PR automatically when you create it. There are a few ways to update it. - -**Descriptions** +If you don't add release notes in the PR template, the `do-not-merge/release-note-label-needed` label is added to your PR automatically after you create it. There are a few ways to update it. To add a release-note section to the PR description: @@ -80,71 +78,80 @@ For PRs with a release note: Your release note here ``` -For PRs without a release note: +For PRs that require additional action from users switching to the new release, include the string "action required" (case insensitive) in the release note: ```release-note - NONE + action required: your release note here ``` -To see how to format your release notes, view the [PR template](https://github.com/kubernetes/kubernetes/blob/master/.github/PULL_REQUEST_TEMPLATE.md) for a brief example. PR titles and body comments can be modified at any time prior to the release to make them friendly for release notes. - -Release notes apply to PRs on the master branch. For cherry-pick PRs, see the [cherry-pick instructions](cherry-picks.md). The only exception to these rules is when a PR is not a cherry-pick and is targeted directly to the non-master branch. In this case, a `release-note-*` label is required for that non-master PR. +For PRs that don't need to be mentioned at release time, just write "NONE" (case insensitive): -**Labels** + ```release-note + NONE + ``` -1. All pull requests are initiated with a `release-note-label-needed` label if you don't specify them in your original PR. If you are a new contributor you won't have access to modify labels; instead, leave a comment as instructed below or ask in your original PR. +The `/release-note-none` comment command can still be used as an alternative to writing "NONE" in the release-note block if it is left empty. -1. Remove the `release-note-label-needed` label and replace it with one of the other `release-note-*` labels: - 1. `release-note-none` is a valid option if the PR does not need to be mentioned at release time - 1. `release-note` labelled PRs generate a release note using the PR title by default OR the release-note block in the PR template, if it's filled in - -**Comments** +To see how to format your release notes, view the [PR template](https://git.k8s.io/kubernetes/.github/PULL_REQUEST_TEMPLATE.md) for a brief example. PR titles and body comments can be modified at any time prior to the release to make them friendly for release notes. -Or, commenting either `/release-note` or `/release-note-none` will also set the `release-note` or `release-note-none` labels respectively. +Release notes apply to PRs on the master branch. For cherry-pick PRs, see the [cherry-pick instructions](cherry-picks.md). The only exception to these rules is when a PR is not a cherry-pick and is targeted directly to the non-master branch. In this case, a `release-note-*` label is required for that non-master PR. Now that your release notes are in shape, let's look at how the PR gets tested and merged. ## The Testing and Merge Workflow -The Kubernetes merge workflow uses comments to run tests and labels for merging PRs. NOTE: For pull requests that are in progress but not ready for review, prefix the PR title with "WIP" and track any remaining TODOs in a checklist in the pull request description. +The Kubernetes merge workflow uses comments to run tests and labels for merging PRs. +NOTE: For pull requests that are in progress but not ready for review, +prefix the PR title with `WIP` or `[WIP]` and track any remaining TODOs +in a checklist in the pull request description. Here's the process the PR goes through on its way from submission to merging: 1. Make the pull request -1. mergebot assigns reviewers +1. `@k8s-merge-robot` assigns reviewers If you're **not** a member of the Kubernetes organization: -1. Reviewer/Kubernetes Member checks that the PR is safe to test. If so, they comment `@k8s-bot ok to test` +1. Reviewer/Kubernetes Member checks that the PR is safe to test. If so, they comment `/ok-to-test` 1. Reviewer suggests edits 1. Push edits to your PR branch 1. Repeat the prior two steps as needed -1. (Optional) Some reviewers prefer that you squash commits at this step +1. (Optional) Some reviewers prefer that you squash commits at this step 1. Owner is assigned and will add the `/approve` label to the PR -If you are a member, or a member comments `@k8s-bot ok to test`, the pre-submit tests will run: +If you are a member, or a member comments `/ok-to-test`, the PR will be considered to be trusted. Then the pre-submit tests will run: 1. Automatic tests run. See the current list of tests on the [MERGE REQUIREMENTS tab, at this link](https://submit-queue.k8s.io/#/info) 1. If tests fail, resolve issues by pushing edits to your PR branch -1. If the failure is a flake, a member can comment `@k8s-bot [e2e|unit] test this issue: #<flake issue>` +1. If the failure is a flake, anyone on trusted PRs can comment `/retest` to rerun failed tests Once the tests pass, all failures are commented as flakes, or the reviewer adds the labels `lgtm` and `approved`, the PR enters the final merge queue. The merge queue is needed to make sure no incompatible changes have been introduced by other PRs since the tests were last run on your PR. -Either the [on call contributor](on-call-rotations.md) will manage the merge queue manually, or the [GitHub "munger"](https://github.com/kubernetes/test-infra/tree/master/mungegithub) submit-queue plugin will manage the merge queue automatically. +Either the [on call contributor](on-call-rotations.md) will manage the merge queue manually, or the [GitHub "munger"](https://git.k8s.io/test-infra/mungegithub) submit-queue plugin will manage the merge queue automatically. 1. The PR enters the merge queue ([http://submit-queue.k8s.io](http://submit-queue.k8s.io)) -1. The merge queue triggers a test re-run with the comment `@k8s-bot test this` - 1. Author has signed the CLA (`cla: yes` label added to PR) +1. The merge queue triggers a test re-run with the comment `/test all [submit-queue is verifying that this PR is safe to merge]` + 1. Author has signed the CLA (`cncf-cla: yes` label added to PR) 1. No changes made since last `lgtm` label applied 1. If tests fail, resolve issues by pushing edits to your PR branch -1. If the failure is a flake, a member can comment `@k8s-bot [e2e|unit] test this issue: #<flake issue>` +1. If the failure is a flake, anyone can comment `/retest` if the PR is trusted 1. If tests pass, the merge queue automatically merges the PR That's the last step. Your PR is now merged. +## Marking Unfinished Pull Requests + +If you want to solicit reviews before the implementation of your pull request is complete, you should hold your pull request to ensure that the merge queue does not pick it up and attempt to merge it. There are two methods to achieve this: + +1. You may add the `/hold` or `/hold cancel` comment commands +2. You may add or remove a `WIP` or `[WIP]` prefix to your pull request title + +The GitHub robots will add and remove the `do-not-merge/hold` label as you use the comment commands and the `do-not-merge/work-in-progress` label as you edit your title. While either label is present, your pull request will not be considered for merging. + + ## Comment Commands Reference -[The commands doc](https://github.com/kubernetes/test-infra/blob/master/commands.md) contains a reference for all comment commands. +[The commands doc](https://git.k8s.io/test-infra/commands.md) contains a reference for all comment commands. ## Automation @@ -152,9 +159,11 @@ The Kubernetes developer community uses a variety of automation to manage pull r ## How the e2e Tests Work -The end-to-end tests will post the status results to the PR. If an e2e test fails, `k8s-ci-robot` will comment on the PR with the test history and the comment-command to re-run that test. e.g. +The end-to-end tests will post the status results to the PR. If an e2e test fails, +`@k8s-ci-robot` will comment on the PR with the test history and the +comment-command to re-run that test. e.g. -> The magic incantation to run this job again is @k8s-bot unit test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR. +> The following tests failed, say /retest to rerun them all. # Why was my PR closed? @@ -180,7 +189,7 @@ things you can do to move the process along: * Ping the assignee (@username) on the PR comment stream, and ask for an estimate of when they can get to the review. - * Ping the assigned on [Slack](http://slack.kubernetes.io). Remember that a person's GitHub username might not be the same as their Slack username. + * Ping the assignee on [Slack](http://slack.kubernetes.io). Remember that a person's GitHub username might not be the same as their Slack username. * Ping the assignee by email (many of us have publicly available email addresses). @@ -211,8 +220,8 @@ Are you sure Feature-X is something the Kubernetes team wants or will accept? Is It's better to get confirmation beforehand. There are two ways to do this: -- Make a proposal doc (in docs/proposals; for example [the QoS proposal](http://prs.k8s.io/11713)), or reach out to the affected special interest group (SIG). Here's a [list of SIGs](https://github.com/kubernetes/community/blob/master/sig-list.md) -- Coordinate your effort with [SIG Docs](https://github.com/kubernetes/community/tree/master/sig-docs) ahead of time +- Make a proposal doc (in docs/proposals; for example [the QoS proposal](http://prs.k8s.io/11713)), or reach out to the affected special interest group (SIG). Here's a [list of SIGs](/sig-list.md) +- Coordinate your effort with [SIG Docs](/sig-docs) ahead of time - Make a sketch PR (e.g., just the API or Go interface). Write or code up just enough to express the idea and the design and why you made those choices Or, do all of the above. @@ -270,7 +279,7 @@ Feature-X. (Do that in its own commit or PR, please.) In your code, if someone might not understand why you did something (or you won't remember why later), comment it. Many code-review comments are about this exact issue. -If you think there's something pretty obvious that we could follow up on, add a TODO. +If you think there's something pretty obvious that we could follow up on, add a TODO. Read up on [GoDoc](https://blog.golang.org/godoc-documenting-go-code) - follow those general rules for comments. @@ -288,7 +297,7 @@ Make the fixups, and don't squash yet. Put them in a new commit, and re-push. Th We might still ask you to clean up your commits at the very end for the sake of a more readable history, but don't do this until asked: typically at the point where the PR would otherwise be tagged `LGTM`. -Each commit should have a good title line (<70 characters) and include an additional description paragraph describing in more detail the change intended. +Each commit should have a good title line (<70 characters) and include an additional description paragraph describing in more detail the change intended. **General squashing guidelines:** @@ -308,17 +317,13 @@ Sometimes we need to remind each other of core tenets of software design - Keep ## 8. It's OK to Push Back -Sometimes reviewers make mistakes. It's OK to push back on changes your reviewer requested. If you have a good reason for doing something a certain way, you are absolutely allowed to debate the merits of a requested change. Both the reviewer and reviewee should strive to discuss these issues in a polite and respectful manner. +Sometimes reviewers make mistakes. It's OK to push back on changes your reviewer requested. If you have a good reason for doing something a certain way, you are absolutely allowed to debate the merits of a requested change. Both the reviewer and reviewee should strive to discuss these issues in a polite and respectful manner. You might be overruled, but you might also prevail. We're pretty reasonable people. Mostly. Another phenomenon of open-source projects (where anyone can comment on any issue) is the dog-pile - your PR gets so many comments from so many people it becomes hard to follow. In this situation, you can ask the primary reviewer (assignee) whether they want you to fork a new PR to clear out all the comments. You don't HAVE to fix every issue raised by every person who feels like commenting, but you should answer reasonable comments with an explanation. -## Common Sense and Courtesy +## 9. Common Sense and Courtesy No document can take the place of common sense and good taste. Use your best judgment, while you put a bit of thought into how your work can be made easier to review. If you do these things your PRs will get merged with less friction. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/release/OWNERS b/contributors/devel/release/OWNERS index 6a3bfd86..afb042fa 100644 --- a/contributors/devel/release/OWNERS +++ b/contributors/devel/release/OWNERS @@ -3,10 +3,6 @@ reviewers: - pwittrock - steveperry-53 - chenopis - - sig-release + - spiffxp approvers: - - saad-ali - - pwittrock - - steveperry-53 - - chenopis - - sig-release + - sig-release-leads diff --git a/contributors/devel/release/README.md b/contributors/devel/release/README.md index 42cd20c5..d6eb9d6c 100644 --- a/contributors/devel/release/README.md +++ b/contributors/devel/release/README.md @@ -1,118 +1,3 @@ -# Kubernetes Release Roles -**Table of Contents** -* [Patch Release Manager](#patch-release-manager) -* [Kubernetes Release Management Team for Major/Minor Releases](#kubernetes-release-management-team-for-majorminor-releases) -* [Individual Contributors](#individual-contributors) +The original content of this file has been migrated to https://git.k8s.io/sig-release/ephemera/README.md -This document captures the requirements and duties of the individuals responsible for Kubernetes releases. - -As documented in the [Kubernetes Versioning doc](https://github.com/kubernetes/kubernetes/blob/master/docs/design/versioning.md), there are 3 types of Kubernetes releases: -* Major (x.0.0) -* Minor (x.x.0) -* Patch (x.x.x) - -Major and minor releases are managed by a **Kubernetes Release Management Team**, and patch releases are managed by the **Patch Release Manager**. Exact roles and duties are defined below. - -## Patch Release Manager - -Patch releases are managed by the **Patch Release Manager**. Duties of the patch release manager include: -* Ensuring the release branch (e.g. `release-1.5`) remains in a healthy state. - * If the build breaks or any CI for the release branch becomes unhealthy due to a bad merge or infrastructure issue, ensure that actions are taken ASAP to bring it back to a healthy state. -* Reviewing and approving [cherry picks](https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md) to the release branch. - * Patch releases should not contain new features, so ensure that cherry-picks are for bug/security fixes only. - * Cherry picks should not destabilize the branch, so ensure that either the PR has had time to stabilize in master or will have time to stabilize in the release branch before the next patch release is cut. -* Setting the exact schedule (and cadence) for patch releases and actually cutting the [releases](https://github.com/kubernetes/kubernetes/releases). - -See the [Patch Release Manager Playbook](patch-release-manager.md) for more details. - -Current and past patch release managers are listed [here](https://github.com/kubernetes/community/wiki). - -## Kubernetes Release Management Team for Major/Minor Releases - -Major and Minor releases are managed by the **Kubernetes Release Management Team** which is responsible for ensuring Kubernetes releases go out on time (as scheduled) and with high quality (stable, with no major bugs). - -Roles and responsibilities within the Kubernetes Release Management Team are as follows. - -#### Release Management Team Lead -The Release Management Team Lead is the person ultimately responsible for ensuring the release goes out on-time with high-quality. All the roles defined below report to the Release Management Team Lead. -* Establishes and communicates responsibilities and deadlines to release management team members, developers/feature owners, SIG leads, etc. -* Escalates and unblocks any issue that may jeopardise the release schedule or quality as quickly as possible. -* Finds people to take ownership of any release blocking issues that are not getting adequate attention. -* Keeps track of, and widely communicates, the status of the release (including status of all sub-leads, all release blockers, etc) and all deadlines leading up to release. -* Manages [exception](https://github.com/kubernetes/features/blob/master/EXCEPTIONS.md) process for features that want to merge after code freeze. - -#### Release Branch Manager -* Manages (initiates and enforces) code freeze on main branch as scheduled for the release. - * Ensures no new features are merged after code complete, unless they've been approved by the [exception process](https://github.com/kubernetes/features/blob/master/EXCEPTIONS.md). -* Cuts the `release-x.x` branch at the appropriate time during the milestone. -* Ensures release branch (e.g. `release-1.5`) remains in a healthy state for the duration of the major or minor release. - * If the build breaks, or any CI for the release branch becomes unhealthy due to a bad merge or infrastructure issue, ensures that actions are taken ASAP to bring it back to a healthy state. -* Initiates automatic fast-forwards of the release branch to pick up all changes from master branch, when appropriate. -* Reviews and approves [cherry picks](https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md) to the release branch. - * Ensures onlyl bug/security fixes (but no new features) are cherry-picked after code complete unless approved by the [exception process](https://github.com/kubernetes/features/blob/master/EXCEPTIONS.md). - * Ensures that cherry-picks do not destabilize the branch by either giving the PR enough time to stabilize in master or giving it enough time to stabilize in the release branch before cutting the release. -* Cuts the actual [release](https://github.com/kubernetes/kubernetes/releases). - -#### Docs Lead -* Sets docs related deadlines for developers and works with Release Management Team Lead to ensure they are widely communicated. -* Sets up release branch for docs. -* Pings feature owners to ensure that release docs are created on time. -* Reviews/merges release doc PRs. -* Merges the docs release branch to master to make release docs live as soon as the release is official. - -#### Features Lead -* Compiles the major themes, new features, known issues, actions required, notable changes to existing behavior, deprecations, etc. and edits them into a release doc checked in to the feature repository (ready to go out with the release). -* Collects and prepares the release notes - -#### Bug Triage Lead -* Figures out which bugs (whether manually created or automatically generated) should be tracked for the release. -* Ensures all bugs being tracked for the release have owners that are responsive. -* Ensures all bugs are triaged as blocking or non-blocking. -* Ensures all bugs that are blocking are being actively worked on, esp after code complete. - -#### Test Infra Lead -* Sets up and maintains all CI for the release branch. - -#### Automated Upgrade Testing Lead -* Ensures that automated upgrade tests provide a clear go/no-go signal for the release. -* Tracks and finds owners for all issues with automated upgrade tests. - -#### Manual Upgrade Testing Lead -* Ensures that any gaps in automated upgrade testing are covered by manual upgrade testing. -* Organizes the manual upgrade testing efforts, including setting up instructions for manual testing, finding manual testing volunteers, and ensuring any issues discovered are communicated widely and fixed quickly. - -#### Testing Lead -* Ensures that all non-upgrade test CI provides a clear go/no-go signal for the release. -* Tracks and finds owners to fix any issues with any (non-upgrade) tests. - -## Individual Contributors - -Release responsiblites of indvidual contributors to the Kubernetes project are captured below. - -### Patch Release - -#### Cherry Picks -If you have a patch that needs to be ported back to a previous release (meaning it is a critical bug/security fix), once it is merged to the Kubernetes `master` branch: -* Mark your PR with the milestone corresponding to the release you want to port back to (e.g. `v1.5`), and add the `cherrypick-candidate` label to it. -* The Patch Release Manager will then review the PR and if it is ok for cherry-picking, will apply a `cherrypick-approved` label to it. -* Once your PR has been marked with the `cherrypick-approved` label by the Patch Release Manager, you should prepare a cherry-pick to the requested branch following the instructions [here](https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md#how-do-cherrypick-candidates-make-it-to-the-release-branch). - -### Major/Minor Release - -#### Propose and Track Feature -If you are developing a feature for Kubernetes, make sure that an issue is open in the [features repository](https://github.com/kubernetes/features/issues). If you are targeting a particular release, make sure the issue is marked with the corresponding release milestone. - -Ensure that all code for your feature is written, tested, reviewed, and merged before code freeze date for the target release. - -During the code freeze period, fix any bugs discovered with you feature, and write feature documentation. - -##### Writing Feature Documentation - -1. Make sure your feature for the upcoming release is on the release tracking board (e.g. [link](https://docs.google.com/spreadsheets/d/1AFksRDgAt6BGA3OjRNIiO3IyKmA-GU7CXaxbihy48ns/edit?usp=sharing) for 1.8). -2. Create a PR with documentation for your feature in the [documents repo](https://github.com/kubernetes/kubernetes.github.io). - * **Your PR should target the release branch (e.g. [`release-1.8`](https://github.com/kubernetes/kubernetes.github.io/tree/release-1.8)), not the [`master`](https://github.com/kubernetes/kubernetes.github.io/tree/master) branch.** - * Any changes to the master branch become live on https://kubernetes.io/docs/ as soon as they are merged, and for releases we do not want docuemntation to go live until the release is cut. -3. Add link to your docs PR in the release tracking board, and notify the docs lead for the release (e.g. [Steve Perry](https://github.com/steveperry-53) for 1.8). -4. The docs lead will review your PR and give you feedback. -5. Once approved, the docs lead will merge your PR into the release branch. -6. When the release is cut, the docs lead will push the docs release branch to master, making your docs live on https://kubernetes.io/docs/. +This file is a placeholder to preserve links. Please remove after 3 months or the release of kubernetes 1.10, whichever comes first. diff --git a/contributors/devel/release/issues.md b/contributors/devel/release/issues.md index 3d7aded4..cccf12e9 100644 --- a/contributors/devel/release/issues.md +++ b/contributors/devel/release/issues.md @@ -1,191 +1,3 @@ -# Targeting issues and PRs to release milestones +The original content of this file has been migrated to https://git.k8s.io/sig-release/ephemera/issues.md -This document describes how to target issues and PRs to a release. -The process for shepherding issues into the release by the owner, release team, and GitHub bot -is outlined below. - -## Definitions - -- *issue owners*: creator, assignees, and user who moved the issue into a release milestone -- *Y days*: refers to business days (using the location local to the release-manager M-F) -- *code slush*: starts when master branch only accepts PRs for release milestone. no additional feature development is merged after this point. -- *code freeze*: starts 2 weeks after code slush. only critical bug fixes are accepted into the release codebase. - -## Requirements for adding an issue to the milestone - -**Note**: Issues with unmet label requirements will automatically be removed from the release milestone. - -When adding an issue to a milestone, the Kubernetes bot will check that the following -labels are set, and comment on the issue with the appropriate instructions. The -bot will attempt to contact the issue creator 3 times (over 3 days) -before automatically removing the issue from the milestone. - -Label categories: - -- SIG label owner -- Priority -- Issue type - -### SIG owner label - -The SIG owner label defines the SIG to which the bot will escalate if the issue is not resolved -or updated by the deadline. If there are no updates after escalation, the -issue may automatically removed from the milestone. - -e.g. `sig/node`, `sig/federation`, `sig/apps`, `sig/network` - -**Note:** - - For test-infrastructure issues use `sig/testing`. - - For GKE and GCE issues use `sig/gcp` once it is created, and `sig/cluster-lifecycle` until then. - -### Priority - -Priority label used by the bot to determine escalation path before moving an issues -out of the release milestone. Also used to determine whether or not a release should be -blocked on the resolution of the issue. - -- `priority/critical-urgent`: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. - - considered a release blocking issue - - code slush: issue owner update frequency: every 3 days - - code freeze: issue owner update frequency: daily - - would require a patch release if left undiscovered until after the minor release. -- `priority/important-soon`: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. - - not considered a release blocking issue - - would not require a patch release - - will automatically be moved out of the release milestone at code freeze -- `priority/important-longterm`: Escalate to the issue owners; move out of the milestone after 1 attempt. - - even less urgent / critical than `priority/important-soon` - - moved out of milestone more aggressively than `priority/important-soon` - -### Issue type - -The issue type is used to help identify the types of changes going into the release over time. -This will allow us to develop a better understanding of what sorts of issues we would miss -with a faster release cadence. - -This will also be used to escalate to the correct SIG GitHub team. - -- `kind/bug`: Fixes a newly discovered bug. - - were not known issues at the start of the development period. -- `kind/feature`: New functionality. -- `kind/cleanup`: Adding tests, refactoring, fixing old bugs. - -## Workflow - -1. An issue is added to the current release milestone (either through creation or update) - - Bot checks to make sure all required labels are set on the issue - - If and only if any labels are missing, comment listing the missing labels and include a link to this document. Mention all issue owners. - ``` - @pwittrock @droot - - **Action required**: Issue is missing the following required labels. Set the labels or the issue will be moved - out of the milestone within 2 days. - - - priority - - severity - - Additional instructions available [here](<link to this doc>) - ``` - - If and only if all labels are present, comment summarizing the state. Mention all issue owners. - ``` - @pwittrock @droot - - Issue label settings: - sig/node: Issue will be escalated to SIG node if needed - priority/critical-urgent: Never automatically move out of a release milestone. Escalate to SIG and contributor through all available channels. - kind/bug: Fixes a bug. - - Additional instructions available [here](<link to this doc>) - ``` - - **If required labels are not applied within 3 days of being moved to the milestone, the issue is moved back out of the milestone. (unless it is critical-urgent)** -2. If labels change, bot checks that the needed labels are present and comments with updated meaning. -3. Code slush - - **priority/important-* issues** - - bot comments mentioning issue owners, expects a 1 time response. - ``` - **Action Required**: - This issue is marked as priority/important-x, and is in the 1.y milestone. Please confirm the following - or the issue will be removed from the milestone. - - 1. it is still targeted at the milestone - 2. it has not been completed and should remain open - 3. it is being actively worked on by the assignee - - **Note**: This issue must be resolved or moved out of the milestone by <date of code freeze> or it will - automatically be moved out of the milestone - ``` - - after 2 days without a reply, bot follows escalation procedure derived from priority label - - important-longterm, remove from milestone and notify the owners - - important-soon, escalate to SIG daily for 2 days. - - removal should add the `release/removed-from-milestone` label and contain the following message - ``` - **Important**: - There has been no confirmation that this issue belongs in the v1.X milestone. - Removing it from the milestone. - ``` - - **priority/critical-urgent issues** - - bot comments on issues not updated within 3 days mentioning issue owners, expects updates every 3 days going forward. - ``` - **Action Required**: - This issue is marked as priority/critical-urgent, but has not been updated in 3 days. Please provide - an update or the issue will be escalated to the SIG. - ``` - - owner updates can be a short ACK, but should include an ETA for completion and any risk factors. - ``` - ACK. In progress - ETA: DD/MM/YYYY - Risks: Complicated fix required - ``` - - ``` - ACK. In progress - ETA: ??? - Risks: Root cause unknown. - ``` -4. Code freeze - - **priority/important-* issues** - - bot comments mentioning issue owners, expects the issue to be escalated in priority or removed from milestone within 2 days. - ``` - **Action Required**: - This issue is marked as priority/important-x, and is in the 1.y milestone. Only critical-urgent issue - are being tracked for the 1.y release. Escalate the priority to critical-urgent within 1 day, **or the issue - will be removed from the milestone**. - ``` - - bot immediately escalates to the SIG. escalation message includes link to all priority/important issues - for the SIG remaining in the milestone. - - **priority/critical-urgent issues** - - bot comments on issues not updated within 1 day mentioning issue owners, expects updates every day going forward. - - bot escalates to SIG after 2 days without updates and applies the label `release/needs-attention` - -## Escalation - -SIGs will have issues needing attention escalated through the following channels - -- Comment mentioning the sig team appropriate for the issue type -- Email the SIG googlegroup list - - bootstrapped with the emails from the [community sig list](https://github.com/kubernetes/community/blob/master/sig-list.md) - - maybe configured to email alternate googlegroup - - maybe configured to directly email SIG leads or other SIG members -- Message the SIG slack channel, mentioning the SIG leads - - bootstrapped with the slackchannel and SIG leads from the [community sig list](https://github.com/kubernetes/community/blob/master/sig-list.md) - - maybe configured to message alternate slack channel and users - -## Issues tracked in other repos - -Some issues are filed in repos outside the [kubernetes repo]. The bot must also be run against these -repos and follow the same pattern. The release team can query issues across repos in the kubernetes org using -a query like [this](https://github.com/issues?utf8=%E2%9C%93&q=is%3Aopen+is%3Aissue+milestone%3Av1.7+org%3Akubernetes+) - -If the bot is not setup against the split repo, the repo owners should setup an umbrella tracking issue -in the kubernetes/kubernetes repo and aggregate the status. - -`Release 1.<minor version> blocking umbrella: <repo name> (size: <number of open issues>)` - -it must also include: - -- a link to the repo with a query for issues in the milestone. See [this](https://github.com/kubernetes/kubectl/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20milestone%3Av1.7) example. -- a list of unresolved issues blocking the release. See -[this](https://github.com/kubernetes/kubernetes/issues/47747) example. - - -[kubernetes repo]: (https://github.com/kubernetes/kubernetes)
\ No newline at end of file +This file is a placeholder to preserve links. Please remove after 3 months or the release of kubernetes 1.10, whichever comes first. diff --git a/contributors/devel/release/patch-release-manager.md b/contributors/devel/release/patch-release-manager.md index 1e444fb9..da1290e5 100644 --- a/contributors/devel/release/patch-release-manager.md +++ b/contributors/devel/release/patch-release-manager.md @@ -1,258 +1,3 @@ -# Patch Release Manager Playbook - -This is a playbook intended to guide new patch release managers. -It consists of opinions and recommendations from former patch release managers. - -Note that patch release managers are ultimately responsible for carrying out -their [duties](README.md#patch-release-manager) in whatever manner they deem -best for the project. -The playbook is more what you call "guidelines" than actual rules. - -## Getting started - -* Add yourself to the [Release Manager table](https://github.com/kubernetes/community/wiki) - so the community knows you're the point of contact. -* Ask a maintainer to add you to the [kubernetes-release-managers](https://github.com/orgs/kubernetes/teams/kubernetes-release-managers/members) - team so you have write access to the main repository. -* Ask to be added to the [kubernetes-security](https://groups.google.com/forum/#!forum/kubernetes-security) - mailing list. -* Ask to be given access to post to the [kubernetes-announce](https://groups.google.com/forum/#!forum/kubernetes-announce) - and [kubernetes-dev-announce](https://groups.google.com/forum/#!forum/kubernetes-dev-announce) - mailing lists. -* Sync up with the outgoing release branch manager to take ownership of any - lingering issues on the branch. -* Run [anago](https://github.com/kubernetes/release) in mock mode to get prompts - for setting up your environment, and familiarize yourself with the tool. - -## Cherrypick requests - -As a patch release manager, you are responsible for reviewing -[cherrypicks](../cherry-picks.md) on your release branch. - -You can find candidate PRs in the [cherrypick queue dashboard](http://cherrypick.k8s.io/#/queue). -Once a cherrypick PR is created and ready for your review, it should show up in -a GitHub search such as [`is:pr is:open base:release-1.6`](https://github.com/kubernetes/kubernetes/pulls?q=is%3Apr%20is%3Aopen%20base%3Arelease-1.6). - -As an example of the kind of load to expect, there were about 150 cherrypick PRs -against the `release-1.6` branch in the 3 months between v1.6.0 and v1.7.0. - -For each cherrypick request: - -1. **Decide if it meets the criteria for a cherrypick** - - Make sure the PR author has supplied enough information to answer: - - * What bug does this fix? - (e.g. *feature X was already launched but doesn't work as intended*) - * What is the scope of users affected? - (e.g. *anyone who uses feature X*) - * How big is the impact on affected users? - (e.g. *pods using X fail to start*) - * How have you verified the fix works and is safe? - (e.g. *added new regression test*) - - Ask the PR author for details if these are missing and not obvious. - If you aren't sure what to do, escalate to the relevant SIGs. - - **Notes** - - * Version bumps (e.g. v0.5.1 -> v0.5.2) for dependencies with their own - release cycles (e.g. kube-dns, autoscaler, ingress controllers, etc.) - deserve special attention because it's hard to see what's changing. - In the past, such bumps have been a significant source of regressions in - the stable release branch. - - Check the release notes for the dependency to make sure there are no new - behaviors that could destabilize the release branch. - Ideally you should only accept version bumps whose release deltas contain - only changes that you would have approved individually, if they had been - part of the Kubernetes release cycle. - - However, this gets tricky when there are fixes you need for your branch - that are tied up with other changes. Ask the cherrypick requester for - context on the other changes and use your best judgment. - - * Historically (up through at least 1.6), patch release managers have - occasionally granted exceptions to the "no new features" rule for - cherrypicks that are confined to plugins like cloudproviders - (e.g. vSphere, Azure) and volumes (e.g. Portworx). - - However, we required that these exceptions be approved by the plugin - owners, who were asked to `/approve` through the normal `OWNERS` process - (despite it being a cherrypick PR). - -1. **Make sure it has an appropriate release note** - - [Good release notes](https://github.com/kubernetes/community/issues/484) - are particularly important for patch releases because cluster admins expect - the release branch to remain stable and need to know exactly what changed. - Take care to ensure every cherrypick that deserves a release note has one - *before you approve it* or else the change may fall through the cracks at - release cut time. - - Also make sure the release note expresses the change from a user's - perspective, not from the perspective of someone contributing to Kubernetes. - Think about what the user would experience when hitting the problem, - not the implementation details of the root cause. - - For example: - - User perspective (good) | Code perspective (bad) - ----------------------- | ---------------------- - *"Fix kubelet crash when Node detaches old volumes after restart."* | *"Call initStuff() before startLoop() to prevent race condition."* - - Ask the PR author for context if it's not clear to you what the release note - should say. - - Lastly, make sure the release note is located where the [relnotes](https://github.com/kubernetes/release/blob/master/relnotes) - script will find it: - - * If the cherrypick PR comes from a branch called `automated-cherry-pick-of-*`, - then the release notes are taken from each parent PR (possibly more than one) - and the cherrypick PR itself is ignored. - - Make sure the cherrypick PR and parent PRs have the `release-note` label. - - * Otherwise, the release note is taken from the cherrypick PR. - - Make sure the cherrypick PR has the `release-note` label. - - **Notes** - - * Almost all changes that are important enough to cherrypick are important - enough that we should inform users about them when they upgrade. - - Rare exceptions include test-only changes or follow-ups to a previous - cherrypick whose release note already explains all the intended changes. - -1. **Approve for cherrypick** - - PRs on release branches follow a different review process than those on the - `master` branch. - Patch release managers review every PR on the release branch, - but the focus is just on ensuring the above criteria are met. - The code itself was already reviewed, assuming it's copied from `master`. - - * For an *automated cherrypick* (created with `hack/cherry_pick_pull.sh`), - you can directly apply the `approved` label as long as the parent PR was - approved and merged into `master`. - If the parent PR hasn't merged yet, leave a comment explaining that you - will wait for it before approving the cherrypick. - We don't want the release branch to get out of sync if the parent PR changes. - - Then comment `/lgtm` to apply the `lgtm` label and notify the author - you've reviewed the cherrypick request. - - * For a *manual patch or cherrypick* (not a direct copy of a PR already merged - on `master`), leave a comment explaining that it needs to get - LGTM+Approval through the usual review process. - - You don't need to do anything special to fall back to this process. - The bot will suggest reviewers and approvers just like on `master`. - - Finally, apply the `cherrypick-approved` label and remove the `do-not-merge` - label to tell the bot that this PR is allowed to merge into a release - branch. - - Note that the PR will not actually merge until it meets the usual criteria - enforced by the merge bot (`lgtm` + `approved` labels, required presubmits, - etc.) and makes its way through the submit queue. - To give cherrypick PRs priority over other PRs in the submit queue, - make sure the PR is in the `vX.Y` release milestone, and that the milestone - has a due date. - -## Branch health - -Keep an eye on approved cherrypick PRs to make sure they aren't getting blocked -on presubmits that are failing across the whole branch. -Also periodically check the [testgrid](https://k8s-testgrid.appspot.com) -dashboard for your release branch to make sure the continuous jobs are healthy. - -Escalate to test owners or [sig-testing](https://github.com/kubernetes/community/tree/master/sig-testing)/[test-infra](https://github.com/kubernetes/test-infra) -as needed to diagnose failures. - -## Release timing - -The general guideline is to leave about 2 to 4 weeks between patch releases on -a given minor release branch. -The lower bound is intended to avoid upgrade churn for cluster administrators, -and to allow patches time to undergo testing on `master` and on the release -branch. -The upper bound is intended to avoid making users wait too long for fixes that -are ready to go. - -The actual timing is up to the patch release manager, who should take into -account input from cherrypick PR authors and SIGs. -For example, some bugs may be serious enough, and have a clear enough fix, -to trigger a new patch release immediately. - -You should attend the [sig-release](https://github.com/kubernetes/community/tree/master/sig-release) -meetings whenever possible to give updates on activity in your release branch -(bugs, tests, cherrypicks, etc.) and discuss release timing. - -When you have a plan for the next patch release, send an announcement -([example](https://groups.google.com/forum/#!topic/kubernetes-dev-announce/HGYsjOFtcdU)) -to [kubernetes-dev@googlegroups.com](https://groups.google.com/forum/#!forum/kubernetes-dev) -(and *BCC* [kubernetes-dev-announce@googlegroups.com](https://groups.google.com/forum/#!forum/kubernetes-dev-announce)) -several working days in advance. -You can generate a preview of the release notes with the [relnotes](https://github.com/kubernetes/release/blob/master/relnotes) -script ([example usage](https://gist.github.com/enisoc/058bf0feddf6bffd8e25aa72f9dc38d6)). - -## Release cut - -A few days before you plan to cut a patch release, put a temporary freeze on -cherrypick requests by removing the `cherrypick-approved` label from any PR that -isn't ready to merge. -Leave a comment explaining that a freeze is in effect until after the release. - -The freeze serves several purposes: - -1. It ensures a minimum time period during which problems with the accepted - patches may be discovered by people testing on `master`, or by continuous - test jobs on the release branch. - -1. It allows the continuous jobs to catch up with `HEAD` on the release branch. - Note that you cannot cut a patch release from any point other than `HEAD` - on the release branch; for example, you can't cut at the last green build. - -1. It allows slow test jobs like "serial", which has a period of many hours, - to run several times at `HEAD` to ensure they pass consistently. - -On the day before the planned release, run a mock build with `anago` to make -sure the tooling is ready. -If the mock goes well and the tests are healthy, run the real cut the next day. - -After the release cut, reapply the `cherrypick-approved` label to any PRs that -had it before the freeze, and go through the backlog of new cherrypicks. - -### Hotfix release - -A normal patch release rolls up everything that merged into the release branch -since the last patch release. -Sometimes it's necessary to cut an emergency hotfix release that contains only -one specific change relative to the last past release. -For example, we may need to fix a severe bug quickly without taking on the added -risk of allowing other changes in. - -In this case, you would create a new, three-part branch of the form -`release-X.Y.Z`, which [branches from a tag](https://github.com/kubernetes/release/blob/master/docs/branching.md#branching-from-a-tag) -called `vX.Y.Z`. -You would then use the normal cherrypick PR flow, except that you target PRs at -the `release-X.Y.Z` branch instead of `release-X.Y`. -This lets you exclude the rest of the changes that already went into -`release-X.Y` since the `vX.Y.Z` tag was cut. - -Make sure you communicate clearly in your release plan announcement that some -changes on the release branch will be excluded, and will have to wait until the -next patch release. - -### Security release - -The Product Security Team (PST) will contact you if a security release is needed -on your branch. -In contrast to a normal release, you should not make any public announcements -or push tags or release artifacts to public repositories until the PST tells you to. - -See the [Security Release Process](../security-release-process.md) doc for more -details. +The original content of this file has been migrated to https://git.k8s.io/sig-release/release-process-documentation/release-team-guides/patch-release-manager-playbook.md +This file is a placeholder to preserve links. Please remove after 3 months or the release of kubernetes 1.10, whichever comes first. diff --git a/contributors/devel/release/patch_release.md b/contributors/devel/release/patch_release.md new file mode 100644 index 00000000..1b074759 --- /dev/null +++ b/contributors/devel/release/patch_release.md @@ -0,0 +1,3 @@ +The original content of this file has been migrated to https://git.k8s.io/sig-release/ephemera/patch_release.md + +This file is a placeholder to preserve links. Please remove after 3 months or the release of kubernetes 1.10, whichever comes first. diff --git a/contributors/devel/release/scalability-validation.md b/contributors/devel/release/scalability-validation.md index 6653889f..8a943227 100644 --- a/contributors/devel/release/scalability-validation.md +++ b/contributors/devel/release/scalability-validation.md @@ -1,125 +1,3 @@ -# Scalability validation of the release +The original content of this file has been migrated to https://git.k8s.io/sig-release/ephemera/scalability-validation.md -Scalability is an integral part of k8s. Numerous issues were identified during release 1.7 while running tests on large clusters (2k-5k nodes). A majority of these manifested only in large clusters - more info under umbrella issue [#47344]. The issues ranged from large cluster setup (both cloud-provider specific and independent ones) to test-infra problems to previously uncaught regressions with performance and resource usage and so on. - -We started [supporting] 5000-node clusters from k8s 1.6 and need to make sure this scale is supported in future releases too. We did it this time by running performance and correctness tests under different configurations manually. But this testing procedure needs to be automated, concrete & well-maintained. - -## Goals - -In this document, we address the following process-related problems wrt scale testing: - -- Automate large cluster tests with a reasonable frequency -- Concretely define the testing configuration used for the release -- Make scalability validation a release prerequisite -- Clearly lay down responsibilities for scale tests - -## Non-Goals - -We do not intend to: - -- Define the set of tests that comprise scalability and correctness suite -- Define SLIs/SLOs (that’s discussed [here]) and thresholds for the tests -- Discuss particular performance issues we’re facing wrt different releases - -While these are interesting from scalability perspective, they are not testing process issues but details of the tests themselves. And can change without requiring much changes in the process. - -## Proposed design - -For each of the problems above we propose solutions and discuss some caveats. This is an open discussion and better solutions may evolve in future. - -### Automate large cluster tests - -Currently, there are 2 kinds of tests needed to validate the claim “kubernetes supports X-node clusters”. These are: - -- Correctness tests - e2e tests verifying expected system behaviours -- Performance tests - e2e tests for stress-testing the system perf ([Feature:Performance]) - -We need to run them on 5k-node clusters, but they’re: - -- Time-consuming (12-24 hrs) -- Expensive (tens of thousands of core hours per run) -- Blocking other large tests (quota limitations + only one large test project available viz. 'kubernetes-scale') - -So we don’t want to run them too frequently. On the other hand, running them too infrequently means late identification and piling up of regressions. So we choose the following middleground: -(B = release-blocking, NB = not release-blocking) - -- Performance tests on 2k-node cluster in GCE/GKE alternatingly each week (NB) - - would catch most scale-related regressions - - would catch bugs in both GCE and GKE -- Performance tests on 5k-node cluster in GCE each week (B) - - would catch scale-related bugs missed by 2k-node test (should be rare) -- Correctness tests on 2k-node cluster in GCE/GKE alternatingly each week (NB) - - correctness tests failing on just large clusters is rare, so weekly is enough - - would catch bugs in both GCE and GKE -- Correctness tests on 5k-node cluster in GCE/GKE alternatingly each week (B for GCE) - - would catch bugs left out by 2k-node (should be rare) - - would verify 5k-node clusters on GKE but without blocking the release - -Here's the proposed schedule (may be fine-tuned later based on test health / release schedule): - -| | Mon | Tue | Wed | Thu | Fri | Sat | Sun -| ---- | :----: | :----: | :----:| :----: | :----: | :----: | :----: -| Week 2k | GCE 5k perf (B) | | | GCE 5k corr (B) | | GKE 2k perf+corr (NB) | -| Week 2k+1 | GCE 5k perf (B) | | | GKE 5k corr (NB) | | GCE 2k perf+corr (NB) | - -Why this schedule? - -- 5k tests might need special attention in case of failures so they should run on weekdays -- Running 5k perf test on Mon gives the whole week for debugging in case of failures -- Running 5k corr test on Thu gives Tue-Wed to debug any setup issues found on Mon -- Running 5k corr tests alternatingly on GKE and GCE validates both setups -- 2k perf and corr tests can be run in parallel to save time -- Running 2k tests on weekend will free up a weekday for the project to run other tests - -Why GKE tests? - -Google is currently using a single project for scalability testing, on both GCE and GKE. As a result we need to schedule them together. There's a plan for CNCF becoming responsible for funding k8s testing, and GCE/GKE tests would be separated to different projects when that happens, with only GCE being funded by them. This ensures fairness across all cloud providers. - -### Concretely define test configuration - -This is a relatively minor issue but it is important that we clearly define the test configuration we use for the release. E.g. there was a confusion this time around testing k8s services, machine-type and no. of the nodes we used (we tested 4k instead of 5k due to a CIDR-setup problem). For ref - [#47344] [#47865]. To solve this, we need to document it using the below template in a file named scalability-validation-report.md placed under kubernetes/features/release-<N>. And this file should be linked from under the scalability section in the release's CHANGELOG.md. - -``` -Validated large cluster performance under the following configuration: -- Cloud-provider - [GCE / GKE / ..] -- No. of nodes - [5k (desired) / 4k / ..] -- Node size, OS, disk size/type -- Master size, OS, disk size/type -- Any non-default config used - (monitoring with stackdriver, logging with elasticsearch, etc) -- Any important test details - (services disabled in load test, pods increased in density test, etc) -- <job-name, run#> of the validating run (to know other specific details from the logs) - -Validated large cluster correctness under the following configuration: -- <similar to above> - -Misc: -<Any important scalability insights/issues/improvements in the release> -``` - -### Make scalability validation a release prerequisite - -The model we followed this time was to create an umbrella issue ([#47344]) for scalability testing and labeling it as a release-blocker. While it helped block the release, it didn’t receive enough traction from individual SIGs as scale tests were not part of the release-blocking suite. As a result, the onus for fixing issues fell almost entirely on sig-scalability due to time constraints. Thus the obvious requirement here is to make the 5k-node tests release blockers. This along with test automation ensures failures are identified quickly and get traction from relevant SIGs. - -### Clearly lay down responsibilities for scale tests - -Responsibilities that lie with sig-scalability: - -- Setting and tuning the schedule of the tests on CI -- Ensuring the project is healthy and quotas are sufficient -- Documenting the testing configuration for the release -- Identifying and fixing performance test failures (triaging/delegating as needed) - -Responsibilities lying with other SIGs/teams as applicable (could be sig-scalability too): - -- Fixing failing correctness test - Owner/SIG for the e2e test (as specified by [test_owners.csv]) -- Fixing performance regression - Owner/SIG of relevant component (as delegated by sig-scalability) -- Testing infrastructure issues - @sig-testing -- K8s-specific large-cluster setup issues - @sig-cluster-lifecycle -- GKE-specific large-cluster setup issues - @goog-gke - - -[#47344]: https://github.com/kubernetes/kubernetes/issues/47344 -[supporting]: http://blog.kubernetes.io/2017/03/scalability-updates-in-kubernetes-1.6.html -[here]: https://docs.google.com/document/d/15rD6XBtKyvXXifkRAsAVFBqEGApQxDRWM3H1bZSBsKQ -[#47865]: https://github.com/kubernetes/kubernetes/issues/47865 -[test_owners.csv]: https://github.com/kubernetes/kubernetes/blob/master/test/test_owners.csv +This file is a placeholder to preserve links. Please remove after 3 months or the release of kubernetes 1.10, whichever comes first. diff --git a/contributors/devel/release/testing.md b/contributors/devel/release/testing.md index a4df363c..2ae76112 100644 --- a/contributors/devel/release/testing.md +++ b/contributors/devel/release/testing.md @@ -1,180 +1,3 @@ -# Kubernetes test sustaining engineering +The original content of this file has been migrated to https://git.k8s.io/sig-release/ephemera/testing.md -This document describes how Kubernetes automated tests are maintained as part -of the development process. - -## Definitions - -The following definitions are for tests continuously run as part of CI. - -- *test* - - *artifact*: a row in [test grid] - - a single test run as part of a test job - - maybe either an e2e test or an integration / unit test -- *test job* - - *artifact*: a tab in [test grid] - - a collection of tests that are run together in shared environment. may: - - run in a specific environment - e.g. [gce, gke, aws], [cvm, gci] - - run under specific conditions - e.g. []upgrade, version skew, soak, serial] - - test a specific component - e.g. [federation, node] -- *test infrastructure* - - not directly shown in the test grid - - libraries and infrastructure common across tests -- *test failure* - - persistently failing test runs for a given test -- *test flake* - - periodically failling test runs for a given test - -## Ownership - -Each test must have an escalation point (email + slack). The escalation point is responsible for -keeping the test healthy. Fixes for test failures caused by areas of ownership outside the -responsibility of the escalation point should be coordinated with other teams by the -test escalation point. - -Escalation points are expected to be responsive within 24 hours, and prioritize test failure -issues over other issues. - -### test - -Each test must have an owning SIG or group that serves as the escalation point for flakes and failures. -The name of the owner should be present in the test name so that it is displayed in the test grid. - -Owners are expected to maintain a dashboard of the tests that they own and -maintain the test health. - -**Note:** e2e test owners are present in the test name - -### test job - -Each test job must have an owning SIG or group that is responsible for the health of the test job. The -owner may also serve as an escalation point for issues impacting a test only in that specific test job -(passing in other test jobs). e.g. If a test only fails on aws or only on gke test jobs, the test job -owner and test owner must identify the owner for resolving the failure. - -Owners of test jobs are expected to maintain a dashboard of the test jobs they own and -maintain the test job health. - -SIGs should update the [job config] and mark the tests that they own. - -### test infrastructure - -Issues with underlying test infrastructure (e.g. prow) should be escalated to sig/testing. - -## Monitoring project wide test health - -Dashboards for Kubernetes release blocking test are present on the [test grid]. - -The following dashboards are expected to remain healthy throughout the development cycle. - -- [release-master-blocking](https://k8s-testgrid.appspot.com/release-master-blocking) - - Tests run against the master branch -- [1.7-master-upgrade & 1.6-master-upgrade](https://k8s-testgrid.appspot.com/master-upgrade) - - Upgrade a cluster from 1.7 to the master branch and run tests - - Upgrade a cluster from 1.6 to the master branch and run tests -- [1.7-master-kubectl-skew](https://k8s-testgrid.appspot.com/master-kubectl-skew) - - Run tests skewing the master and kubectl by +1/-1 version - -## Triaging ownership for test failures - -When a test is failing, it must be quickly escalated to the correct owner. Tests that -are left to fail for days or weeks become toxic and create noise in the system health -metrics. - -The [build cop] is expected to ensure that the release blocking tests remain -perpetually healthy by monitoring the test grid and escalating failures. - -On test failures, the build cop will follow the [sig escalation](#sig-test-escalation) path. - -*Tests without a responsive owner should be assigned a new owner or disabled.* - -### test failure - -A test is failing. - -*Symptom*: A row in the test grid is consistently failing across multiple jobs - -*How to check for symptom*: Go to the [triage tool], and -search for the failing test by name. Check to see if it is failing across -multiple jobs, or just one. - -*Action*: Escalate to the owning SIG present in the test name (e.g. SIG-cli) - -### test job failure - -A test *job* is unhealthy causing multiple unrelated tests to fail. - -*Symptom*: Multiple unrelated rows in the test grid are consistently failing in a single job, -but passing in others jobs. - -*How to check for symptom*: Go to the [test grid]. Are a bunch of tests failing or just a couple? Are -those tests passing on other jobs? - -*Action*: Escalate to the owning SIG for the test job. - -### test failure (only on specifics job) - -A test is failing, but only on specific jobs. - -*Symptom*: A row in the test grid is consistently failing on a single job, but passing on other jobs. - -*How to check for symptom*: Go to the [triage tool], and -search for the failing test by name. Check to see if it is failing across -multiple jobs, or just one. - -*Action*: Escalate to the owning SIG present in the test name (e.g. SIG-cli). They -will coordinate a fix with the test job owner. - -## Triaging ownership for test flakes - -To triage ownership flakes, follow the same escalation process for failures. Flakes are considered less -urgent than persistent failures, but still expected to have a root cause investigation within 1 week. - -## Broken test workflow - -SIGs are expected to proactively monitor and maintain their tests. The build cop will also -monitor the health of the entire project, but is intended as backup who will escalate -failures to the owning SIGs. - -- File an issue for the broken test so it can be referenced and discovered - - Set the following labels: `priority/failing-test`, `sig/*` - - Assign the issue to whoever is working on it - - Mention the current build cop (TODO: publish this somewhere) -- Root cause analysis of the test failure is performed by the owner -- **Note**: The owning SIG for a test can reassign ownership of a resolution to another SIG only after getting - approval from that SIG - - This is done by the target SIG reassigning to themselves, not the test owning SIG assigning to someone else. -- Tests failure is resolved either by fixing the underlying issue or disabling the test - - Disabling a test maybe the correct thing to do in some cases - such as upgrade tests running e2e tests for alpha - features disable in newer releases. -- SIG owner monitors the test grid to make sure the tests begin to pass -- SIG owner closes the issue - -## SIG test escalation - -The build cop should monitor the overall test health of the project, and ensure ownership for any given -test does not fall through the cracks. When the build cop observer a test failure, they should first -search to see if an issue has been filed already, and if not (optionally file an issue and) escalate to the SIG -escalation point. If the escalation point is unresponsive within a day, the build cop should escalate to the SIG -googlegroup and/or slack channel, mentioning the SIG leads. If escalation through the SIG googlegroup, -slack channel and SIG leads is unsuccessful, the build cop should escalate to SIG release through the -googlegroup and slack - mentioning the SIG leads. - -The SIG escalation points should be bootstrapped from the [community sig list]. - -## SIG Recommendations - -- Figure out which e2e test jobs are release blocking for your SIG. -- Develop a process for making sure the SIGs test grid remains healthy and resolving test failures. -- Consider moving the e2e tests for the SIG into their own test jobs if this would make maintaining them easier. -- Consider developing a playbook for how to resolve test failures and how do identify whether or not another SIG owns the resolution of the issue. - -[community sig list]: https://github.com/kubernetes/community/blob/master/sig-list.md -[triage tool]: https://storage.googleapis.com/k8s-gubernator/triage/index.html -[test grid]: https://k8s-testgrid.appspot.com/ -[build cop]: https://github.com/kubernetes/community/blob/master/contributors/devel/on-call-build-cop.md -[release-master-blocking]: https://k8s-testgrid.appspot.com/release-master-blocking#Summary -[1.7-master-upgrade]: https://k8s-testgrid.appspot.com/1.7-master-upgrade#Summary -[1.6-master-upgrade]: https://k8s-testgrid.appspot.com/1.6-master-upgrade#Summary -[1.7-master-kubectl-skew]: https://k8s-testgrid.appspot.com/1.6-1.7-kubectl-skew -[job config]: https://github.com/kubernetes/test-infra/blob/master/jobs/config.json +This file is a placeholder to preserve links. Please remove after 3 months or the release of kubernetes 1.10, whichever comes first. diff --git a/contributors/devel/running-locally.md b/contributors/devel/running-locally.md index 3f693840..077da2af 100644 --- a/contributors/devel/running-locally.md +++ b/contributors/devel/running-locally.md @@ -25,7 +25,7 @@ Getting started locally #### Linux -Not running Linux? Consider running [Minikube](http://kubernetes.io/docs/getting-started-guides/minikube/), or on a cloud provider like [Google Compute Engine](../getting-started-guides/gce.md). +Not running Linux? Consider running [Minikube](http://kubernetes.io/docs/getting-started-guides/minikube/), or on a cloud provider like [Google Compute Engine](https://kubernetes.io/docs/getting-started-guides/gce/). #### Docker @@ -170,7 +170,3 @@ KUBE_DNS_DOMAIN="cluster.local" ``` To know more on DNS service you can check out the [docs](http://kubernetes.io/docs/admin/dns/). - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/scalability-good-practices.md b/contributors/devel/scalability-good-practices.md index 23ed7e2a..2b941a75 100644 --- a/contributors/devel/scalability-good-practices.md +++ b/contributors/devel/scalability-good-practices.md @@ -108,7 +108,7 @@ This looks fine-ish if you don't know that LIST are very expensive calls. Object `Informer` is our library that provides a read interface of the store - it's a read-only cache that provides you a local copy of the store that will contain only object that you're interested in (matching given selector). From it you can GET, LIST, or do whatever read operations you want. `Informer` also allows you to register functions that will be called when an object is created, modified or deleted, which is what most people want. -The magic behind `Informers` is that they are populated by the WATCH, so they don't stress API server too much. Code for Informer is [here](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/cache/shared_informer.go). +The magic behind `Informers` is that they are populated by the WATCH, so they don't stress API server too much. Code for Informer is [here](https://git.k8s.io/kubernetes/staging/src/k8s.io/client-go/tools/cache/shared_informer.go). In general: use `Informers` - if we were able to rewrite most vanilla controllers to them, you'll be able to do it as well. If you don't you may dramatically increase CPU requirements of the API server which will starve it and make it too slow to meet our SLOs. @@ -160,7 +160,7 @@ When designing new features/thinking about refactoring you should: - when adding complex logic make sure that you don't add it on a critical path of any basic workflow ## Closing remarks -We know that thinking about performance impact of changes that all of us write is hard, this is exactly why we want you to help us cater for it, by always having problems that we mentioned here in the back of your mind. In return we'll answer all your question and doubts about possible impact of your changes if you post them either to #sig-scale Slack channel, or cc @kubernetes/sig-scalability-pr-reviews in your PR/proposal. +We know that thinking about performance impact of changes that all of us write is hard, this is exactly why we want you to help us cater for it, by always having problems that we mentioned here in the back of your mind. In return we'll answer all your question and doubts about possible impact of your changes if you post them either to #sig-scalability Slack channel, or cc @kubernetes/sig-scalability-pr-reviews in your PR/proposal. * * * diff --git a/contributors/devel/scheduler.md b/contributors/devel/scheduler.md index 47985c98..a7a81a10 100755..100644 --- a/contributors/devel/scheduler.md +++ b/contributors/devel/scheduler.md @@ -88,7 +88,3 @@ However, the choice of policies can be overridden by passing the command-line fl config file. (Note that the config file format is versioned; the API is defined in [plugin/pkg/scheduler/api](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/api/)). Thus to add a new scheduling policy, you should modify [plugin/pkg/scheduler/algorithm/predicates/predicates.go](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithm/predicates/predicates.go) or add to the directory [plugin/pkg/scheduler/algorithm/priorities](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithm/priorities/), and either register the policy in `defaultPredicates()` or `defaultPriorities()`, or use a policy config file. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/scheduler_algorithm.md b/contributors/devel/scheduler_algorithm.md index a115a982..1d3eb418 100755..100644 --- a/contributors/devel/scheduler_algorithm.md +++ b/contributors/devel/scheduler_algorithm.md @@ -8,7 +8,7 @@ The purpose of filtering the nodes is to filter out the nodes that do not meet c - `NoDiskConflict`: Evaluate if a pod can fit due to the volumes it requests, and those that are already mounted. Currently supported volumes are: AWS EBS, GCE PD, ISCSI and Ceph RBD. Only Persistent Volume Claims for those supported types are checked. Persistent Volumes added directly to pods are not evaluated and are not constrained by this policy. - `NoVolumeZoneConflict`: Evaluate if the volumes a pod requests are available on the node, given the Zone restrictions. -- `PodFitsResources`: Check if the free resource (CPU and Memory) meets the requirement of the Pod. The free resource is measured by the capacity minus the sum of requests of all Pods on the node. To learn more about the resource QoS in Kubernetes, please check [QoS proposal](../design-proposals/resource-qos.md). +- `PodFitsResources`: Check if the free resource (CPU and Memory) meets the requirement of the Pod. The free resource is measured by the capacity minus the sum of requests of all Pods on the node. To learn more about the resource QoS in Kubernetes, please check [QoS proposal](../design-proposals/node/resource-qos.md). - `PodFitsHostPorts`: Check if any HostPort required by the Pod is already occupied on the node. - `HostName`: Filter out all nodes except the one specified in the PodSpec's NodeName field. - `MatchNodeSelector`: Check if the labels of the node match the labels specified in the Pod's `nodeSelector` field and, as of Kubernetes v1.2, also match the `scheduler.alpha.kubernetes.io/affinity` pod annotation if present. See [here](https://kubernetes.io/docs/user-guide/node-selection/) for more details on both. @@ -38,7 +38,3 @@ Currently, Kubernetes scheduler provides some practical priority functions, incl The details of the above priority functions can be found in [plugin/pkg/scheduler/algorithm/priorities](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithm/priorities/). Kubernetes uses some, but not all, of these priority functions by default. You can see which ones are used by default in [plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go). Similar as predicates, you can combine the above priority functions and assign weight factors (positive number) to them as you want (check [scheduler.md](scheduler.md) for how to customize). - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/security-release-process.md b/contributors/devel/security-release-process.md index b4b22100..e0b55f68 100644 --- a/contributors/devel/security-release-process.md +++ b/contributors/devel/security-release-process.md @@ -1,103 +1,3 @@ -# Security Release Process +The original content of this file has been migrated to https://git.k8s.io/sig-release/security-release-process-documentation/security-release-process.md -Kubernetes is a large growing community of volunteers, users, and vendors. The Kubernetes community has adopted this security disclosures and response policy to ensure we responsibly handle critical issues. - -## Product Security Team (PST) - -Security vulnerabilities should be handled quickly and sometimes privately. The primary goal of this process is to reduce the total time users are vulnerable to publicly known exploits. - -The Product Security Team (PST) is responsible for organizing the entire response including internal communication and external disclosure but will need help from relevant developers and release managers to successfully run this process. - -The initial Product Security Team will consist of four volunteers subscribed to the private [Kubernetes Security](https://groups.google.com/forum/#!forum/kubernetes-security) list. These are the people who have been involved in the initial discussion and volunteered: - -- Brandon Philips `<brandon.philips@coreos.com>` [4096R/154343260542DF34] -- Jess Frazelle `<jessfraz@google.com>` [4096R/0x18F3685C0022BFF3] -- CJ Cullen `<cjcullen@google.com>` -- Tim Allclair `<tallclair@google.com>` [4096R/0x5E6F2E2DA760AF51] -- Jordan Liggitt `<jliggitt@redhat.com>` [4096R/0x39928704103C7229] - -**Known issues** - -- We haven't specified a way to cycle the Product Security Team; but we need this process deployed quickly as our current process isn't working. I (@philips) will put a deadline of March 1st 2017 to sort that. - -## Release Manager Role - -Also included on the private [Kubernetes Security](https://groups.google.com/forum/#!forum/kubernetes-security) list are all [Release Managers](https://github.com/kubernetes/community/wiki). - -It is the responsibility of the PST to add and remove Release Managers as Kubernetes minor releases created and deprecated. - -## Disclosures - -### Private Disclosure Processes - -The Kubernetes Community asks that all suspected vulnerabilities be privately and responsibly disclosed via the Private Disclosure process available at [https://kubernetes.io/security](https://kubernetes.io/security). - -### Public Disclosure Processes - -If you know of a publicly disclosed security vulnerability please IMMEDIATELY email [kubernetes-security@googlegroups.com](mailto:kubernetes-security@googlegroups.com) to inform the Product Security Team (PST) about the vulnerability so they may start the patch, release, and communication process. - -If possible the PST will ask the person making the public report if the issue can be handled via a private disclosure process. If the reporter denies the PST will move swiftly with the fix and release process. In extreme cases you can ask GitHub to delete the issue but this generally isn't necessary and is unlikely to make a public disclosure less damaging. - -## Patch, Release, and Public Communication - -For each vulnerability a member of the PST will volunteer to lead coordination with the Fix Team, Release Managers and is responsible for sending disclosure emails to the rest of the community. This lead will be referred to as the Fix Lead. - -The role of Fix Lead should rotate round-robin across the PST. - -All of the timelines below are suggestions and assume a Private Disclosure. The Fix Lead drives the schedule using their best judgment based on severity, development time, and release manager feedback. If the Fix Lead is dealing with a Public Disclosure all timelines become ASAP. - -### Fix Team Organization - -These steps should be completed within the first 24 hours of Disclosure. - -- The Fix Lead will work quickly to identify relevant engineers from the affected projects and packages and CC those engineers into the disclosure thread. This selected developers are the Fix Team. A best guess is to invite all assignees in the OWNERS file from the affected packages. -- The Fix Lead will get the Fix Team access to private security repos to develop the fix. - -### Fix Development Process - -These steps should be completed within the 1-7 days of Disclosure. - -- The Fix Lead and the Fix Team will create a [CVSS](https://www.first.org/cvss/specification-document) using the [CVSS Calculator](https://www.first.org/cvss/calculator/3.0). The Fix Lead makes the final call on the calculated CVSS; it is better to move quickly than make the CVSS perfect. -- The Fix Team will notify the Fix Lead that work on the fix branch is complete once there are LGTMs on all commits in the private repo from one or more relevant assignees in the relevant OWNERS file. - -If the CVSS score is under 4.0 ([a low severity score](https://www.first.org/cvss/specification-document#i5)) the Fix Team can decide to slow the release process down in the face of holidays, developer bandwidth, etc. These decisions must be discussed on the kubernetes-security mailing list. - -### Fix Disclosure Process - -With the Fix Development underway the Fix Lead needs to come up with an overall communication plan for the wider community. This Disclosure process should begin after the Fix Team has developed a Fix or mitigation so that a realistic timeline can be communicated to users. - -**Disclosure of Forthcoming Fix to Users** (Completed within 1-7 days of Disclosure) - -- The Fix Lead will email [kubernetes-announce@googlegroups.com](https://groups.google.com/forum/#!forum/kubernetes-announce) and [kubernetes-security-announce@googlegroups.com](https://groups.google.com/forum/#!forum/kubernetes-security-announce) informing users that a security vulnerability has been disclosed and that a fix will be made available at YYYY-MM-DD HH:MM UTC in the future via this list. This time is the Release Date. -- The Fix Lead will include any mitigating steps users can take until a fix is available. - -The communication to users should be actionable. They should know when to block time to apply patches, understand exact mitigation steps, etc. - -**Optional Fix Disclosure to Private Distributors List** (Completed within 1-14 days of Disclosure): - -- The Fix Lead will make a determination with the help of the Fix Team if an issue is critical enough to require early disclosure to distributors. Generally this Private Distributor Disclosure process should be reserved for remotely exploitable or privilege escalation issues. Otherwise, this process can be skipped. -- The Fix Lead will email the patches to kubernetes-distributors-announce@googlegroups.com so distributors can prepare builds to be available to users on the day of the issue's announcement. Distributors can ask to be added to this list by emailing kubernetes-security@googlegroups.com and it is up to the Product Security Team's discretion to manage the list. - - TODO: Figure out process for getting folks onto this list. -- **What if a vendor breaks embargo?** The PST will assess the damage. The Fix Lead will make the call to release earlier or continue with the plan. When in doubt push forward and go public ASAP. - -**Fix Release Day** (Completed within 1-21 days of Disclosure) - -- The Release Managers will ensure all the binaries are built, publicly available, and functional before the Release Date. - - TODO: this will require a private security build process. -- The Release Managers will create a new patch release branch from the latest patch release tag + the fix from the security branch. As a practical example if v1.5.3 is the latest patch release in kubernetes.git a new branch will be created called v1.5.4 which includes only patches required to fix the issue. -- The Fix Lead will cherry-pick the patches onto the master branch and all relevant release branches. The Fix Team will LGTM and merge. -- The Release Managers will merge these PRs as quickly as possible. Changes shouldn't be made to the commits even for a typo in the CHANGELOG as this will change the git sha of the already built and commits leading to confusion and potentially conflicts as the fix is cherry-picked around branches. -- The Fix Lead will request a CVE from [DWF](https://github.com/distributedweaknessfiling/DWF-Documentation) and include the CVSS and release details. -- The Fix Lead will email kubernetes-{dev,users,announce,security-announce}@googlegroups.com now that everything is public announcing the new releases, the CVE number, the location of the binaries, and the relevant merged PRs to get wide distribution and user action. As much as possible this email should be actionable and include links how to apply the fix to users environments; this can include links to external distributor documentation. -- The Fix Lead will remove the Fix Team from the private security repo. - -### Retrospective - -These steps should be completed 1-3 days after the Release Date. The retrospective process [should be blameless](https://landing.google.com/sre/book/chapters/postmortem-culture.html). - -- The Fix Lead will send a retrospective of the process to kubernetes-dev@googlegroups.com including details on everyone involved, the timeline of the process, links to relevant PRs that introduced the issue, if relevant, and any critiques of the response and release process. -- The Release Managers and Fix Team are also encouraged to send their own feedback on the process to kubernetes-dev@googlegroups.com. Honest critique is the only way we are going to get good at this as a community. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> +This file is a placeholder to preserve links. Please remove after 3 months or the release of kubernetes 1.10, whichever comes first. diff --git a/contributors/devel/staging.md b/contributors/devel/staging.md new file mode 100644 index 00000000..8776cf02 --- /dev/null +++ b/contributors/devel/staging.md @@ -0,0 +1,34 @@ +# Staging Directory and Publishing + +The [staging/ directory](https://git.k8s.io/kubernetes/staging) of Kubernetes contains a number of pseudo repositories ("staging repos"). They are symlinked into Kubernetes' [vendor/ directory](https://git.k8s.io/kubernetes/vendor/k8s.io) for Golang to pick them up. + +We publish the staging repos using the [publishing bot](https://git.k8s.io/publishing-bot). It uses `git filter-branch` essentially to [cut the staging directories into separate git trees](https://de.slideshare.net/sttts/cutting-the-kubernetes-monorepo-in-pieces-never-learnt-more-about-git) and pushing the new commits to the corresponding real repositories in the [kubernetes organization on Github](https://github.com/kubernetes). + +The list of staging repositories and their published branches are listed in [publisher.go inside of the bot](https://git.k8s.io/publishing-bot/cmd/publishing-bot/publisher.go). Though it is planned to move this out into the k8s.io/kubernetes repository. + +At the time of this writing, this includes the branches + +- master, +- release-1.8 / release-5.0, +- and release-1.9 / release-6.0 + +of the follwing staging repos in the k8s.io org: + +- api +- apiextensions-apiserver +- apimachinery +- apiserver +- client-go +- code-generator +- kube-aggregator +- metrics +- sample-apiserver +- sample-controller + +Kubernetes tags (e.g., v1.9.1-beta1) are also applied automatically to the published repositories, prefixed with kubernetes- (e.g., kubernetes-1.9.1-beta1). The client-go semver tags (on client-go only!) including release-notes are still done manually. + +The semver tags are still the (well tested) official releases. The kubernetes-1.x.y tags have limited test coverage (we have some automatic tests in place in the bot), but can be used by early adopters of client-go and the other libraries. Moreover, they help to vendor the correct version of k8s.io/api and k8s.io/apimachinery. + +If further repos under staging are need, adding them to the bot is easy. Contact one of the [owners of the bot](https://git.k8s.io/publishing-bot/OWNERS). + +Currently, the bot is hosted on the CI cluster of Redhat's OpenShift (ready to be moved out to a public CNCF cluster if we have that in the future). diff --git a/contributors/devel/strategic-merge-patch.md b/contributors/devel/strategic-merge-patch.md index b79c46ba..4f45ef8e 100644 --- a/contributors/devel/strategic-merge-patch.md +++ b/contributors/devel/strategic-merge-patch.md @@ -35,9 +35,9 @@ container lists to merge together based on the `name` field. To solve this problem, Strategic Merge Patch uses the go struct tag of the API objects to determine what lists should be merged and which ones should not. -Currently the metadata is available as struct tags on the API objects -themselves, but will become available to clients as Swagger annotations in the -future. In the above example, the `patchStrategy` metadata for the `containers` +The metadata is available as struct tags on the API objects +themselves and also available to clients as [OpenAPI annotations](https://github.com/kubernetes/kubernetes/blob/master/api/openapi-spec/README.md#x-kubernetes-patch-strategy-and-x-kubernetes-patch-merge-key). +In the above example, the `patchStrategy` metadata for the `containers` field would be `merge` and the `patchMergeKey` would be `name`. @@ -216,7 +216,7 @@ item that has duplicates will delete all matching items. `setElementOrder` directive provides a way to specify the order of a list. The relative order specified in this directive will be retained. -Please refer to [proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/preserve-order-in-strategic-merge-patch.md) for more information. +Please refer to [proposal](/contributors/design-proposals/cli/preserve-order-in-strategic-merge-patch.md) for more information. ### Syntax @@ -295,7 +295,7 @@ containers: `retainKeys` directive provides a mechanism for union types to clear mutual exclusive fields. When this directive is present in the patch, all the fields not in this directive will be cleared. -Please refer to [proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/add-new-patchStrategy-to-clear-fields-not-present-in-patch.md) for more information. +Please refer to [proposal](/contributors/design-proposals/api-machinery/add-new-patchStrategy-to-clear-fields-not-present-in-patch.md) for more information. ### Syntax diff --git a/contributors/devel/testing.md b/contributors/devel/testing.md index 7306d2d4..6a7c7be6 100644 --- a/contributors/devel/testing.md +++ b/contributors/devel/testing.md @@ -1,9 +1,6 @@ # Testing guide -Updated: 5/21/2016 - **Table of Contents** -<!-- BEGIN MUNGE: GENERATED_TOC --> - [Testing guide](#testing-guide) - [Unit tests](#unit-tests) @@ -21,7 +18,6 @@ Updated: 5/21/2016 - [Run a specific integration test](#run-a-specific-integration-test) - [End-to-End tests](#end-to-end-tests) -<!-- END MUNGE: GENERATED_TOC --> This assumes you already read the [development guide](development.md) to install go, godeps, and configure your git client. All command examples are @@ -163,7 +159,7 @@ See `go help test` and `go help testflag` for additional info. is [table driven testing](https://github.com/golang/go/wiki/TableDrivenTests) - Example: [TestNamespaceAuthorization](https://git.k8s.io/kubernetes/test/integration/auth/auth_test.go) * Each test should create its own master, httpserver and config. - - Example: [TestPodUpdateActiveDeadlineSeconds](https://github.com/kubernetes/kubernetes/blob/master/test/integration/pods/pods_test.go) + - Example: [TestPodUpdateActiveDeadlineSeconds](https://git.k8s.io/kubernetes/test/integration/pods/pods_test.go) * See [coding conventions](coding-conventions.md). ### Install etcd dependency @@ -205,7 +201,7 @@ make test-integration # Run all integration tests. ``` This script runs the golang tests in package -[`test/integration`](https://github.com/kubernetes/kubernetes/tree/master/test/integration). +[`test/integration`](https://git.k8s.io/kubernetes/test/integration). ### Run a specific integration test @@ -223,7 +219,3 @@ version and the watch cache test is skipped. ## End-to-End tests Please refer to [End-to-End Testing in Kubernetes](e2e-tests.md). - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/update-release-docs.md b/contributors/devel/update-release-docs.md index 1e0988db..68c6707c 100644 --- a/contributors/devel/update-release-docs.md +++ b/contributors/devel/update-release-docs.md @@ -1,7 +1,5 @@ # Table of Contents -<!-- BEGIN MUNGE: GENERATED_TOC --> - - [Table of Contents](#table-of-contents) - [Overview](#overview) - [Adding a new docs collection for a release](#adding-a-new-docs-collection-for-a-release) @@ -10,7 +8,6 @@ - [Updating docs in release branch](#updating-docs-in-release-branch) - [Updating docs in gh-pages branch](#updating-docs-in-gh-pages-branch) -<!-- END MUNGE: GENERATED_TOC --> # Overview @@ -109,7 +106,3 @@ docs go away. If the change added or deleted a doc, then update the corresponding `_includes/nav_vX.Y.html` file as well. - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/updating-docs-for-feature-changes.md b/contributors/devel/updating-docs-for-feature-changes.md index 309b809d..1171d397 100644 --- a/contributors/devel/updating-docs-for-feature-changes.md +++ b/contributors/devel/updating-docs-for-feature-changes.md @@ -70,7 +70,3 @@ Anyone making user facing changes to kubernetes. This is especially important f * Include screen shots or pictures in documents for GUIs * *TODO once we have a standard widget set we are happy with* - include diagrams to help describe complex ideas (not required yet) - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/developer-guides/vagrant.md b/contributors/devel/vagrant.md index b53b0002..98d150ac 100755..100644 --- a/contributors/devel/developer-guides/vagrant.md +++ b/contributors/devel/vagrant.md @@ -227,7 +227,7 @@ my-nginx 3 3 3 3 1m We did not start any Services, hence there are none listed. But we see three replicas displayed properly. Check the -[guestbook](https://github.com/kubernetes/kubernetes/tree/%7B%7Bpage.githubbranch%7D%7D/examples/guestbook) +[guestbook](https://git.k8s.io/examples/guestbook) application to learn how to create a Service. You can already play with scaling the replicas with: @@ -427,6 +427,3 @@ provider, which uses nfs by default. For example: export KUBERNETES_VAGRANT_USE_NFS=true ``` -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/welcome-to-kubernetes-new-developer-guide.md b/contributors/devel/welcome-to-kubernetes-new-developer-guide.md index 8cddd2fc..2a87c336 100644 --- a/contributors/devel/welcome-to-kubernetes-new-developer-guide.md +++ b/contributors/devel/welcome-to-kubernetes-new-developer-guide.md @@ -1,94 +1,3 @@ +This page has moved to: -Welcome to Kubernetes! (New Developer Guide) -============================================ - -_This document assumes that you know what Kubernetes does. If you don't, check -out [the Kubernetes keynote](https://www.youtube.com/watch?v=8SvQqZNP6uo) and -try the demo at [https://k8s.io/](https://k8s.io/)._ - -Introduction ------------- - -Have you ever wanted to contribute to the coolest cloud technology? This -document will help you understand the organization of the Kubernetes project and -direct you to the best places to get started. By the end of this doc, you'll be -able to pick up issues, write code to fix them, and get your work reviewed and -merged. - -If you have questions about the development process, feel free to jump into our -[Slack Channel](http://slack.k8s.io/) or join our [mailing -list](https://groups.google.com/forum/#!forum/kubernetes-dev). - -Special Interest Groups ------------------------ - -Kubernetes developers work in teams called Special Interest Groups (SIGs). At -the time of this writing there are [31 SIGs](../../sig-list.md). - -The developers within each SIG have autonomy and ownership over that SIG's part -of Kubernetes. SIGs organize themselves by meeting regularly and submitting -markdown design documents to the -[kubernetes/community](https://github.com/kubernetes/community) repository. You -can see an example of the design document process at -[kubernetes/community#645](https://github.com/kubernetes/community/pull/645). -Like everything else in Kubernetes, a SIG is an open, community, effort. Anybody -is welcome to jump into a SIG and begin fixing issues, critiquing design -proposals and reviewing code. - -Most people who visit the Kubernetes repository for the first time are -bewildered by the thousands of [open -issues](https://github.com/kubernetes/kubernetes/issues) in our main repository. -But now that you know about SIGs, it's easy to filter by labels to see what's -going on in a particular SIG. For more information about our issue system, check -out -[issues.md](https://github.com/kubernetes/community/blob/master/contributors/devel/issues.md). - -Downloading, Building, and Testing Kubernetes ---------------------------------------------- - -This guide is non-technical, so it does not cover the technical details of -working Kubernetes. We have plenty of documentation available under -[github.com/kubernetes/community/contributors/devel](https://github.com/kubernetes/community/tree/master/contributors/devel). -Check out -[development.md](https://github.com/kubernetes/community/blob/master/contributors/devel/development.md) -for more details. - -Pull-Request Process --------------------- - -The pull-request process is documented in [pull-requests.md](pull-requests.md). -As described in that document, you must sign the [CLA](../../CLA.md) before -Kubernetes can accept your contribution. - -There is an entire SIG -([sig-contributor-experience](../../sig-contributor-experience/README.md)) -devoted to improving your experience as a contributor. Contributing to -Kubernetes should be easy. If you find a rough edge, let us know! File an issue -in the appropriate repository if you know it, send a message on [our Slack -channel](https://kubernetes.slack.com/messages/C1TU9EB9S/details) or to the -[owners](https://github.com/kubernetes/community/blob/master/contributors/devel/owners.md) -of contributor-experience. Better yet, help us fix it by joining the SIG; just -show up to one of the [bi-weekly -meetings](https://docs.google.com/document/d/1qf-02B7EOrItQgwXFxgqZ5qjW0mtfu5qkYIF1Hl4ZLI/edit). - -The Release Process and Code Freeze ------------------------------------ - -Every so often @k8s-merge-robot will refuse to merge your PR, saying something -about release milestones. This happens when we are in a code freeze for a -release. In order to ensure Kubernetes is stable, we stop merging everything -that's not a bugfix, then focus on making all the release tests pass. This code -freeze usually lasts two weeks and happens once per quarter. See the [1.8 release -schedule](https://github.com/kubernetes/features/blob/master/release-1.8/release-1.8.md) -for more information. - -If you're new to Kubernetes, you won't have to worry about this too much. After -you've contributed for a few months, you will be added as a [community -member](https://github.com/kubernetes/community/blob/master/community-membership.md) -and take ownership of some of the tests. At this point, you'll work with members -of your SIG to review PRs coming into your area and track down issues that occur -in tests. - -Thanks for reading! - - +https://git.k8s.io/community/contributors/guide/ diff --git a/contributors/devel/writing-a-getting-started-guide.md b/contributors/devel/writing-a-getting-started-guide.md index b1d65d60..a879e27d 100644 --- a/contributors/devel/writing-a-getting-started-guide.md +++ b/contributors/devel/writing-a-getting-started-guide.md @@ -94,8 +94,3 @@ These guidelines say *what* to do. See the Rationale section for *why*. if you use another Configuration Management tool -- you just have to do some manual steps during testing and deployment. - - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/devel/writing-good-e2e-tests.md b/contributors/devel/writing-good-e2e-tests.md index 72bdbb72..0658aad2 100644 --- a/contributors/devel/writing-good-e2e-tests.md +++ b/contributors/devel/writing-good-e2e-tests.md @@ -146,7 +146,7 @@ right thing. Here are a few pointers: -+ [E2e Framework](https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/framework.go): ++ [E2e Framework](https://git.k8s.io/kubernetes/test/e2e/framework/framework.go): Familiarise yourself with this test framework and how to use it. Amongst others, it automatically creates uniquely named namespaces within which your tests can run to avoid name clashes, and reliably @@ -160,7 +160,7 @@ Here are a few pointers: should always use this framework. Trying other home-grown approaches to avoiding name clashes and resource leaks has proven to be a very bad idea. -+ [E2e utils library](https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/util.go): ++ [E2e utils library](https://git.k8s.io/kubernetes/test/e2e/framework/util.go): This handy library provides tons of reusable code for a host of commonly needed test functionality, including waiting for resources to enter specified states, safely and consistently retrying failed @@ -178,9 +178,9 @@ Here are a few pointers: + **Follow the examples of stable, well-written tests:** Some of our existing end-to-end tests are better written and more reliable than others. A few examples of well-written tests include: - [Replication Controllers](https://github.com/kubernetes/kubernetes/blob/master/test/e2e/rc.go), - [Services](https://github.com/kubernetes/kubernetes/blob/master/test/e2e/service.go), - [Reboot](https://github.com/kubernetes/kubernetes/blob/master/test/e2e/reboot.go). + [Replication Controllers](https://git.k8s.io/kubernetes/test/e2e/apps/rc.go), + [Services](https://git.k8s.io/kubernetes/test/e2e/network/service.go), + [Reboot](https://git.k8s.io/kubernetes/test/e2e/lifecycle/reboot.go). + [Ginkgo Test Framework](https://github.com/onsi/ginkgo): This is the test library and runner upon which our e2e tests are built. Before you write or refactor a test, read the docs and make sure that you @@ -229,7 +229,3 @@ Unreachable nodes are evacuated and then repopulated upon rejoining [Disruptive] Note that opening issues for specific better tooling is welcome, and code implementing that tooling is even more welcome :-). - -<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> -[]() -<!-- END MUNGE: GENERATED_ANALYTICS --> diff --git a/contributors/guide/README.md b/contributors/guide/README.md new file mode 100644 index 00000000..860647c7 --- /dev/null +++ b/contributors/guide/README.md @@ -0,0 +1,247 @@ +**OWNER:** + +sig-contributor-experience + +## Disclaimer +Hello! This is the starting point for our brand new contributor guide, currently underway as per [issue#6102](https://github.com/kubernetes/website/issues/6102) and in need of help. Please be patient, or fix a section below that needs improvement, and submit a pull request! + +Many of the links below should lead to relevant documents scattered across the community repository. Often, the linked instructions need to be updated or cleaned up. + +* If you do so, please move the relevant file from its previous location to the community/contributors/guide folder, and delete its previous location. +* Our goal is that all contributor guide specific files live in this folder. + +Please find _Improvements needed_ sections below and help us out. + +For example: + +_Improvements needed_ +* kubernetes/community/CONTRIBUTING.md -> Needs a rewrite + +* kubernetes/community/README.md -> Needs a rewrite + +* Individual SIG contributing documents -> add a link to this guide + +# Welcome + +Welcome to Kubernetes! This document is the single source of truth for how to contribute to the code base. Please leave comments / suggestions if you find something is missing or incorrect. + +- [Before you get started](#before-you-get-started) + - [Sign the CLA](#sign-the-cla) + - [Setting up your development + environment](#setting-up-your-development-environment) + - [Community Expectations](#community-expectations) + - [Code of Conduct](#code-of-conduct) + - [Thanks](#thanks) +- [Your First Contribution](#your-first-contribution) + - [Find something to work on](#find-something-to-work-on) + - [Find a good first topic](#find-a-good-first-topic) + - [Learn about SIGs](#learn-about-sigs) + - [File an Issue](#file-an-issue) +- [Contributing](#contributing) + - [Communication](#communication) + - [GitHub workflow](#github-workflow) + - [Open a Pull Request](#open-a-pull-request) + - [Code Review](#code-review) + - [Testing](#testing) + - [Security](#security) + - [Documentation](#documentation) + - [Issues Management or Triage](#issues-management-or-triage) +- [Community](#community) + - [Events](#events) + - [Meetups](#meetups) + - [KubeCon](#kubecon) + - [Mentorship](#mentorship) + +# Before you get started + +## Sign the CLA + +Before you can contribute, you will need to sign the [Contributor License Agreement](/CLA.md). + +## Setting up your development environment + +If you haven’t set up your environment, please find resources [here](/contributors/devel). These resources are not well organized currently; please have patience as we are working on it. + +_Improvements needed_ +* A new developer guide will be created and linked to in this section. + + * RyanJ from Red Hat is working on this + +## Community Expectations + +Kubernetes is a community project. Consequently, it is wholly dependent on its community to provide a productive, friendly and collaborative environment. + +The first and foremost goal of the Kubernetes community to develop orchestration technology that radically simplifies the process of creating reliable distributed systems. However a second, equally important goal is the creation of a community that fosters easy, agile development of such orchestration systems. + +We therefore describe the expectations for members of the Kubernetes community. This document is intended to be a living one that evolves as the community evolves via the same pull request and code review process that shapes the rest of the project. It currently covers the expectations of conduct that govern all members of the community as well as the expectations around code review that govern all active contributors to Kubernetes. + +### Code of Conduct + +Please make sure to read and observe our [Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md) + +### Thanks + +Many thanks in advance to everyone who contributes their time and effort to making Kubernetes both a successful system as well as a successful community. The strength of our software shines in the strengths of each individual community member. Thanks! + +# Your First Contribution + +Have you ever wanted to contribute to the coolest cloud technology? We will help you understand the organization of the Kubernetes project and direct you to the best places to get started. You'll be able to pick up issues, write code to fix them, and get your work reviewed and merged. + +Please be aware that due to the large number of issues our triage team deals with, we cannot offer technical support in GitHub issues. If you have questions about the development process, feel free to jump into our [Slack Channel](http://slack.k8s.io/) or join our [mailing list](https://groups.google.com/forum/#!forum/kubernetes-dev). You can also ask questions on [ServerFault](https://serverfault.com/questions/tagged/kubernetes) or [Stack Overflow](https://stackoverflow.com/questions/tagged/kubernetes). The Kubernetes team scans Stack Overflow on a regular basis, and will try to ensure your questions don't go unanswered. + +## Find something to work on + +Help is always welcome! For example, documentation (like the text you are reading now) can always use improvement. There's always code that can be clarified and variables or functions that can be renamed or commented. There's always a need for more test coverage. +You get the idea - if you ever see something you think should be fixed, you should own it. Here is how you get started. + +### Find a good first topic + +There are multiple repositories within the Kubernetes community and a full list of repositories can be found [here](https://github.com/kubernetes/). +Each repository in the Kubernetes organization has beginner-friendly issues that provide a good first issue. For example, [kubernetes/kubernetes](https://git.k8s.io/kubernetes) has [help wanted issues](https://go.k8s.io/help-wanted) that should not need deep knowledge of the system. +Another good strategy is to find a documentation improvement, such as a missing/broken link, which will give you exposure to the code submission/review process without the added complication of technical depth. Please see [Contributing](#contributing) below for the workflow. + +### Learn about SIGs + +#### Sig structure + +You may have noticed that some repositories in the Kubernetes Organization are owned by Special Interest Groups, or SIGs. We organize the Kubernetes community into SIGs in order to improve our workflow and more easily manage what is a very large community project. The developers within each SIG have autonomy and ownership over that SIG's part of Kubernetes. +SIGs also have their own CONTRIBUTING.md files, which may contain extra information or guidelines in addition to these general ones. These are located in the SIG specific community documentation directories, for example: sig-docs' is in the kubernetes/community repo's [/sig-docs/CONTRIBUTING.md](/sig-docs/CONTRIBUTING.md) file and similarly for other SIGs. + +Like everything else in Kubernetes, a SIG is an open, community, effort. Anybody is welcome to jump into a SIG and begin fixing issues, critiquing design proposals and reviewing code. SIGs have regular [video meetings](https://kubernetes.io/community/) which everyone is welcome to. Each SIG has a kubernetes slack channel that you can join as well. + +There is an entire SIG ([sig-contributor-experience](/sig-contributor-experience/README.md)) devoted to improving your experience as a contributor. +Contributing to Kubernetes should be easy. If you find a rough edge, let us know! Better yet, help us fix it by joining the SIG; just +show up to one of the [bi-weekly meetings](https://docs.google.com/document/d/1qf-02B7EOrItQgwXFxgqZ5qjW0mtfu5qkYIF1Hl4ZLI/edit). + +#### Find a SIG that is related to your contribution + +Finding the appropriate SIG for your contribution will help you ask questions in the correct place and give your contribution higher visibility and a faster community response. + +For Pull Requests, the automatically assigned reviewer will add a SIG label if you haven't done so. See [Open A Pull Request](#open-a-pull-request) below. + +For Issues we are still working on a more automated workflow. Since SIGs do not directly map onto Kubernetes subrepositories, it may be difficult to find which SIG your contribution belongs in. Here is the [list of SIGs](/sig-list.md). Determine which is most likely related to your contribution. + +*Example:* if you are filing a cni issue, you should choose SIG-networking. + +Follow the link in the SIG name column to reach each SIGs README. Most SIGs will have a set of GitHub Teams with tags that can be mentioned in a comment on issues and pull requests for higher visibility. If you are not sure about the correct SIG for an issue, you can try SIG-contributor-experience [here](/sig-contributor-experience#github-teams), or [ask in Slack](http://slack.k8s.io/). + +_Improvements needed_ + +* Open pull requests with all applicable SIGs to not have duplicate information in their CONTRIBUTING.md and instead link here. Keep it light, keep it clean, have only one source of truth. + +### File an Issue + +Not ready to contribute code, but see something that needs work? While the community encourages everyone to contribute code, it is also appreciated when someone reports an issue (aka problem). Issues should be filed under the appropriate Kubernetes subrepository. + +*Example:* a documentation issue should be opened to [kubernetes/website](https://github.com/kubernetes/website/issues). + +Make sure to adhere to the prompted submission guidelines while opening an issue. + +# Contributing + +(From:[here](/contributors/devel/collab.md)) + +Kubernetes is open source, but many of the people working on it do so as their day job. In order to avoid forcing people to be "at work" effectively 24/7, we want to establish some semi-formal protocols around development. Hopefully these rules make things go more smoothly. If you find that this is not the case, please complain loudly. + +As a potential contributor, your changes and ideas are welcome at any hour of the day or night, weekdays, weekends, and holidays. Please do not ever hesitate to ask a question or send a pull request. + +Our community guiding principles on how to create great code as a big group are found [here](/contributors/devel/collab.md). Beginner focused information can be found below in [Open a Pull Request](#open-a-pull-request) and [Code Review](#code-review). + +### Communication + +It is best to contact your [SIG](#learn-about-sigs) for issues related to the SIG's topic. Your SIG will be able to help you much more quickly than a general question would. + +For questions and troubleshooting, please feel free to use any of the methods of communication listed [here](/communication.md). The [kubernetes website](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/) also lists this information. + +## GitHub workflow + +To check out code to work on, please refer to [this guide](./github-workflow.md). + +## Open a Pull Request + +Pull requests are often called simply "PR". Kubernetes generally follows the standard [github pull request](https://help.github.com/articles/about-pull-requests/) process, but there is a layer of additional kubernetes specific (and sometimes SIG specific) differences. + +The first difference you'll see is that a bot will begin applying structured labels to your PR. + +The bot may also make some helpful suggestions for commands to run in your PR to facilitate review. These `/command` options can be entered in comments to trigger auto-labeling and notifications. The command reference is [here](https://go.k8s.io/bot-commands). + +Common new contributor PR issues are: + +* not having correctly signed the CLA ahead of your first PR (see [Sign the CLA](#sign-the-cla) section) +* finding the right SIG or reviewer(s) for the PR (see [Code Review](#code-review) section) and following any SIG specific contributing guidelines +* dealing with test cases which fail on your PR, unrelated to the changes you introduce (see [Test Flakes](http://velodrome.k8s.io/dashboard/db/bigquery-metrics?orgId=1)) + +The pull request workflow is described in detail [here](/contributors/devel/pull-requests.md#the-testing-and-merge-workflow). + +## Code Review + +For a brief description of the importance of code review, please read [On Code Review](/contributors/devel/community-expectations.md#code-review). There are two aspects of code review: giving and receiving. + +To make it easier for your PR to receive reviews, consider the reviewers will need you to: + +* write [good commit messages](https://chris.beams.io/posts/git-commit/) +* break large changes into a logical series of smaller patches which individually make easily understandable changes, and in aggregate solve a broader issue +* label PRs with appropriate SIGs and reviewers: to do this read the messages the bot sends you to guide you through the PR process + +Reviewers, the people giving review, are highly encouraged to revisit the [Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md) and must go above and beyond to promote a collaborative, respectful Kubernetes community. When reviewing PRs from others [The Gentle Art of Patch Review](http://sage.thesharps.us/2014/09/01/the-gentle-art-of-patch-review/) suggests an iterative series of focuses which is designed to lead new contributors to positive collaboration without inundating them initially with nuances: + +* Is the idea behind the contribution sound? +* Is the contribution architected correctly? +* Is the contribution polished? + +## Testing + +Testing is the responsibility of all contributors and is in part owned by all sigs, but is also coordinated by [sig-testing](/sig-testing). + +The main testing overview document is [here](/contributors/devel/testing.md). + +There are three types of test in kubernetes. The location of the test code varies with type, as does the specifics of the environment needed to successfully run the test: + +* Unit: These confirm that a particular function behaves as intended. Golang includes native ability for unit testing via the [testing](https://golang.org/pkg/testing/) package. Unit test source code can be found adjacent to the corresponding source code within a given package. For example: functions defined in [kubernetes/cmd/kubeadm/app/util/version.go](https://git.k8s.io/kubernetes/cmd/kubeadm/app/util/version.go) will have unit tests in [kubernetes/cmd/kubeadm/app/util/version_test.go](https://git.k8s.io/kubernetes/cmd/kubeadm/app/util/version_test.go). These are easily run locally be any developer on any OS. +* Integration: These tests cover interactions of package components or interactions between kubernetes components and some other non-kubernetes system resource (eg: etcd). An example would be testing whether a piece of code can correctly store data to or retrieve data from etcd. Integration tests are stored in [kubernetes/test/integration/](https://git.k8s.io/kubernetes/test/integration). Running these can require the developer set up additional functionality on their development system. +* End-to-end ("e2e"): These are broad tests of overall kubernetes system behavior and coherence. These are more complicated as they require a functional kubernetes cluster built from the sources to be tested. A separate document [here](/contributors/devel/e2e-tests.md) details e2e testing and test cases themselves can be found in [kubernetes/test/e2e/](https://git.k8s.io/kubernetes/test/e2e). + +Continuous integration will run these tests either as pre-submits on PRs, post-submits against master/release branches, or both. The results appear on [testgrid](https://testgrid.k8s.io). + +sig-testing is responsible for that official infrastructure and CI. The associated automation is tracked in the [test-infra repo](https://git.k8s.io/test-infra). If you're looking to run e2e tests on your own infrastructure, [kubetest](https://git.k8s.io/test-infra/kubetest) is the mechanism. + +## Security + +_Improvements needed_ +* Please help write this section. + +## Documentation + +_Improvements needed_ +* Please help write this section. + +## Issues Management or Triage + +Have you ever noticed the total number of [open issues](https://issues.k8s.io)? This number at any given time is typically high. Helping to manage or triage these open issues can be a great contribution to the Kubernetes project. This is also a great opportunity to learn about the various areas of the project. Refer to the [Kubernetes Issue Triage Guidelines](/contributors/devel/issues.md) for more information. + +# Community + +If you haven't noticed by now, we have a large, lively, and friendly open-source community. We depend on new people becoming members and regular code contributors, so we would like you to come join us. To find out more about our community structure, different levels of membership and code contributors, please [explore here](/community-membership.md). + +_Improvements needed_ + +* The top level k/community/README.md should be a good starting point for what the community is and does. (see above instructions on rewriting this file) + +## Events +Kubernetes is the main focus of CloudNativeCon/KubeCon, held twice per year in EMEA and in North America. Information about these and other community events is available on the CNCF [events](https://www.cncf.io/events/) pages. + +### Meetups + +_Improvements needed_ +* include link to meetups +* information on CNCF support for founding a Meetup + +### KubeCon + +_Improvements needed_ +* write friendly blurb about KubeCon, and include links + +## Mentorship + +_Improvements needed_ +* Link and mini description for Kubernetes Pilots should go here. diff --git a/contributors/devel/git_workflow.png b/contributors/guide/git_workflow.png Binary files differindex 80a66248..80a66248 100644 --- a/contributors/devel/git_workflow.png +++ b/contributors/guide/git_workflow.png diff --git a/contributors/guide/github-workflow.md b/contributors/guide/github-workflow.md new file mode 100644 index 00000000..f3b77090 --- /dev/null +++ b/contributors/guide/github-workflow.md @@ -0,0 +1,229 @@ +## Workflow + + + +### 1 Fork in the cloud + +1. Visit https://github.com/kubernetes/kubernetes +2. Click `Fork` button (top right) to establish a cloud-based fork. + +### 2 Clone fork to local storage + +Per Go's [workspace instructions][go-workspace], place Kubernetes' code on your +`GOPATH` using the following cloning procedure. + +Define a local working directory: + +```sh +# If your GOPATH has multiple paths, pick +# just one and use it instead of $GOPATH here. +# You must follow exactly this pattern, +# neither `$GOPATH/src/github.com/${your github profile name/` +# nor any other pattern will work. +working_dir=$GOPATH/src/k8s.io +``` + +> If you already do Go development on github, the `k8s.io` directory +> will be a sibling to your existing `github.com` directory. + +Set `user` to match your github profile name: + +```sh +user={your github profile name} +``` + +Both `$working_dir` and `$user` are mentioned in the figure above. + +Create your clone: + +```sh +mkdir -p $working_dir +cd $working_dir +git clone https://github.com/$user/kubernetes.git +# or: git clone git@github.com:$user/kubernetes.git + +cd $working_dir/kubernetes +git remote add upstream https://github.com/kubernetes/kubernetes.git +# or: git remote add upstream git@github.com:kubernetes/kubernetes.git + +# Never push to upstream master +git remote set-url --push upstream no_push + +# Confirm that your remotes make sense: +git remote -v +``` + +### 3 Branch + +Get your local master up to date: + +```sh +cd $working_dir/kubernetes +git fetch upstream +git checkout master +git rebase upstream/master +``` + +Branch from it: +```sh +git checkout -b myfeature +``` + +Then edit code on the `myfeature` branch. + +#### Build + +*Note:* If you are using `CDPATH`, you must either start it with a leading colon, or unset the variable. The make rules and scripts to build require the current directory to come first on the CD search path in order to properly navigate between directories. + +```sh +cd $working_dir/kubernetes +make +``` + +To remove the limit on the number of errors the Go compiler reports (default +limit is 10 errors): +```sh +make GOGCFLAGS="-e" +``` + +To build with optimizations disabled (enables use of source debug tools): + +```sh +make GOGCFLAGS="-N -l" +``` + +To build binaries for all platforms: + +```sh +make cross +``` + +#### Install etcd + +```sh +hack/install-etcd.sh +``` + +#### Test + +```sh +cd $working_dir/kubernetes + +# Run all the presubmission verification. Then, run a specific update script (hack/update-*.sh) +# for each failed verification. For example: +# hack/update-gofmt.sh (to make sure all files are correctly formatted, usually needed when you add new files) +# hack/update-bazel.sh (to update bazel build related files, usually needed when you add or remove imports) +make verify + +# Alternatively, run all update scripts to avoid fixing verification failures one by one. +make update + +# Run every unit test +make test + +# Run package tests verbosely +make test WHAT=./pkg/api/helper GOFLAGS=-v + +# Run integration tests, requires etcd +# For more info, visit https://git.k8s.io/community/contributors/devel/testing.md#integration-tests +make test-integration + +# Run e2e tests by building test binaries, turn up a test cluster, run all tests, and tear the cluster down +# Equivalent to: go run hack/e2e.go -- -v --build --up --test --down +# Note: running all e2e tests takes a LONG time! To run specific e2e tests, visit: +# https://git.k8s.io/community/contributors/devel/e2e-tests.md#building-kubernetes-and-running-the-tests +make test-e2e +``` + +See the [testing guide](/contributors/devel/testing.md) and [end-to-end tests](/contributors/devel/e2e-tests.md) +for additional information and scenarios. + +Run `make help` for additional information on these make targets. + +### 4 Keep your branch in sync + +```sh +# While on your myfeature branch +git fetch upstream +git rebase upstream/master +``` + +Please don't use `git pull` instead of the above `fetch` / `rebase`. `git pull` +does a merge, which leaves merge commits. These make the commit history messy +and violate the principle that commits ought to be individually understandable +and useful (see below). You can also consider changing your `.git/config` file via +`git config branch.autoSetupRebase always` to change the behavior of `git pull`. + +### 5 Commit + +Commit your changes. + +```sh +git commit +``` +Likely you go back and edit/build/test some more then `commit --amend` +in a few cycles. + +### 6 Push + +When ready to review (or just to establish an offsite backup or your work), +push your branch to your fork on `github.com`: + +```sh +git push -f ${your_remote_name} myfeature +``` + +### 7 Create a pull request + +1. Visit your fork at https://github.com/$user/kubernetes +2. Click the `Compare & Pull Request` button next to your `myfeature` branch. +3. Check out the pull request [process](/contributors/devel/pull-requests.md) for more details and + advice. + +_If you have upstream write access_, please refrain from using the GitHub UI for +creating PRs, because GitHub will create the PR branch inside the main +repository rather than inside your fork. + +#### Get a code review + +Once your pull request has been opened it will be assigned to one or more +reviewers. Those reviewers will do a thorough code review, looking for +correctness, bugs, opportunities for improvement, documentation and comments, +and style. + +Commit changes made in response to review comments to the same branch on your +fork. + +Very small PRs are easy to review. Very large PRs are very difficult to review. +At the assigned reviewer's discretion, a PR may be switched to use +[Reviewable](https://reviewable.k8s.io) instead. Once a PR is switched to +Reviewable, please ONLY send or reply to comments through Reviewable. Mixing +code review tools can be very confusing. + +#### Squash and Merge + +Upon merge (by either you or your reviewer), all commits left on the review +branch should represent meaningful milestones or units of work. Use commits to +add clarity to the development and review process. + +Before merging a PR, squash any _fix review feedback_, _typo_, _merged_, and +_rebased_ sorts of commits. + +It is not imperative that every commit in a PR compile and pass tests +independently, but it is worth striving for. + +In particular, if you happened to have used `git merge` and have merge +commits, please squash those away: they do not meet the above test. + +A nifty way to manage the commits in your PR is to do an [interactive +rebase](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History), +which will let you tell git what to do with every commit: + +```sh +git fetch upstream +git rebase -i upstream/master +``` + +For mass automated fixups (e.g. automated doc formatting), use one or more +commits for the changes to tooling and a final commit to apply the fixup en +masse. This makes reviews easier.
\ No newline at end of file diff --git a/community/2016-events/Kubernetes_1st_Bday.md b/events/2016/Kubernetes_1st_Bday.md index 58290215..58290215 100644 --- a/community/2016-events/Kubernetes_1st_Bday.md +++ b/events/2016/Kubernetes_1st_Bday.md diff --git a/community/2016-events/developer-summit-2016/KubDevSummitVoting.md b/events/2016/developer-summit-2016/KubDevSummitVoting.md index 46909c02..46909c02 100644 --- a/community/2016-events/developer-summit-2016/KubDevSummitVoting.md +++ b/events/2016/developer-summit-2016/KubDevSummitVoting.md diff --git a/community/2016-events/developer-summit-2016/Kubernetes_Dev_Summit.md b/events/2016/developer-summit-2016/Kubernetes_Dev_Summit.md index f66df508..f66df508 100644 --- a/community/2016-events/developer-summit-2016/Kubernetes_Dev_Summit.md +++ b/events/2016/developer-summit-2016/Kubernetes_Dev_Summit.md diff --git a/community/2016-events/developer-summit-2016/application_service_definition_notes.md b/events/2016/developer-summit-2016/application_service_definition_notes.md index e8f4c0c5..e8f4c0c5 100644 --- a/community/2016-events/developer-summit-2016/application_service_definition_notes.md +++ b/events/2016/developer-summit-2016/application_service_definition_notes.md diff --git a/community/2016-events/developer-summit-2016/cluster_federation_notes.md b/events/2016/developer-summit-2016/cluster_federation_notes.md index 362cbafe..362cbafe 100644 --- a/community/2016-events/developer-summit-2016/cluster_federation_notes.md +++ b/events/2016/developer-summit-2016/cluster_federation_notes.md diff --git a/community/2016-events/developer-summit-2016/cluster_lifecycle_notes.md b/events/2016/developer-summit-2016/cluster_lifecycle_notes.md index f42df9f6..f42df9f6 100644 --- a/community/2016-events/developer-summit-2016/cluster_lifecycle_notes.md +++ b/events/2016/developer-summit-2016/cluster_lifecycle_notes.md diff --git a/community/2016-events/developer-summit-2016/k8sDevSummitSchedule.pdf b/events/2016/developer-summit-2016/k8sDevSummitSchedule.pdf Binary files differindex 6cd8d01b..6cd8d01b 100644 --- a/community/2016-events/developer-summit-2016/k8sDevSummitSchedule.pdf +++ b/events/2016/developer-summit-2016/k8sDevSummitSchedule.pdf diff --git a/community/2016-events/developer-summit-2016/statefulset_notes.md b/events/2016/developer-summit-2016/statefulset_notes.md index d019d256..d019d256 100644 --- a/community/2016-events/developer-summit-2016/statefulset_notes.md +++ b/events/2016/developer-summit-2016/statefulset_notes.md diff --git a/community/2016-events/fixit201606.md b/events/2016/fixit201606.md index f7d43641..f7d43641 100644 --- a/community/2016-events/fixit201606.md +++ b/events/2016/fixit201606.md diff --git a/community/2017-events/05-leadership-summit/announcement.md b/events/2017/05-leadership-summit/announcement.md index dc03cb69..99ed097b 100644 --- a/community/2017-events/05-leadership-summit/announcement.md +++ b/events/2017/05-leadership-summit/announcement.md @@ -1,5 +1,5 @@ This is an announcement for the 2017 Kubernetes Leadership Summit, which will occur on June 2nd, 2017 in San Jose, CA. -This event will be similar to the [Kubernetes Developer's Summit](https://github.com/kubernetes/community/blob/master/community/2016-events/developer-summit-2016/Kubernetes_Dev_Summit.md) in November +This event will be similar to the [Kubernetes Developer's Summit](/events/2016/developer-summit-2016/Kubernetes_Dev_Summit.md) in November 2016, but involving a smaller smaller audience comprised solely of leaders and influencers of the community. These leaders and influences include the SIG leads, release managers, and representatives from several companies, including (but not limited to) Google, Red Hat, CoreOS, WeaveWorks, Deis, and Mirantis. diff --git a/community/2017-events/05-leadership-summit/session-notes/0300-0345_CODEORGANIZATION.md b/events/2017/05-leadership-summit/session-notes/0300-0345_CODEORGANIZATION.md index 8328fdbf..a6fa64de 100644 --- a/community/2017-events/05-leadership-summit/session-notes/0300-0345_CODEORGANIZATION.md +++ b/events/2017/05-leadership-summit/session-notes/0300-0345_CODEORGANIZATION.md @@ -39,7 +39,7 @@ Ideally, code would be divided along lines of ownership, by SIG and subproject, Next up: * [kubeadm](https://github.com/kubernetes/kubernetes/issues/35314) * Federation -* [cloudprovider](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/README.md) and [cluster](https://github.com/kubernetes/kubernetes/blob/master/cluster/README.md) +* [cloudprovider](https://git.k8s.io/kubernetes/pkg/cloudprovider/README.md) and [cluster](https://git.k8s.io/kubernetes/cluster/README.md) * Scheduler * [Kubelet](https://github.com/kubernetes/kubernetes/issues/444) diff --git a/community/2017-events/05-leadership-summit/session-notes/0905-0930_STATE_OF_THE_KUBE.md b/events/2017/05-leadership-summit/session-notes/0905-0930_STATE_OF_THE_KUBE.md index 05287aad..05287aad 100644 --- a/community/2017-events/05-leadership-summit/session-notes/0905-0930_STATE_OF_THE_KUBE.md +++ b/events/2017/05-leadership-summit/session-notes/0905-0930_STATE_OF_THE_KUBE.md diff --git a/community/2017-events/05-leadership-summit/session-notes/0940-1000_PROJECT_GOALS_ROADMAP.md b/events/2017/05-leadership-summit/session-notes/0940-1000_PROJECT_GOALS_ROADMAP.md index 453b3931..453b3931 100644 --- a/community/2017-events/05-leadership-summit/session-notes/0940-1000_PROJECT_GOALS_ROADMAP.md +++ b/events/2017/05-leadership-summit/session-notes/0940-1000_PROJECT_GOALS_ROADMAP.md diff --git a/community/2017-events/05-leadership-summit/session-notes/1015-1115_UPDATES_GOVERNANCE_AND_CNCF.md b/events/2017/05-leadership-summit/session-notes/1015-1115_UPDATES_GOVERNANCE_AND_CNCF.md index 7e49baa8..7e49baa8 100644 --- a/community/2017-events/05-leadership-summit/session-notes/1015-1115_UPDATES_GOVERNANCE_AND_CNCF.md +++ b/events/2017/05-leadership-summit/session-notes/1015-1115_UPDATES_GOVERNANCE_AND_CNCF.md diff --git a/community/2017-events/05-leadership-summit/session-notes/1145_1245_ARCHITECTURAL_ROADMAP.md b/events/2017/05-leadership-summit/session-notes/1145_1245_ARCHITECTURAL_ROADMAP.md index 8349c40d..8349c40d 100644 --- a/community/2017-events/05-leadership-summit/session-notes/1145_1245_ARCHITECTURAL_ROADMAP.md +++ b/events/2017/05-leadership-summit/session-notes/1145_1245_ARCHITECTURAL_ROADMAP.md diff --git a/community/2017-events/05-leadership-summit/session-notes/1345-1430_SIGGOVERNANCE.md b/events/2017/05-leadership-summit/session-notes/1345-1430_SIGGOVERNANCE.md index 397c7c6a..397c7c6a 100755 --- a/community/2017-events/05-leadership-summit/session-notes/1345-1430_SIGGOVERNANCE.md +++ b/events/2017/05-leadership-summit/session-notes/1345-1430_SIGGOVERNANCE.md diff --git a/community/2017-events/05-leadership-summit/session-notes/1500-1545_COMMUNITY_LADDER.md b/events/2017/05-leadership-summit/session-notes/1500-1545_COMMUNITY_LADDER.md index 865d4999..865d4999 100644 --- a/community/2017-events/05-leadership-summit/session-notes/1500-1545_COMMUNITY_LADDER.md +++ b/events/2017/05-leadership-summit/session-notes/1500-1545_COMMUNITY_LADDER.md diff --git a/community/2017-events/05-leadership-summit/session-notes/1600-1740_PRESENTATIONS_FROM_BREAKOUTS_AND_CALL_TO_ACTIONS.md b/events/2017/05-leadership-summit/session-notes/1600-1740_PRESENTATIONS_FROM_BREAKOUTS_AND_CALL_TO_ACTIONS.md index fa52f1dc..fa52f1dc 100644 --- a/community/2017-events/05-leadership-summit/session-notes/1600-1740_PRESENTATIONS_FROM_BREAKOUTS_AND_CALL_TO_ACTIONS.md +++ b/events/2017/05-leadership-summit/session-notes/1600-1740_PRESENTATIONS_FROM_BREAKOUTS_AND_CALL_TO_ACTIONS.md diff --git a/community/2017-events/05-leadership-summit/session-notes/readme.md b/events/2017/05-leadership-summit/session-notes/readme.md index c7d61f0e..c7d61f0e 100644 --- a/community/2017-events/05-leadership-summit/session-notes/readme.md +++ b/events/2017/05-leadership-summit/session-notes/readme.md diff --git a/events/2017/12-contributor-summit/ContribSummitInformation.md b/events/2017/12-contributor-summit/ContribSummitInformation.md new file mode 100644 index 00000000..be0f6bbd --- /dev/null +++ b/events/2017/12-contributor-summit/ContribSummitInformation.md @@ -0,0 +1,70 @@ +# 2017 Kubernetes Contributor Summit Information +fka DevSummit + +## What: +The Contributor Summit provides an avenue for Kubernetes contributors to connect face to face and mindshare about future community development and community governance endeavors. + +In some sense, the summit is a real-life extension of the community meetings and SIG meetings. + +## When: +December 5th, 2017 (before KubeCon) +All day event with happy hour reception to close + +**Agenda:** +Times in CST +- 08:00 am - Registration and light breakfast +- 09:00 am - Welcome and start of content +- 12:00 pm - Lunch provided +- 01:00 pm - Sessions resume +- 05:00 pm - Happy Hour onsite + +View the complete session schedule [here](schedule.png). +Session schedule will be available onsite in print as well as on TVs. + +## Where: +Austin Convention Center +500 E Cesar Chavez St, +Austin, TX 78701 +(same location as KubeCon) + +**Registration:** +You'll pick up your summit badge at the same registration tables as KubeCon.* +Level 1, near Hall 1 + +**Breakfast:** +Level 3, Outside of Meeting Room 5 + +**Sessions:** +Level 3, Meeting Room 5 - Welcome @ 9am +Level 3, Meeting Room 4, Meeting Room 6a, Meeting Room 6b - Breakout session rooms + +*_Note-Your summit registration is NOT your KubeCon registration. Tickets are almost sold out for KubeCon, please purchase those separately from the [event page](http://events.linuxfoundation.org/events/kubecon-and-cloudnativecon-north-america/attend/register)._ + +## Invitations: +Selected attendees are Kubernetes upstream contributors including SIG members and lottery winners from a [members of standing](/community-membership.md) pool. +We realize that this is not the best approach now that our community is growing in exponential ways. In 2018, we will be unfolding a new strategy for bringing our contributors together so all voices can be heard. + +Invites went out on October 19th. There are ten spots left. Please reach out to parispittman@google.com to claim one. + +## Format: +"loosely-structured unconference" +Rather than speakers and presentations, we will have moderators/facilitators and discussion topics, alongside all-day completely unstructured hacking. + +The discussion sessions will be in an open fishbowl format — rings of chairs, with inner rings driving the discussion — where anyone can contribute. The various sessions will be proposed and voted on by the community in the weeks leading up to the event. This allows the community to motivate the events of the day without dedicating precious day-of time to choosing the sessions and making a schedule. + +There will be 2 rooms dedicated to these sessions running in parallel all day. Each session should last between 45 minutes and an hour. There will be a smaller room available for unstructured discussion all day. + +## Call for Topics: +Call for topics and voting is now closed. You can view the complete list of proposed topics and abstracts [here](https://docs.google.com/spreadsheets/d/1miMinwk3Cp_4KV0xj36gIT3XdN4JtDcnAhkLZxG-qCQ/edit?usp=sharing). + +## Desired Outcomes: +* Generate notes from the sessions to feed the project's documentation and knowledge base, and also to keep non-attendees plugged in +* Make (and document) recommendations and decisions for the near-term and mid-term future of the project +* Come up with upcoming action items, as well as leaders for those action items, for the various topics that we discuss + + +## Misc: + +A photographer and videographer will be onsite collecting b-roll and other shots for KubeCon. If you would rather not be involved, please reach out to an organizer on the day of so we may accomodate you. + +Further details to be updated on this doc. Please check back for a complete guide. diff --git a/events/2017/12-contributor-summit/Meeting Room 4ABC - 1st set, Tues fishbowl.PNG b/events/2017/12-contributor-summit/Meeting Room 4ABC - 1st set, Tues fishbowl.PNG Binary files differnew file mode 100644 index 00000000..c866d23c --- /dev/null +++ b/events/2017/12-contributor-summit/Meeting Room 4ABC - 1st set, Tues fishbowl.PNG diff --git a/events/2017/12-contributor-summit/Meeting Room 6A and 6B - 1st Set, Tues fishbowl.PNG b/events/2017/12-contributor-summit/Meeting Room 6A and 6B - 1st Set, Tues fishbowl.PNG Binary files differnew file mode 100644 index 00000000..8c334121 --- /dev/null +++ b/events/2017/12-contributor-summit/Meeting Room 6A and 6B - 1st Set, Tues fishbowl.PNG diff --git a/events/2017/12-contributor-summit/breaking-up-the-monolith.md b/events/2017/12-contributor-summit/breaking-up-the-monolith.md new file mode 100644 index 00000000..265dcd60 --- /dev/null +++ b/events/2017/12-contributor-summit/breaking-up-the-monolith.md @@ -0,0 +1,76 @@ +# Breaking Up The Monolith + +Note Takers: Josh Berkus(@jberkus), Aaron Crickenberger (@spiffxp) + +Here's what we're covering: build process, issue tracking, logistics. + +## Question: are we commited emotionally 100% to splitting the monolith + +Opinion: commited, but need to define what that means (past provides the impetus) + +We've been trying [the monolith] for so long and it's too expensive. + +- erictune: as the project matures the core can't keep growing at the same rate +- mattfarina: reason to break it up - developer experience,difficulty to contribute pushes people away from contributing (hard to step into issue queue, etc) +- clayton: I'm scared by this because kubectl is never going to be simpler, it may be easier to just write a new one +- erictune: ecosystem yes, more contributors... narrowly defined core no +- bburns: we don't do a good job of measuring the cost of this, we love breaking things up because it's "cleaner" but we're not estimating the cost in terms of the human effort and complexity +- lavalamp: respectfully disagree, we _have_ estimated the cost, knew it was going to be painfu but we haven't had a choice +- timstclair: nobody has ever answered the question to be of how we're going to deliver a set of binaries as part of a release, there isn't a cohesive plan for doing this +- timstclair: how are you actually going to get test infra to test all the things +- bburns: how do you actually find the commit that causes a failure in e2e +- lavalamp: have you see the doc I have posted everywhere this comes up? (TODO: link) main repo becomes integration point, binary artifacts produced by other repos +- clayton: kubectl is a very "big" thing, if we said the problem is that kubectl itself is too big, it might be better to focus on something new and novel, it might be better to build a community around that +- dims/matt: can we have a list of four or five items we want to move out in priority + - Kubectl + - Client-Go? (but that's already published separately) (this is way too involved for this session) + - Try to tackle "how do we deal with client-go" elsewhere + - kubeadm is in main repo to catch release train, wants to move out, but it's fairly hard. We tried to move it out, but couldn't and follow the release train. +- clayton: government approach: why build one thing when we can build two for twice the cost? an (extracted) client-go that's all high-perf, a completely rewritten client-go that's human focused and more reusable + +## Question: Do we understand the problem we're trying to solve with the repo split + +Assumption: contributor velocity is correlated with velocity, new contributor experince +Assumption: "big tangled ball of pasta" is hard to contribute to + +- thockin: our codebase is not organized well (need to make it easier to actually *find* what you want to contribute to) one of the things we could do is "move code around" until it looks effectively like there are multiple repos in our repo? eg if we centralized all of the kubelet logic in one directory except for one util directory, maybe that would be an improvement +- jberkus: people who work on other large projects say that modularity is the way to go. Not necessarily seperate repos, but a modular architecture. +- thockin: what we've done with github bots alleviates a lot of the pain we've had with github +- dims: two problems + - vendoring in all kinds of sdk's that we don't care about, they get dragged in (e.g. AWS SDK) and if you're only working on eg: openstack related stuff, it's hard to get signoff on getting other stuff in if it's in your own repo it's easier to do that than in the main repo + - (guess we missed #2) +- erictune: we've already promised people cloud providers you can write in different ways, extensibility of apis, operators, if we deliver on all this we will have enough modularity to improve user/developer experience +- mattfarina: I think you're right about that +- First thing: we're breaking up cloud providers, it helps let the long tail of cloud providers go out + - what about storage providers, if we break them up there's a clear interface for storage providers +- second thing: here are api/components that are required, required but swappable, optional... how can you do that swapping around of stuff in the monorepo +- robertbailey: when I proposed this session I thought we had all agreed that breaking the repo was the way to go, but maybe we don't have that conensus could we try and figure out who is the decision maker / set of decision makers there and try to come to consensus +- dchen1107: I heard a lot of benefits of the split. So there's a huge cost for release management, as the release manager, I don't know what's in the other repositiory. Clear APIs and interfaces could be built without needing separate repositories. Gave example of Docker and CRI, which got API without moving repos. +- Example of Cloud Foundry, build process which took two weeks. How do we ensure that we're delivering security updates quickly? +- thockin: We can put many things in APIs and cloud providers are a good example of that. Splitting stuff out into multiple repos comes down to the question of: are we a piece of software or a distribution? You'll have to come to my talk to see the rest. +- spiffxp: Increasing our dependancy on integration tools will add overhead and process we don't have now. But I understand that most OSS people expect multiple repos and it's easier for them. Github notifications are awful, having multiple repos would make this better. The bots have improved the automation situation, but not really triaging notifications. How many issues do we have that are based on Github. Maybe we should improve the routing of GH notificaitons instead? +- solly: if we split, we really need to make it easier for tiny repos to plug into our automation and other tools. I shouldn't have to figure out who to ask about getting plugged in, we should have a doc. We need to make sure it's not possible for repos to fall through the cracks like Heapster, which I work on. +- bgrant: We're already in the land of 90 repos. We don't need to debate splitting, we're alread split. We have incubator, and kubernates-client. I think client has helped a lot, we have 7 languages and that'll grow. +- bgrant: the velocity of things in the monorepo are static +- thockin: if issues in the main repo are more than 6 weeks old, nobody looks at it. There's like 400-500 abandoned "network" issues. +- bgrant: we tested gitlab and gerrit, and importing kubernetes failed. We didn't find them that much better. +- spiffxp: we have hundreds of directories with no owners files, which is one of the reasons for excessive notifications. +- bgrant: kubernetes, as a API-driven system, you have to touch the api to do almost anything. we've added mechanisms like CRD to extend the API. We need SDKs to build kube-style APIs. +- dims: I want to focus on the cost we're paying right now. First we had the google KMS provider. Then we had to kick out the KMS provider, and there was a PR to add the gRPC interface, but it didn't go into 1.9. +- thockin: the cloud provider stuff is an obvious breeding ground for new functionality, how and if we should add a whole seperate grpc plugin interface is a seperate question +- jdumars: the vault provider thing was one of the ebtter things that happened, it pushed us at MS to thing about genercizing the solution, it pushed us to think about what's better for the community vs. what's better for the provider +- jdumars: flipside is we need to have a process where people can up with a well accepted / adopted solution, the vault provider thing was one way of doing that +- lavalamp: I tend to think that most extension points are special snowflakes and you can't have a generic process for adding a new extension point +- thockin: wandering back to kubernetes/kubrnetes "main point", looking at staging as "already broken out", are there other ones that we want to break out? +- dims: kubeadm could move out if needed, could move it to staging for sure +- thockin: so what about the rest? eg: kubelet, kube-proxy... do we think that people will concretely get benefits from that? or will that cause more pain +- thockin: we recognize this will slow down things +- lavalamp: there are utility functions that people commonly use and there's no good common place +- lavalamp: for kubectl at least it's sphaghetti code that pulls in lots of packages and makes it difficult to do technically +- thockin: do we think that life would be better at the end of that tunnel, would things be better if kubectl was a different repository, etc. +- timallclair: I'm worried about dependancy management, godeps is already a nightmare and with multiple repos it would be worse. +- luxas: in the kubeadm attack plan, we need to get a release for multiple repos. We need the kubeadm repo to be authoritative, and be able to include it in a build. +- pwittrock: how has "staging" improved development? can we see any of the perceived or hoped-for benefits by looking at staging repos as example use cases? +- lavalamp: getting to the "staging" state and then stopping is because api-machinery was unblocked once we got there +- thockin: the reason I consider "staging" solved, is you have to untangle a lot of the stuff already +- erictune: I would make a plea that we finish some of our started-but-not-finished breakaparts
\ No newline at end of file diff --git a/events/2017/12-contributor-summit/cloud-native-design-refactoring-across-k8s.md b/events/2017/12-contributor-summit/cloud-native-design-refactoring-across-k8s.md new file mode 100644 index 00000000..66aadf88 --- /dev/null +++ b/events/2017/12-contributor-summit/cloud-native-design-refactoring-across-k8s.md @@ -0,0 +1,60 @@ +**Contributor Summit - KubeCon 2017** + +**Cloud Native Design/Refactoring across Kubernetes** + +*Lead: Joe Beda(**[@jbeda](https://twitter.com/jbeda)*)* + +*Co-lead: Amit Kumar Jaiswal(**[@AMIT_GKP](https://twitter.com/AMIT_GKP)*)* + +**Abstract** + +Explore how to cleanly support cloud-native behaviors, such as standardized Kubernetes logs, injectable configuration and common queryable APIs in Kubernetes components. While this discussion isn't only about containers, orchestrating OpenStack deployment/management within Kubernetes via OpenStack-Helm or Kolla-Kubernetes paves the way for better upgrade capabilities. They will also improve the ability to run individual Kubernetes services independently or in combination with other adjacent technologies. + +**Agenda** + +1. Cloud Native Ecosystems + +2. Kubernetes Abstractions & Primitives + +3. Kubernetes Design Patterns + +4. Refactoring across Kubernetes + +5. Benefits of using Kubernetes + +**Issues** + +** **- We’re looking for someone to help us out on issues related to refactoring across Kubernetes like [Issues#51405](https://github.com/kubernetes/kubernetes/issues/51405), [#46735](https://github.com/kubernetes/kubernetes/issues/46735), [#54090](https://github.com/kubernetes/kubernetes/issues/54090), [#55151](https://github.com/kubernetes/kubernetes/issues/55151) + + - Consolidating volume util files like *pkg/volume/util.go, pkg/volume/util/util.go, pkg/volume/util/volumehelper/volumehelper.go *[need better documentation of each file is to emcompass] + + - Enhancing e2e tests to track cloud provider’s API Quotas + + - Like changing fluentd to use CRI log Path and eventually deprecates the old container path + + - Issues with deploying/adopting Kubernetes for specific applications use-cases + +**Questions** + + - Security and granularity across Kubernetes + + - Kube API issues like it should not depend on *--cloud-provider* and *--cloud-config* + + - Helping us out with API documentation and Configuring Best Practices? + + - What are the new tools to deal with testing frameworks for NFV/SDN across Kubernetes? + + - Kubelet Flag subfields precedence v/s files/ConfigMaps to Kubelet Config + + - How to dynamically provision an AWS EBS volume from a snapshot? + + - Work around Object definition in OpenAPI schema like definition of the *PersistentVolumeSpec, PersistenceVolumeClaimSpec* + + - Work around support for FUSE client for K8s cephfs. + + + +**Audience Feedback** + + - + diff --git a/events/2017/12-contributor-summit/cloud-provider.md b/events/2017/12-contributor-summit/cloud-provider.md new file mode 100644 index 00000000..a18a0fb7 --- /dev/null +++ b/events/2017/12-contributor-summit/cloud-provider.md @@ -0,0 +1,86 @@ +# CloudProvider Update +*Lead: Walter Fender / cheftako* +*Notetaker: Jago Macleod / jagosan* +4pm + +Goals: + +Make it easy for any new cloud provider to be able to add support for new cloud provider without having to submit code to k/k repo. [good shape!] +For currently in-tree cloud providers to be able to build and release without having to ‘wait’ for upstream kubernetes. (e.g. security patches and features) +Also suboptimal that for technical reasons, other cloudprovider’s ‘init’s are being run in the hosting cloud provider’s environments. + +[TODO: name?]: From a user’s perspective, you might like a more modular cloudprovider. You might want ELB but not the storage from that cloudprovider. + +cheftako: where are we: + +There is a PR out that is a WIP that proves that in GCE it is possible to bring up a cluster + +Kube-controller-manager is disabled. cloud-controller-manager is working there + +Kube-mark tests are a challenge. No support for conditional behavior in test-infra. Now we need an n1-standard2 master. + +Other outstanding issues: + +Who has seen the following in the tests: + +If !GCE -> skip. + +Many raise hands. This is recognized as a gap that needs to be addressed. + +Also need volunteers for each of the cloud providers. Lot of work to be done. + +Thockin would like to minimize the amount of code that lives outside of k/k repo. If we were to take all of lifecycle and move it out of k/k, there is a lot of functionality that could diverge, or could be missing for a user in certain cloud providers. + +Take a page out of API Server book and make a generic controller manager and use for kube-controller-manager and cloud-controller-manager. + +Comment: spent a couple of days looking through the GCE work and found ~4 bugs. So there are a bunch of other + +Broke node controller out into 4 segments. + +Overview on where we are in the overall project: external cloud provicer + +We have a long way to go before all the in-tree cloud providers are moved out. + +One of the interesting questions is that eventually there are no in-tree cloud providers; there is now a distributed CI issue where all cloud providers have a mechanism through which to set up their own continuous testing and post results back to testgrid, to expose when changes to one area of the code inadvertently break one or many or all cloud providers. + +MS has a proof of concept basically done. Some questions: legally, what does this mean? E2E tests - how to make blocking. OpenStack -- in e2e tests, there is no tag to ensure nothing was broken in e2e. + +Jaice: to complicate matters, there may be primitives that make it necessary to make the testing much broader. There is a greater chance that the flakes occur in this world. + +thockin - on indemnity, we can make a dedicated repo that is still CNCF owned but controlled by cloud provider + +Exercise the top level interface + +Multiple levels of test here. In the final state, there is no ‘cloudprovider’ interface. You can test that binary / or binaries to test specific cloud implementations; then there is another level which is the e2e tests of basically everything that is the kubernetes functionality. + +When it comes to timing: the original plan was to depend on Flex Volumes. Subsequent discussions have tended toward depending instead on CSI as the volume solution. This means the timing is no earlier than Q3. + +Victory looks like this: + +Major cloud providers shipping and running cloud-controller-manager. Volumes can remain behind until CSI is production ready. + +Also, when we can eventually delete all the in-tree cloud provider code, thockin will buy the cake and the keg. + +Today, when you deal with CCM, and it says ‘what is my cloud provider’ it turns off cloud plugin, and volumes doesn’t work. There is, therefore, a new flag. Set cloudprovider=external and some other flag, can’t remember off top of head. Intention is that this will be a temporary second flag and will be removed as soon as possible. + +A lot of really good work has been done. Need to do a better job of organizing code in KCM & CCM. + +Open Question: will we ship cloud provider specific CCM’s as part of the kubernetes release? + +Or, what cloud providers are included or easily available with the binaries distributed in the Kubernetes release? + +Can we define better criteria than ‘you showed up before date xxxxxx’. + +Perhaps there is something that will work for a ‘vanilla’ system or locally, and some way to discover and install CCMs for available cloud providers. Likely the vendor / distributor will be packaging up some other process by which kubernetes is installed and configured. + +Thockin - think about what you do when you compile the linux kernel. Choose options during initialization and customize to hearts content. At the end, you get an artifact. Kbuild. Did a hack 18 months ago and proved it’s possible. + +thockin - do we have concrete list? Not a great list. We are going through the KEP process. AI: call to action. + +We are committed; we are making progress; we meet wednesdays at 10am Pacific Time. + +Fejta: think about a local provider. Move as much as possible onto something that is like minikube, e.g. Great enthusiasm for this concept. + +AI: fejta to share information on how to submit test results to testgrid and how to leverage the tooling for external cloud providers. + +Summary - yes! diff --git a/events/2017/12-contributor-summit/container-identity.md b/events/2017/12-contributor-summit/container-identity.md new file mode 100644 index 00000000..06a53a79 --- /dev/null +++ b/events/2017/12-contributor-summit/container-identity.md @@ -0,0 +1,168 @@ +Container identity + +Lead: Greg Castle + +How are people using k8s service accounts + +* Default service accounts? + +* User: + + * (Zalando) + + * 55 clusters + + * Only allow pre-set service accounts postgres "operator service accounts" + + * Don’t enable RBAC + + * Namespaces: usually use the default namespace, more sophisticated clients can use namespaces + + * Cluster per product + + * CRD that defines request for an OAuth2 token + + * Q: Would you want to tie it to a Kubernetes service account. E.g. annotation on a service account to get an OAuth2 token + + * (IBM) + + * Strict way of "cutting up cluster" + + * No team has access to the Kubernetes API + + * Humans do, pods don’t + + * Admins create service accounts and role bindings + + * Issues with pod authors in the namespace can run any service account. + + * Is there a missing binding to control the right to use a service account? + + * Namespace boundary + + * Application + + * (VMware) + + * 1 cluster + + * Projects are per-namespace (200 namespaces) + + * Wrote an admission controller between which pod is using which service account. Denies certain kind of changes. + + * Ownership model "I created this object, only I can change it" + + * Namespace but wanted sub-namespace primitives + + * Service accounts have labels, need to match the pod. + + * Q: Why not just different namespaces? + + * Don’t want to give users the ability to create namespaces + + * (Jetstat) + + * Dev, staging, and prod cluster + + * Namespace per team + + * Their developers don’t have permissions to create workloads + + * CI system is the thing that creates workloads, that controls stuff + + * (CoreOS) + + * Customers create namespaces that are controlled by CI/CD + + * Users have read-access to cluster + + * Central catalog controls the service account + + * Authoritative on RBAC + + * Catalog is currently controlled by CoreOS, can be used by users later + + * Namespaces + + * Catalog is enabled on a namespace + + * Also use service accounts to authenticate CI/CD systems + +* Container identity + + * Authenticate workloads against each other + + * Does this fit into the current way you use service accounts? + +* What about envoy istio? + + * Complementary efforts + + * Services can do service to service auth + + * Maybe not solving "I want to talk to s3, I want to talk to GitHub" + + * Give them a better way to provision x509 certs + + * Istio will be dependent on container identity + + * Istio and SPIFFE? + + * Istio is using SPIFFE conformant x509, not the workloads API + +* Q: how does Zalando RBAC CRDs? + + * Trust two engineers that have the permissions to create the CRDs + + * Emergency procedure with time limited API + +* Q: multiple clusters working together? + + * IBM: going to face it + + * CoreOS: thinking about users authenticating across multiple clusters through sharing public keys. + +* Liggitt: namespaces are the boundaries of trust + + * Originally thought about using pod templates so admins could create specific templates that pods MUST use. + + * Reference pod template in RC or deployment instead of inlining them + + * Never bubbled up to be the most important thing yet + +* Liggitt: Kubernetes API isn’t really designed for things that hold credentials + + * Moving the mounting and creation of the service account to the kubelet through flex volumes + + * Helps you leverage the node level integrations + +* Q: what about the secrets proposal? + + * KMS is being added + + * No change for encrypting the secret as you read it + +* Slides + + * Authorization should apply to the identity, rather than the namespace + + * Is multiple SA per namespace an anti-pattern? + + * Useful for bounding process within pods to certain permissions + + * Problem when a SA from cluster scoped manifests + +* Q: kerberos and vault bindings implemented as flex volumes + + * Is it important to have standard identity mechanisms? (Yes) + + * Can we add it to conformance? + + * Avoid lock-in for IAM + + * Cluster identity also becomes an issue too + + * Pod/UID/Namespace/Cluster + +* Potentially use OpenID Connect format so others can use that spec + diff --git a/events/2017/12-contributor-summit/current-state-of-client-libraries.md b/events/2017/12-contributor-summit/current-state-of-client-libraries.md new file mode 100644 index 00000000..c7ac0bad --- /dev/null +++ b/events/2017/12-contributor-summit/current-state-of-client-libraries.md @@ -0,0 +1,38 @@ +Story so far is described at https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/csi-client-structure-proposal.md +- linked from https://github.com/kubernetes-client/community/blob/master/design-docs/clients-library-structure.md + +Would like libs to be more "fluent" - should join up with the native language features e.g. JavaDoc, per-platform + +OpenAPI (Swagger) has limitations which force ugly workarounds. + - e.g. doesn't cover returning different types in response to a call e.g. a result or an error. + - OpenAPI 3 fixes a number of these limitations + +Key difficulty is which verbs apply to which objects - Go doesn't natively have this concept. + +OpenAPI doc is produced by running an api-server, which knows how to dispatch verbs to objects, and having it generate the OpenAPI + +Should we deprecate SPDY in favour of websockets? (probably yes) + +What do people want in a client library? + - Go lib smaller than 40MB + - Go lib with dynamic types (this exists?) + - Rust + - C++ + +Are there client libs for things like service catalog? Where do those live? + +How can clients talk protobufs to api-server? Set the content-type. + +K8s protobuf def uses an 'envelope' type covering things like storage version + +Protobuf not self-describing so not great for api-server CRUD operations + +A few bits of extra information in the protobuf definition would be great - e.g. pluralisation + +Client SIG is relatively under-represented and relatively easy to join in - please come along! + +kubeconfig semantics are not standardised - when reimplemented in different languages this can give unexpected results. Example: URL with no scheme supported by Go but not Java + +How should clients deal with rolling authorisation certificates? + +There are type wrinkles, like "intorstring" and "quantity"
\ No newline at end of file diff --git a/events/2017/12-contributor-summit/dashboard-ux-breakout.md b/events/2017/12-contributor-summit/dashboard-ux-breakout.md new file mode 100644 index 00000000..3dad6fe2 --- /dev/null +++ b/events/2017/12-contributor-summit/dashboard-ux-breakout.md @@ -0,0 +1,94 @@ +Kubernetes Dashboard UX breakout session 12.5.17, led by Rahul Dhide ([rahuldhide](https://github.com/rahuldhide)) and Dan Romlein + +* **Resources:** + + * Dashboard [User Types #975](https://github.com/kubernetes/dashboard/issues/975) + + * Dashboard [User Types and Use Cases](https://docs.google.com/document/d/1urAlgRP7AbcdsOMQ_piQQ6O1XTDIum_LmOUe8xsC4pE/edit) + + * [SIG UI weekly](/sig-ui) + +* **Notes** + + * 2018 Dashboard strategy. + + * [Deck](https://docs.google.com/presentation/d/1q1G1vWCrenI3GVsyF4d-2rpyVrdJgFW7fIbvcAAd9Lg/edit?ts=5a26f250) + + * [Github issue](https://github.com/kubernetes/dashboard/issues/2556) + + * **Kubectl access via Dashboard**. + + * Dhaval: Provide context to issues. + + * **Third-party Widgets**. + + * **Custom Views**. + + * Dhaval: Custom views will be very useful to share the event details, contextual information, and logs with specific time ranges. It will help our development teams to quickly analyze the issues. + + * Henning: UI should allow users to pick the important properties in different views/widgets. + + * **Integrations and CRDs**. + + * **Onboarding experience**. + + * Jared: Currently there is no crosslinking between docs and the UI. We can explore the concept and impact further. + + * **Design system.** + + * e.g. Standard for how a pod’s status is displayed. + + * Different use cases/personas + + * application developer + + * application operator + + * (multi-)cluster operator + + * **Feedback from Dhaval and Hennings:** + + * "We don’t use Dashboard, because of lack of authorization control." + + * "Dashboard today caters to the dev perspective, which is OK." + + * But for ops, currently missing dependency stacks (e.g. OS version) "I want to know different node versions." + + * "Our developers use Dashboard for the logs" + + * "When we perform troubleshooting & debugging we expect 30 min before and after incident. + + * Want to be able to link users to docs for more info; "What’s a pod? What’s a deployment?" + + * *Contextual docs* displayed in UI. + + * Idea: Dashboard could scrape docs. + + * Kubernetes docs working on expanding [glossary](https://kubernetes.io/docs/reference/glossary/?fundamental=true) + + * Dashboard is backwards compatible. + + * Demo of [Kubernetes Operational View](https://github.com/hjacobs/kube-ops-view) + + * Use Case: + + * Onboarding: explain resource limits vs. resource requests. + + * Quickly looking at a cluster and knowing what’s going on. + + * "I really like this UI" + + * Wanted to gamify K8s with this UI. + + * "My cluster has 118 nodes, so the ability to filter would be important." (general scale issues) + + * "Problem we’re running into is that the number of nodes will crash the browser, so we need some way to select cluster first" + + * Defining a view is hard, Dashboard shouldn’t attempt to do that. + + * Use case for custom Dashboard re-skinning. + + * "My exec looks at the look and feel." + +* Kubernetes Dashboard 2017 User Survey: [https://docs.google.com/forms/d/e/1FAIpQLScnxeub_xh7Lp4iZO1RKdTgIYK_cTwqFKv1WD-Cue4tZHcbhw/viewform?usp=sf_link](https://docs.google.com/forms/d/e/1FAIpQLScnxeub_xh7Lp4iZO1RKdTgIYK_cTwqFKv1WD-Cue4tZHcbhw/viewform?usp=sf_link) + diff --git a/events/2017/12-contributor-summit/enabling-kubernetes-ecosystem.md b/events/2017/12-contributor-summit/enabling-kubernetes-ecosystem.md new file mode 100644 index 00000000..50be4b66 --- /dev/null +++ b/events/2017/12-contributor-summit/enabling-kubernetes-ecosystem.md @@ -0,0 +1,134 @@ +# Kubernetes Ecosystem + +Notes by @directxman12 + +* How do we e.g. have an object validator + + * Went looking, found 2, second b/c first didn’t work well enough + +* How do we enable people to build tools that consume and assist with working with Kubernetes + + * Kube-fuse, validatators, etc + + * How do we make those discoverable + + * Difficult to find via GitHub search + + * Difficult to find via search engines (find blog posts, etc, and not the tools) + +* Proposal: structured content (tags, categories, search) for registering and discoverability + +* Are there examples of ecosystems that we can follow + + * Package managers (PyPI, crates.io, NPM) + + * Doesn’t quite fit one-to-one b/c some stuff is best practices, etc + + * Docs, stuff like CNI plugins, etc doesn’t actually work well the same view as things that run on Kubernetes + + * At the basic point, just need to find what’s there + + * Traditional package managers focus on consuming the packages + + * If you don’t have packaging, it’s hard to approach (see also: Freshmeat problems) + + * Hadoop: classes that map the Hadoop ecosystem, infographs + + * Wordpress, Drupal are good examples + + * Ansible Galaxy + + * Chrome Extensions/Eclipse plugins + + * Look for things tagged, if comments say "broken" and last update is old, don’t use it + + * Packagist (PHP package repository) + + * Integrates with GitHub, pulls in details about updates, README, extensibility, etc from GitHub + + * Just helps with discoverability + + * Steam + + * Has curated lists by users + +* Opinion: End users need focused, distributable bundles + + * Most people don’t need to do all of everything + + * Different systems for apps vs critical infrastructure + + * Critical infrastructure doesn’t change much + + * Still need to discover when initially building out your system + +* Issue: people get overwhelmed with choice + + * We don’t want to endorse things -- users should choose + + * We could let people rank/vote/etc + + * For example, what’s wrong with an "awesome list" + + * Need to look at human-consumable media, not necessarily machine-consumable + +* Question: do we currate + + * Do we require "discoverable" things to be maintained/use a CLA/etc? + + * Opinion: no, we can’t possibly curate everything ourselves + +* It’s problematic if you can discover new things, but they’re not supported by your Kubernetes distros + + * Not as much a problem for apps stuff, but harder for infrastructure + +* Doesn’t GitHub have labels, stars, etc + + * Yes! + + * We could just say "always label your GitHub repo with a XYZ label if you’re a CRI plugin" + + * Comes a bit back to curration to discover benefits of each + + * Enterprises, gitlab make this infeasible + +* Core infrastructure is a bit of an edge case, perhaps focus on addons like logging, monitoring, etc + + * Still comes back to: "if distro doesn’t support well, will it still work" + +* Issue: there’s things people don’t know they can choose + + * E.g. external DNS providers + +* Have a partially curated list of topics, but not curate actual content + + * Maybe leave that up to distros (have different collections of options -- e.g. open-source only, etc) + + * Have "awesome-kubernetes" + +* Let SIGs curate their lists? + + * Having all the SIG be different is difficult and confusing + + * SIG leads don’t necessarily want to be the gatekeepers + + * We don’t necessarily want to tempt SIG leads with being the gatekeepers + +* If we have something "official", people will assume that stuff is tested, even if it’s not + +* Can we just have distributions that have way more options (a la Linux distros) + + * There are currently 34 conformant distros + +* If we have ecosystem.k8s.io it’s really easy for people to find how to find things, otherwise could be hard + + * E.g. people don’t necessarily know awesome lists are a thing to search for + +* Someone should do a prototype, and then we can have the conversation + +* Question: where should this convo continue + + * SIG Apps? + + * Breakout group from SIG Apps? + diff --git a/events/2017/12-contributor-summit/extending-kubernetes.md b/events/2017/12-contributor-summit/extending-kubernetes.md new file mode 100644 index 00000000..ee21aea2 --- /dev/null +++ b/events/2017/12-contributor-summit/extending-kubernetes.md @@ -0,0 +1,215 @@ +# Extending Kubernetes +Note Taker: Clayton Coleman ([smarterclayton](https://github.com/smarterclayton)) + +* Questions + + * Do we have enough extension mechanisms? + + * See below + + * Implementing network injection that isn’t a CNI injection of some form is hard + + * e.g. adding in arbitrary network devices, for example + + * Are Flex Volumes enough? + + * Maybe? + + * Are we doing ok on kubectl extensions? + + * Yes, we’re heading in the right direction with plugins + + * Kubectl itself should be developed using its own mechanisms + + * Extension points: + + * OpenAPI metadata (operate on object w/o knowing about it) + + * Subresources (generic API-level interfaces for certain facets) + + * Plugins (git-style) + + * Can we do custom validation in kubectl? + + * Do it via admission webhooks (beta in 1.9) + + * Can run validation asynchronously, and report it (put a condition in) + + * Client-side validation is iffy + + * Should we have phased hooks? + + * Should more complex application lifecycle layer on top? + + * Are we consistent enough across our hooks to enable users to build something sane? + + * Can we do referential integrity + + * Technically, with a webhook + + * Generally not the best idea (causes issues with components that don’t expect it) + + * Can maybe do with initializers on a namespace + + * Generally go for eventual consistency (e.g. wait for the secret to exist before starting the pod) + + * Reasons + + * Surface errors to users synchronously + + * Do it on the client-side + + * Avoid dealing with the full matrix of failure modes + + * How do we not re-invent a service mesh with auth webhooks? + + * It’s an open question + + * Can we do conversions on CRDs + + * Maybe with a webhook (ongoing discussion) + + * Why aren’t Custom API servers easy enough + + * Need OpenShift certificate signing mechanisms in Kube, or similar (also exists similarly in Istio) + + * Storage + + * Re-use existing etcd + + * Use your own etcd + + * Planned backing custom API servers with CRD storage + + * Why aren’t they more uniform + + * Because they came at different times + + * It’s hard to fix them now (need more versioning) + + * Different layers have different needs + + * Declarative is API + + * Below Kubelet is gRPC + + * gRPC for KMS, CSI + + * CNI is shell execution (mainly for legacy reasons) + + * Can we make custom controllers easiers? + + * OperatorKit + + * Need better docs (being worked on) + + * Untyped vs generated typed informers and listers + + * No Go generics exists + + * Can use interfaces + + * Can use conversions (may be harder than it needs to be) + + * Can wrap in a generic type (e.g. RBAC types) + + * Generating clients for CRDs (look for stts’s blog post on generators and informers) + +* Existing extension mechanisms + + * API Extensions + + * External APIs (Aggregated APIs) + + * Custom Resources + + * Admission/Early-Phase ext points + + * Initializers + + * Can’t do in updates + + * Webhooks + + * Need to make them easier to implement (and a good example) + + * Pod Presets (ish) + + * Webhooks have to be fast, initializers are more async (normal controller style, with restraints) + + * Finalizers (late-phase ext points) + + * Flex Volumes + + * Being used to prototype container identity work, too + + * CSI (soon™) + + * (grpc) CRI + + * (binaries) CNI + + * (webhook) Auth plugins + + * Authz + + * Authn + + * Groups from authn hook ??? + + * (API server) Custom metrics API servers + + * (grpc) Device plugins + + * (grpc) KMS (Key Management) + + * (git style) kubectl plugins + + * (API usage) Any controller + + * External Cloud Provider + + * Pod lifecycle isn’t extensible + + * Object phase + + * Init containers (do exist) + + * defer containers + + * Lee’s debug container + + * Logging plugins + + * Was a proposal for logging volumes, didn’t get follow-up + +* New Extension type things + + * Enumerations in the API + + * Conditions (new condition types) + + * Loose coordination between multiple controllers + + * Signal to end users + + * Exists until formalized as a field + + * runc hooks + + * Could be a CRI concern + + * Optional CRI sub-interfaces? + + * Identity injection (custom CA certs in every pod, etc) + + * Simplifying the creation of controllers with controller generation ie. metacontroller: [https://github.com/GoogleCloudPlatform/kube-metacontroller](https://github.com/GoogleCloudPlatform/kube-metacontroller) + + * API server builder: [https://github.com/kubernetes-incubator/apiserver-builder](https://github.com/kubernetes-incubator/apiserver-builder) + + * Operator pattern toolkits: + + * Rook Team: [https://github.com/rook/operator-kit](https://github.com/rook/operator-kit) + + * GiantSwarm: [https://github.com/giantswarm/operatorkit](https://github.com/giantswarm/operatorkit) + diff --git a/events/2017/12-contributor-summit/feature-roadmap-2018.jpg b/events/2017/12-contributor-summit/feature-roadmap-2018.jpg Binary files differnew file mode 100644 index 00000000..6b69204a --- /dev/null +++ b/events/2017/12-contributor-summit/feature-roadmap-2018.jpg diff --git a/events/2017/12-contributor-summit/feature-roadmap-2018.md b/events/2017/12-contributor-summit/feature-roadmap-2018.md new file mode 100644 index 00000000..d2ae01ac --- /dev/null +++ b/events/2017/12-contributor-summit/feature-roadmap-2018.md @@ -0,0 +1,336 @@ +Contributor summit - Kubecon 2017 + +**@AUTHORS - CONNOR DOYLE** + +**@SLIDE AUTHORS ARE THE PRESENTERS** + +**@SKETCHNOTES AUTHOR DAN ROMLEIN** + +# 2018 features and roadmap update + +Presenters: Jaice, Aparna, Ihor, Craig, Caleb + +Slidedeck: https://docs.google.com/presentation/d/10AcxtnYFT9Btg_oTV4yWGNRZy41BKK9OjK3rjwtje0g/edit?usp=sharing + +What is SIG PM: the "periscope" of Kubernetes + +* They look out for what’s next, translate what’s going on in the community to what that means for Kubernetes + +* Responsible for Roadmap and Features Process + +Understanding the release notes is difficult sometimes since the docs aren't always done by the time the release team is compiling the notes. + +## Retrospective on 2017 + +* What did we do since Seattle + +* Feedback on how we can enhance SIG-PM + +* Moving from "product-driven" to "project-driven" where SIGs are defining their roadmaps based on market, end-use input etc. + +Major 2017 Features: + +* (Apps) Workloads API to Stable + +* (Scalability) 5k nodes in a single cluster + +* (Networking) NetworkPolicy + +* (Node) CRI, GPU support + +* (Auth) RBAC stable + +* (Cloud providers) Out of core + +* (Cluster Lifecycle) kubeadm enhancements - road to GA in 2018? + +* (Autoscaler & instrumentation): custom metrics for HPA + +* (Storage) CSI (Container Storage Interface) + +How did we do on the contributor roadmap? + +* [See 2017 roadmap slides](https://docs.google.com/presentation/d/1GkDV6suQ3IhDUIMLtOown5DidD5uZuec4w6H8ve7XVo/edit) + +Audience Feedback: + +* Liked that it felt like there was a common vision or theme last year. + +* Liked that there was a PM rep saying "do docs", "your tests are failing" etc + +* Leadership summit 6 months ago: shot heard called was "stability releases, stability releases". Not sure that 1.8 was really a stability release, not sure 1.9 is either. Will 2018 be the year of stability (on the desktop) + + * (Brian Grant) Come to the feature branch session! + + * Notion of stability needs to be captured as a feature or roadmap item to talk and brag about. Quantify, talk about as roadmap item + + * Idea for 2018: how do you measure and gamify stability? See a number in the red, people will want to fix. Testing, code quality, other metrics - might improve stability organically + + * Context of stability and testing: achievement was conformance program. 30+ conformant distros! + + * Want to see project continue to be useful: within your SIG, invest in conformance, extending suite. Going back to what is and is not k8s - define core, extensions: don't compromise the stability of the core. + + * Please come see Eric Tune and define stability : + + * "cluster crashed" + + * "too many features" + + * "an API I was using stopped working" + + * "community is chaotic, how do I navigate that" + +* There are many new alpha features in each new release. Please prioritize graduating and stabilizing the existing features. (More than 50 APIs already) + +* Looking for volunteers on writing a stability proposal? + +* Jaice has one already! + + * *May* have broken the comment limit on Google Docs + + * Need to define lens: + + * architecture, community etc; look at a proposal under each. + + * Brian is working on arch stability. + + * Contribex is looking at mentorship and ladder. + + * Myriad of ways to approach problem set. How do we mature the processes of the Kubernetes ecosystem? + +* Looking for co-authors? + +## Proposals and KEPs + +(Caleb) + +Please hang out in SIG-Arch, SIG-Contribex, SIG-PM to drive this process forward + +Looking at "is this feature ready to move to the next stability milestone?" (Alpha to Beta, Beta to GA etc) + +* Proposals are now **K**ubernetes **E**nhancement **P**roposals + + * Piece-by-piece over on more multiple releases (living documents) + + * Looked at a lot of other open source projects, e.g. the Rust community (RFC Process) + + * designed in the era of GitHub; decided on a lightweight process that works well with the VCS. + +* Talk about what we want to do without a long spec doc, about what we agreed to ahead of time, but* don't want to diverge 2 years later*. + +* Helps tracking individual features (easier to read release notes) + + * Release note writing take tracking down a lot of docs, GitHUb issues, design docs, Slack and Google Group comments; combine from a bunch of places + + * Hard to tell from the release notes what's important and what's a minor detail. + +* Every SIG should set their own roadmap - the KEP proposal enables that. + +* Template that asks you to think ahead for the lifecycle of the feature; let people know what you're planning to do. + + * It's a medium for comms; not saying "It has to be done this way" but saying why this is important. + + * Inspired by "[Toward Go 2](https://blog.golang.org/toward-go2)" blog post by rsc + +* Has been tested - [draft KEP for Cloud Providers](https://github.com/kubernetes/community/pull/1455), Tim St. Clair has tested. + +* Want to make easier for new contributors to write KEPs. + +* Starting with "what is a unit of work, how do people care" + +Questions: + +* Are KEPS google docs, or pull requests, etc? How do you submit one? + + * Original intend: something that lives in source control. Discoverable like any part of the code. Attempt to combine design proposals and umbrella GitHub issues, link to 10s of other issues. They will live as long as we're a project; doesn't depend on hosting providers. + + * Vision is that writing KEPs, know from them what the roadmap is; can write the blog post based on the value articulated in the KEPs. + + * Right now they are [buried in community repo](/contributors/design-proposals/architecture/0000-kep-template.md) 3 levels deep: a lot of great feedback so far. Joe wouldn't object to making it more discoverable. + + * Kep.k8s.io for rendered versions? + + * Rust community has a sub-repo just for this (rust-lang/rfcs) + + * More than one person has said that KEPs weren't known about - move to somewhere discoverable sooner rather than later. Who can own that action? Matt Farina from Samsung is keen to help but doesn't have resources to lead. + + * *Can only have its own repo if we get rid of features repo!* + + * Don't want anyone to do anything not adding value to work; hope is that KEP is worthwhile and adds value. Caleb will help drive, and create repos as needed. + +## Features + +(Jaice) + +Features repo and process: I will say what I'm not happy about and what I've heard, but I do want to say there has been so much work from Ihor and SIG-PM to get where we are. If you haven't seen the machinations of feature/release product you won't know! + + +At velocity we have outgrown the notion of a central body leading. Seeing increasing cross-SIG concern where some SIGs rely on others to get their work done. + +We need to find ways to detangle those dependencies earlier. + +KEPs are focusing on enhancements, changes in ecosystem. A feature has a connotation of being user facing or interactive. KEP can be for something contributor facing, doc facing transparent. Get out of mindset we're delivering a "thing" - we're delivering value to user, contributor community, or both. + +Currently the process is cumbersome. We want to follow agile principles about local control, local understanding, sharing amongst teams. + +### The steel thread + +KEP provides opportunity for "steel thread" of custody. A KEP, as a living Markdown doc with metadata, lets you see high level idea at any group of time. A SIG can break this down into meltable ideas for each release. A KEP can last multiple milestones. In terms of features/issues, defined by SIGs, issue should be actionable. If a SIG writes an issue for that release milestone it should be approachable by any contributor who is a SIG member. + +PRs roll up to the issue: links to this issue, which links to this KEP. For the first time we would be able to see at the PR level, what the grand scheme behind everything we do, is. Not heavyweight - just linking of issues. Limit administrivia, paperwork to delivery value. + +Q: + +* For CloudProvider - discovery, had an idea, people started digging in; how do you keep updating the KEP? Do you update as you go, you odn' tknow what you need? + + * Yes, it's a living doc first and foremost. Has metadata (alpha, beta, GA). The issues worked on per milestone - say you pick 3, you document those issues; if during the course of the PRs to complete those issues you realise it's a misstep, you close the issue, you update the KEP, say you're modifying this; look at prior versions to see what it looked like before/what it looks like now. As you move to next release milestone the issues reflect that change: in terms of KEP, issues, planning. + +* Give an analysis on why issues in feature repo didn't solve this problem and why KEPs will? + + * From features standpoint: SIGs interacted with features only when they had to. By trying to keep all that info separate from PRs, issues in each milestone: no way to tie that work back to the feature issue. No way to easily understand where in this features repo issue, long if multi-milestone: where was work, and the link to the issue? Eliminate the work + + * KEP is not solving that problem. KEP is saying "how do we best define and discuss the value we're adding to the ecosystem". Learns from patterns and antipatterns of other communities. + + * Features process is central body of people not doing the work. + + * Some friction with the GitHub mechanics of issues, relabeling quarterly, etc; would be nice to keep pace with the number of issues. Moving to files to provide value will help make that clearer. + + * Managing issues in the features repo: hundreds of issues from the last year, created spreadsheets etc. Bringing value but consuming extra resources. Not synced with features repo. Good we have a spreadsheet, but difficult to manage. + + * Started going through this process in SIG-Cluster LIfecycle: replacing GitHub thread (no one updates top) with a living document (use git history for mutations; see concise summary) - think it will be a huge improvement. + + * KEPs are in Google Docs now, haven't converted to PRs + +* Does this mean the features repo is going away? + + * "Yes, eventually". + +* We need clear communication about where we are in this process. Are KEPs required, encouraged, etc? Trying with one project, be useful to know what the granularity is. + + * "It's in Alpha now" - trying to validate some assumptions. + + * Has momentum with SIG-Arch and SIG-PM + + * Assumption we will try and see if it works. + + * Want to steward people into trying it to see if it's valuable. If not, how do we make it so? + + * Community size and velocity now makes it difficult to radiate information to get what you need to know. + + * Kubernetes 1.10 would be a great place. + +* Will it still be true in 1.10 that you need features? Can you have KEP instead of a Feature issue? Make it very obvious for 1.10 please. + + * One thing we need to sort out + + * 1.10 will have existing features process + + * Cluster Lifecycle are experimenting with how to do differently + + * Would love to see SIGs try the KEP process but it's not a full surrogate for the features process yet. + +* Existing features can be sorted, filtered etc - will there be tooling around KEPs? Until that's ready, I wouldn't want to switch + + * Proposals are different to features: issues will be associated with a KEP. In some ways it will be better as it's in the repo itself. + +* How are the issues tied to the KEP? + + * Ties to this markdown file, in /kep/____ + + * We need to work out the details. + + * When I create issue template, it will have a link to that KEP. + + * *Make sure it's searchable!* + +* (Jago Macleod) Be explicit of scope of a KEP. A trend to more SIG ownership, code, roadmap etc; context is a SIG plans their own work. Certain faster moving SIGs, more capacity, to solve problems in the wrong component. Some KEPs will span SIG boundaries and should be explicit about this in comms. + +* (Tim Hockin) GitHub UX is not going to be great for this. It's not built for what we're trying to do. All of this predicates on having an actual site that manages, a web app? + +If that's true, can we start to collect the expertise? + +Is there a SaaS tool for this? + + * See other projects using tracking tooling, used on files in source control + + * Won't be pretty or searchable in first iteration. + + * Want to see built out as they're consumed. Don't build tooling before people are using the process + + * *GH search is less than usable* + + * Do you have a place to host this? We will find a place to host if someone is willing to write. Matt will help write. + + * As someone involved with Maven repositories (trello) - it's a total pain in the neck. More in favour of PRs, discoverability, etc. Keep PRs for process. Push a site for discoverability. + +* (Joe Beda) Current processes are discussion, argument, write-only: comment threads no-one reads. Make it more discoverable, more readable: check stuff into GitHub, use a static site generator, publishes it somehow, crosslinks between documents. Just like when you go to an RFC, reference one there's a link; that's the ecosystem of discoverability we want. + + * If someone comes to project and we have no way of telling what's happening. + +* (Eric Tune) I created features process so I'm a little attached. This discussion is a refactor vs rewrite debate. The feature repo template is there; in GitHub; change it. See how many incremental changes we can make to solve proposals. Put fields there and yell at people who don't fill it in. + +* (Angus, Bitnami) As someone involved in Rust community: + + * fairly well-read Rust weekly news summary, including a section with a summary on outstanding proposals, broken down into early and late discussion phases. Read late phase to see what's a done deal vs. what's up in the air. There's also a RFC podcast, where they get a couple of people, have a chat show about what's involved in that proposal. Lots of ways as a community member to stay up to date. + + * Fixed timeline: trying to get approval. If nothing happens by the end it's approved by default. + + * May not want to copy but good to know. + +* (Daniel Smith, Google) summarising: problems are mostly discoverability: we'll write tooling. Why can't we write tooling against that existing process? Both current and hypothetical new process suffer + + * Interacting with objects in GitHub is hard at our scale, we have a limited number of tokens. + + * Procedural aspects: if I look at a PR, there's a link to an issue: that will tie up to a KEP. + + * Discoverability of relationships: exposed in the API. Links on GitHub are implemented by full-text search, not hard to do this. + +* (Chen Goldberg) Schedule follow-up sometime this week - communicating is hard, we're all here! + +* (Henning, Zalando) Structured format with some metadata: all in one repo, or distributed amongst incubator projects etc? Are KEPs with a unique identity going to live in other project repos? Was this an idea? + + * Idea is to have as consolidated as possible: given committee questions of "what is Kubernetes, what follows our PM norms". Problem is visibility for people who trust workloads in the software we create. Want to provide this so someone can determine why a thing was done a certain way. Projects outside the core repo: would follow a similar process in a perfect world. + +If you're not excited about features process, try a KEP! + +Reach out to SIG-PM - calebamiles@ + +### The long view + +(Jaice) Have planning sessions about what SIGs are hoping to deliver for any milestone. Want to facility planning meetings in a more structured way (Planning as a Service). I did this for a proposal SIG-Cluster Lifecycle are working on: to have the discussion early and untangle dependencies, see when things could go off the rails, etc. Will talk to SIG leaders. + +This planning activity will be one of the key success factors for the project moving forward. + +Roadmap for 2018 (30min summary) +-------------------------- + +Notes by @jberkus + +Speakers (check spelling): Apprena Singha, Igor, Jaice DuMars, Caleb Miles, someone I didn't get. SIG-PM + +We have the roadmap, and we have this thing called the features process, which some of you may (not) love. And then we write a blog post, because the release notes are long and most of the world doesn't understand them. + +Went over SIG-PM mission. We had several changes in how the community behave over 2017. We are moving to a model where SIGs decide what they're going to do instead of overall product decisions. + +2017 Major Features listed (workloads API, scalability, networkpolicy, CRI, GPU support, etc.). See slides. The question is, how did we do following the 2017 roadmap? + +Last year, we got together and each SIG put together a roadmap. In your SIG, you can put together an evaluation of how close we came to what was planned. + +Q: Last year we kept hearing about stability releases. But I'm not sure that either 1.8 or 1.9 was a "stability release". Will 2018 be the "year of the stability release?" + +Q: Somehow the idea of stability needs to be captured as a feature or roadmap item. + +Q: More clearly defining what is in/out of Kubernetes will help stability. + +Q: What do we mean by stability? Crashing, API churn, too many new features to track, community chaos? + +Q: Maybe the idea for 2018 is to just measure stability. Maybe we should gamify it a bit. + +Q: The idea is to make existing interfaces and features easy to use for our users and stable. In SIG-Apps we decided to limit new features to focus everything on the workloads API. + +Proposals are now KEPs (Kubernetes Enhancement Proposals) are a way to catalog major initiatives. KEPs are big picture items that get implemented in stages. This idea is based partly on how the Rust project organizes changes. Every SIG needs to set their own roadmap, so the KEP is just a template so that SIGs can plan ahead to the completion of the feature and SIG-PM and coordinate with other SIGs. + +Q: How do you submit a KEP? +A: It should live in source control. Each KEP will releate to dozens or hundreds of issues, we need to preserve that as history. + +If you look at the community repo, there's a draft KEP template in process. We need to make it a discoverable doc. diff --git a/events/2017/12-contributor-summit/feature-workflow.md b/events/2017/12-contributor-summit/feature-workflow.md new file mode 100644 index 00000000..8bb0fa47 --- /dev/null +++ b/events/2017/12-contributor-summit/feature-workflow.md @@ -0,0 +1,85 @@ + +Feature Workflow +---------------- + +Notes by @jberkus + +TSC: Getting something done in Kubernetes is byzantine. You need to know someone, who to ask, where to go. If you aren't already involved in the Kubernetes community, it's really hard to get involved. Vendors don't know where to go. + +Jeremy: we had to watch the bug tracker to figure out what sig owned the thing we wanted to change. + +TSC: so you create a proposal. But then what? Who needs to buy-in for the feature to get approved? + +Dhawal: maybe if it's in the right form, SIGs should be required to look at it. + +Robert B: are we talking about users or developers? Are we talking about people who will build features or people who want to request features? + +???: Routing people to the correct SIG is the first hurdle. You have to get the attention of a SIG to do anything. Anybody can speak in a SIG meeting, but ideas do get shot down. + +Caleb: we've had some success in the release process onboarding people to the right SIG. Maybe this is a model. The roles on the release team are documented. + +Anthony: as a release team, we get scope from the SIGs. The SIGs could come up with ideas for feature requests/improvement. + +Tim: there's a priority sort, different projects have different priorities for developers. You need a buddy in the sig. + +Clayton: review bandwidth is a problem. Review buddies hasn't really worked. If you have buy-in but no bandwidth, do you really have buy-in? + +TSC: The KEP has owners, you could have a reviewer field and designate a reviewer. But there's still a bandwidth problem. + +Dhawal: many SIG meetings aren't really traceable because they're video meetings. Stuff in Issues/PRs are much more referencable for new contributors. If the feature is not searchable, then it's not available for anyone to check. If it is going to a SIG, then you need to update the issue, and summarize the discussions in the SIG. + +TSC: Just because a feature is assigned to a SIG doesn't mean they'll acutally look at it. SIGs have their own priorities. There's so many issues in the backlog, nobody can deal with it. My search for sig/scheduling is 10 different searches to find all of the sig/scheduling issues. SIG labels aren't always applied. And then you have to prioritize the list. + +???: Test plans also seem to be late in the game. This could be part of the KEP process. And user-facing-documentation. + +Robert B: but then there's a thousand comments. the KEP proposal is better. + +???: The KEP process could be way to heavy-weight for new contributors. + +???: new contributors should not be starting on major features. The mentoring process should take them through minor contributions. We have approximately 200 full-time contributors. We need to make those people more effective. + +TSC: even if you're a full timer, it's hard to get things in and get a reviewer. Every release, just about everything that it's p0 or p1 gets cut, because the person working on it can't get the reviewer all of the stuff lined up. + +Caleb: you need to spend some time in the project before you can make things work. + +Dhawal: is there a way to measure contributor hours? Are people not getting to things because people are overcommitting? + +Jago: The problem is that the same people who are on the hook for the complicated features are the people who you need to review your complicated feature. Googlers who work on this are trying to spread out their own projects to that they have more time at the end of the review cycle. + +Jaice: If you're talking about a feature, and you can't get anyone to talk about it, either the right people aren't in the room, or there just aren't enough people to make it happen. If we do "just enough" planning to decide what we can do and not do, then we'll waste a lot less effort. We need to know what a SIG's "velocity" is. + +Connor: the act of acquiring a shepard is itself subject to nepotism. You have to know the right people. We need a "hopper" for shepherding. + +Tim: not every contributor is equal, some contributors require a lot more effort than others. + +Robert: A "hopper" would circumvent the priority process. + +Josh: there will always be more submitters than reviewers. We've had this issue in Postgres forever. The important thing is to have a written, transparent process so that when things get rejected it's clear why. Even if it's "sorry, the SIG is super-busy and we just can't pay attention right now." + +Dhawal: there needs to be a ladder. The contributor ladder. + +TSC: a lot of folks who work on Kube are a "volunteer army." A lot of folks aren't full-time. + +Caleb: there is a ladder. People need to work hard on replacing themselves, so that they're not stuck doing the same thing all the time. How do you scale trust? + +???: Kubernetes is a complicated system, and not enough is written down, and a lot of what's there we'd like to change. It's a lot easier for a googler to help another googler, because they're in the same office, and the priorities alighn. That's much harder to do across organizations, because maybe my company doesn't care about the VMWare provider. + +Jaice: for the ladder, is there any notion that in order to assend the ladder you have to have a certain number of people you shepherded in? There should be. + +TSC: frankly, mentoring people is more important than writing code. We need to bring more people into Kubernetes in order to scale the community. + +Josh: we need the process to be documented, for minor features and major ones. Maybe the minor feature process belongs to each SIG. + +Jaice: the KEP is not feature documentation, it's process documentation for any major change. It breaks down into multiple features and issues. + +???: The KEP needs to include who the shepherds should be. + +Clayton: reviewer time is the critical resource. The prioritization process needs to allocate that earlier to waste less. + +Jeremy: the people we sell to are having problems we can't satisfy in Kubernetes. We have a document for a new feature, but we need every SIG to look at it (multi-network). This definitely needs a KEP, but is a KEP enough? We've probably done too much talking. + +Clayton: the conceptual load on this is so high that people are afraid of it. This may be beyond what we can do in the feature horizon. It's almost like breaking up the monolith. + +Robert: even small changes you need buy-in across SIGs. Big changes are worse. + +Connor: working groups are one way to tackle some of these big features. diff --git a/events/2017/12-contributor-summit/kubernetes-client-libraries-open-space.md b/events/2017/12-contributor-summit/kubernetes-client-libraries-open-space.md new file mode 100644 index 00000000..d23327d2 --- /dev/null +++ b/events/2017/12-contributor-summit/kubernetes-client-libraries-open-space.md @@ -0,0 +1,68 @@ +_Notes from the open space discussion at the Kubernetes Contributor Summit 2017 lead by @brendandburns_ + +Three high-level topics: + +* Client libraries and autogeneration +* OpenAPI description status +* Fluent libraries (ie. those with native sensibilities) - is this scalable given need for lots of hand-written code? + +Early shout-out for the [Common LISP client](https://github.com/brendandburns/cl-k8s) :) + +Currently Java, Python, .Net, Javascript clients are all silver. Reference to the [badge and capabilities descriptions](/contributors/design-proposals/api-machinery/csi-new-client-library-procedure.md#client-capabilities) +Python still outside the [clients org](https://github.com/kubernetes-client/) - should this move from incubator? + +Lots of code in client libraries to handle limitations in OpenAPI 2, some of which is fixed in 3. What is required/should we move to OpenAPI 3? + +The Go client is a special case, predates the swagger work and bypasses the autogeneration. This leads to it being an odd first-class citizen. + +Noted that currently client libraries generated against specific version of API +> “I can’t currently write a library which supports multiple versions of the API from the same client” + +Discussion of packaging. Should clients be packaged as one thing (with support for multiple versions in subpackages) or with each package representing support for a specific version? + +Autogeneration is desirable, given the wide number of languages. But what should be the canonical description. Noted that +go-restful, which is currently in use, (starts from Go) vs go-swagger (which starts from an OpenAPI description), worthwhile discussing which mode would work best now. + +Would it be possible to annotate the Go types to avoid some problems in the generator code? + +The API documentation is currently poor, mainly non-language specific API documentation. + +What kind of examples do we expect for each client? It would be great to create a set of canonical usecases for clients and for each of them to have (maybe autogenerated) examples for each. + +There are some Client Guidelines, @mbohlool would have a link? + +Note that the websockets protocol is under-documented. Kubernetes Java client has a description of the protocol and is the best place to look for details. + +SPDY still in use. Should this be deprecated? + +Question about what additional clients, or improvements, people would like to see: +* A smaller Go client +* Rust +* C++ + +Informers, should this be part of the API and available so as to be easier to use from other languages? This is not currently part of the client capabilities set mentioned above. We’re probably better off building a generic multiwatch controller than supporting the informers in all other client libraries. + +Service Catalogue and other extension points. Advice is to use the generator tools to generate your own clients. See [kubernetes-client/gen](https://github.com/kubernetes-client/gen) + +There is a plan to split the go client in to two - one hand rolled, one autogenerated. + +API supports proto as a wire format, but client generation here is hard. The protocol doesn’t know the path to which it needs to be sent. Protos are also generated from the Go structs. + +At least have a common language for description APIs. Proto vs Go. + +Worth noting that kubernetes-client is actually relatively low barrier to entry for getting started contributing to Kubernetes. We should talk about that more publicly. + +Discussion of issues with kubeconfig. + +* There is no spec for kubeconfig? +* One proposal, move folks to kubeconfig.d (ie. one config file per cluster) +* No clients support (apart from Go) support merging kubeconfigs. +* Problem with changes here is the large number of kubeconfigs in existence. +* Can we have a new config file format (v2) which exists alongside the existing one. + +Noting that kubeconfig contains several things, which arguably shouldn't be as closely coupled. + +* Identities +* List of clusters +* Namespace +* Current context diff --git a/events/2017/12-contributor-summit/onboarding-new-developers-through-better-docs.md b/events/2017/12-contributor-summit/onboarding-new-developers-through-better-docs.md new file mode 100644 index 00000000..ddb6d688 --- /dev/null +++ b/events/2017/12-contributor-summit/onboarding-new-developers-through-better-docs.md @@ -0,0 +1,232 @@ +Onboarding Developers through Better Documentation + +@ K8s Contributor Summit, 12/5 + +Note Taker: Jared Bhatti ([jaredbhatti](https://github.com/jaredbhatti)), Andrew Chen ([chenopis](https://github.com/chenopis)), Solly Ross ([directxman12](https://github.com/DirectXMan12)) + +[[TOC]] + +### Goal + +* Understand how we’re currently onboarding developers now + +* Understand the "rough edges" of the onboarding experience for new users + +* Understand how we can better target the needs of our audience + +* Understand if our contributor process will meet the documentation needs. + +Note: the focus is on documentation, so we won’t be able to act on suggestions outside of that (but we will consider them!). + +### Initial thoughts + +* Where we were at the beginning of the 2017: ([@1.4](https://v1-4.docs.kubernetes.io/docs/)) + +* Where we are now: ([@ 1.8](https://kubernetes.io/docs/home/)) + + * We are tied into the launch process + + * We have analytics on our docs + + * We have a full time writer and sig docs contrib group + + * Everything is in templates, we have a style guide + + * Better infrastructure (netlify) + +### Meeting Notes + +* Introductions + + * Jared: SIG docs maintainer + + * Andrew: SIG docs maintainer, techwriter + + * Radika: 1.8 release team, SIG docs member + + * Phil Wittrock: interested b/c it determines how system is used + + * ?? + + * Paul Morie: interested b/c hit challenges that could be avoided with better dev docs (patterns used in Kube can’t be easily found on internet) + + * Nikita: work on CRDs + + * ??: docs are a good intro to core Kube codebase + + * Michael Rubin + + * ??: dashboard contributor, onboarding is a key part of the dashboard experience + + * Steve Wong: storage SIG, on-prem SIG, trying to get head around on-prem, docs geared towards cloud + + * Solly Ross (@directxman12): want to avoid changes to fiddly bits confusing people + + * Morgan Bauer: had issues with people changing fundamental bits out from under other contributors + + * Sahdev Zala: docs are important for new contributors, first place they look, contribex + + * Brad Topol: like explaining stuff, kube conformance WG + + * Tomas: working on side project, looking for inspiration on docs for that + + * Garrett: ContribEx, looking to reduce the question "how do I get started" + + * Stefan (@sttts): where do we push docs changes to mechanics + + * Josh Berkus: ContribEx + +* contributor docs + + * Avoid the process for finding which of the 49 people have the right info 6 months after it merges + + * Have contributor docs which merge with "internal features" (e.g. API registration changes) + + * Have a flag for internal release notes, bot which checks + +* Issue: opinion: we don’t want to write dev docs because they’ll change + + * Issue is it’s hard to capture every little thing + + * Good start is documentation on generalities (80%/20%): e.g building an API server + + * Started out with no docs on how to do that (has gotten better) + +* Big concepts don’t change a lot, describe those + + * Ask questions: + + * What are the big nouns + + * What are the big verbs for those nouns + + * How are the nouns relate + + * Don’t fall into trap of (for example) tracepoints (accidentally creating a lot of small little fiddly points) + +* Need a way for "heads up this is changing". If you have questions, ask these specific people. + + * Very SIG dependent (some do meetings, others mailing-list focused, some have a low bus number) + +* How is your SIG interacting with docs + + * Service Catalog + + * In docs folder + + * How do we interact with docs + + * Depends on moving into kubernetes org + +* Issue: lots of docs with bits and pieces + + * Not organized into "as a developer, you’re interested in these links" + + * Spend a week, come up with a ToC + + * Many docs in different repos + + * Should have guidelines on where to put things + + * Should have templates for how to write different types of docs + +* Issue: wrong abstraction layers are exposed + + * Documenting a complex system has less payoff than simplifying + +* Issue: extending Kube blurs the line between contributor and user + + * Put more docs into the same place + + * "Crafting User Journeys": user stories buckets: Customer User Journeys ([project](https://github.com/kubernetes/website/projects/5)) + + * Need champion for different personas + +* Issue: people (e.g. Storage SIG) want to write docs, but they don’t know where to start + + * Both contributors and users, and both (plugin writing, for example, or how to provision storage securely) + + * Templates/guidelines mentioned above may be helpful + + * Talk to SIG docs (SIG docs wants to try and send delagates to SIG, but for now, come to meetings) + +* Question: should there even *be* a split between user and contributor docs? + + * No, but we have historically (didn’t used to have person-power to implement it) + +* Onboarding docs push for 2018 + + * Point people from moment one to start understanding how to contribute (design patterns, etc) + + * How do we organize, make things findable + + * How do we organize allowing people to contribute right now + + * Main site is all user docs right now + + * Starting to look at auto-import docs + + * User journeys project (above) + + * Testing right now + +* Lots of people writing blog posts + + * Any way to aggregate to some of that? + + * Wow to avoid "this doesn’t work anymore"? + + * There is a Kubernetes blog, being migrated under k8s.io + + * People can contribute to the blog for ephemeral stuff + + * We guarantee that k8s.io stuff is updated + + * For blogs, put a "sell-by date" + +* Request: join dashboard conversation + + * How do we link from dashboard to docs and vice-versa + +* Question: where do the docs of tools like kops fit in + + * More broadly hits on Kubernetes "kernel" vs wrapper projects + + * Right now, it’s checked into repo, so it should have docs on the main site + + * Documentation is endorsing, at bit + + * (but why is kops checked into the repo as opposed to other installers) + + * Have a easy way for projects (incubator and main org) to have their own docs sub-sites (e.g. gh-pages-style for the Kubernetes sites) + +### Follow-Ups + +* Suggestion: Presentation on how to structure site, write docs, user studies done, etc (why are Kube docs the way they are) + + * "I want to get started fast" + + * "I want to build the next thing that gives me value" (task-driven) + + * "Something broke, where’s the reference docs" (troubleshooting) + +### Projects In Development + +* Customer User Journeys ([project](https://github.com/kubernetes/website/projects/5)) + +* Revamp of [docs landing page](https://kubernetes.io/docs/home/) + +* Building out a glossary ([project](https://github.com/kubernetes/website/issues)) + +* More doc sprints ([like this one](https://docs.google.com/document/d/1Ar4ploza6zA1JF3YO4e0lzq1RRBWWx8cAZtRQMMEdsw/edit)) + +### Continue participating! + +* Content exists under [Kubernetes/website](https://github.com/kubernetes/website) (feel free to fix an issue by [creating a PR](https://kubernetes.io/docs/home/contribute/create-pull-request/)) + +* Join the [kubernetes SIG Docs group](https://groups.google.com/forum/#!forum/kubernetes-sig-docs) + +* Attend the next Kubernetes docs community meeting (Tuesdays @ 10:30am, PST) + +* And join the kubernetes sig-docs slack channel ([kubernetes.slack.com](http://kubernetes.slack.com/) #sig-docs) + diff --git a/events/2017/12-contributor-summit/role-of-sig-lead.md b/events/2017/12-contributor-summit/role-of-sig-lead.md new file mode 100644 index 00000000..54e3d26f --- /dev/null +++ b/events/2017/12-contributor-summit/role-of-sig-lead.md @@ -0,0 +1,110 @@ +# What is the role of a sig lead +*Notetaker: @MHBauer* +Lots of sig leads are present in the room + +If there is a doc it’s years out of date + +Joe wants to redefine the session +Power and role of sig leads is defined in how we pick these people. Voting? Membership? What do we do for governance in general + + +Paul: Roles that are valuable that are not sig lead. Svc cat, moderator of discussions and took notes. Secratary of committee. + +Back to joe; what does the sig think. Each leader doesn’t know. Who shows up to a meeting isn’t helpful as not everyone can attend. What does the lead thing? Allows too much personal power. Decision maker or tech lead? + +Meeting form last year is that sig lead was facilitator. Have been more tech leads now. Sig leads got invites to the contributor summit. + +Erik from testing. Failures in tests associated with a sig lead to no accountability by a sig member. Assigned the label to the sig with no action afterwards. Need accountability from the sigs or the leads to get testing failures cleared + +Owners files, maintainers, approvers, of particular parts of the code base. As a sig lead. Mostly doing facilitating, and not touching the code. Owners files own the codebase. Not the leads of the sigs. Sig leads elevate problems to the people in the owners files. + +Daniiel smith - api - would not be helpful to have a facilitator, mostly very technical details. + +Paul, tech vs facilitator fulls you in different directions. Facilitator with an agenda is a dictator. If you have a tech agenda, you should not try to lead all the meetings. Meeting facilitator role that aaron did, operated the meeting agenda. Hands to talk. Stopped advocating for techical topics, and mainly doing moderation. Everyone else focuses on technical topics. + +Joe - 3 roles, process facilitatior, technical facilictator, techincal design and lead. Implication that the process leader is not technical. Build consensus vs “we’re doing it this way” + +Down - on sig node. Facilitatior and tech lead. Detach from. If she doesn’s make the agenda there is not one. + +Joe -need someone who cares + +Eric chang - every sig needs a person doing X,Y, Z. docs, tests, etc. fundamental goals. Sig lead has too much to do. Formalize roles and fill them with people, especially different people for each role. +Large sigs have more people to allocate to alt roles. + +Jessie - Can’t have the person who facilitates be unbiased techincally. Best solution vs alternative monetary gain. + +Joe - people as sig leads in name only to game the system as being important + +Erik - adding or retiring members. + +Do any sigs have an election process? Not that we know of. Sig node has a rotation. One member retired as they moved from working on the open source to working on their internal system. Some people turned down a lead, based on the involvememt necessary. + +Sig leads should already should be fulfilling the role, vs assigning titles and duties to expect + +Define roles, and all sigs must provide names. + +Document process for how to choose role. Fair and equitable. A generic process that’s generic. + +People in the sig decide. Who’s in the sig? How do we say a sig or sig member is doing a bad job? + +How to decide who gets to decide? + +Which things does a sig own. People own things, not sigs. With a document proposed to assess new owners. Subteams and persons. + +People who have non-code duties, but are also still participants and lead-worthy. + + Does sig-pm write any code? + +Long term sigs own artifacts. They own process and have long term documentation. + +Things not in kubernetes/kubernetes don’t have necessarily an ownersfile. + +Everything should role up to a sig. + +Owners cut across a project and end up having a person with lots of power vs a sig. + +Sigs and humans as owners. + +Erik f - Tests assigned to owners seems to be a fialure. A sig having a dashboard seems successful. The sig being responsible seems successful, not a person responsible. Things assigned to sigs, not to people. + +Suggestion of a person on a single sig. Limits. Pick what you find important, commit to it. +Might be career limiting if you can olnly work on one thing. +Doubts of same person leading multiple sigs. +Leading a sig vs participating. +Multiple roles again. + +Sig lead, to facilitate, sig pm representative to put technical demands and objectives to the wider community. +What to do when they fail? +Separation of powers. Do we have it now? Are the people who lead a meeting the people who code and merge and approve. + +Joe - gut check summary +Set of roles to nominate +Clear lines of sight which owns what code +Some are large and may have subdivisions (do people own code or do groups) +Starting point for sigs +Everyone should document the process and how it got that way +Possible future date to reboot the sigs. Evolution vs revolution. Merge and split. +Sig lead retired in frvor of multiple roles. “Lead” creates perverse incentives. Roles with terrible titles. POLITICS! + +Find a set of roles. + +What’s the bar to form a sig? +If I want to have a component, I need a sig! + +Retire and merge some sigs. Sig for each cloud and a common cloudprovider and a cross-cloud sig. +Cluster-lifecycle plus cluster-ops + +Daniel - avoid selecting process followers, not for decision making skills. Avoid over specifying. + +Joe - We need to expect some responsibility from the sig to the rest of the project. Avoid overfit. + +Associate owners files with sigs, maintainers vs sig leads. Contributor ladder is disorganized. Owners contributors maintainers. + +4 minutes. +Work with the steering committee to work this out. Steeering committee has to do this to help this make. + +Hard to participate in some sigs, +Suggestion of office hours + +Spotty communication of what a sig is doing. Authority and responsibility of what to communicate what a sig is doing. + diff --git a/events/2017/12-contributor-summit/scaling-new-contributors.md b/events/2017/12-contributor-summit/scaling-new-contributors.md new file mode 100644 index 00000000..925e18ac --- /dev/null +++ b/events/2017/12-contributor-summit/scaling-new-contributors.md @@ -0,0 +1,61 @@ +# Scaling New Contributors +*Lead: Vishnu Kannan* +*Notetaker: Chris Love* + +New pull requests are plateauing in k/k +Takes much longer for a PR to get through + +Contributor Experience +3 mo for PR merged - minor PR that was a fix +Nobody reviewed - was ignored +PRs are lacking process to get them on milestones +Lack of organization and understanding PR process +There is a need for a sig buyin +We have a explicate state machine, instead of a implicate state machine +We do have a well defined process for new contributors to follow +We need to have committed resources to a PR, +File an issue first, then file a PR +When the needs sig is there, the issue will be processed and you can find out which sig + +What is the problem +No time to review PRs + +Mentoring Program +Mentoring program overview +Common theme is that we do not have time to review PR +Bootcamps for new contributors + +Lot of Common Complaints + +Tests did not pass what happened +Devstats +Everyone needs to look at the stats +We are currently bottlenecked at approvers not reviews + +How to fix the problem +SIG Leads are too busy +Have another role within the sigs for mentors +Can we label the issues in terms of complexity +Sig-cli has it documented in a contributor guide +Incentivizing mentoring +How do we help take the load off of the approvers pool, how do we improve reviews +How many times a reviewer say lgtm +Reviewer performance +We need to document what a good review looks like +Guidance on progress to contributor -> reviewer -> maintainer +How does maintain a good citizenship in the community +Put the documentation into kubernetes.io +Simple use-case examples on how to get a PR in +Give contributors some responsibilities +Planned workshop on how to get a PR merged +Contributor based workshop +Ask questions on #kubernetes-dev +We need to document how to get answers +We need to move our documents into kubernetes.io +Sigs may not want to have new contributors +We need to cleanup those who are inactive in k/k - OWNERS and MAINTAINERS +How do we create incentives? +If they are autosigned an issue and do not respond - we need to measure that metric +How do we fix the release process in terms of what is in a release, because we do not have a target we do not know what we are shooting at +We need to figure out what new contributors are working on +Start off by using kubernetes then become contributors diff --git a/events/2017/12-contributor-summit/scaling-up-and-scaling-down-and-addon-management.md b/events/2017/12-contributor-summit/scaling-up-and-scaling-down-and-addon-management.md new file mode 100644 index 00000000..d7f3e3a0 --- /dev/null +++ b/events/2017/12-contributor-summit/scaling-up-and-scaling-down-and-addon-management.md @@ -0,0 +1,104 @@ +# Scaling Up & Scaling Down & Addon Management Session… + +Notes by @justinsb + +## Scaling Up & Scaling Down + +Lots of users that want to run on a single node cluster - one node, one core +thockin’s position: 1 node 1 core should continue to work (with 3.75GB, but ideally less than 2GB) +Works today but only just - e.g. you only have 0.2 cores left today +Addons / core components written with a different target in mind +Some of the choices not optimal for single node (e.g. fluentd / ES) + +Cluster Proportional Autoscaler tries to solve this + Targets a daemonset / deployment and changes number of pods + +CPVPA Cluster Proportional Vertical Pod Autoscaler + Applies a step function based on the number of cluster cores to change the resources needed by a target deployment / daemonset +Prototype being used today in GKE for calico + Needs a pod per target which leads to higher resource usage + Idea: to have a single pod that uses a CRD to amortize the costs across multiple targets + Problems we want to address: + We want more granularity at the low end, both because it matters more proportionately + We want to avoid constant rescaling - changing resources requires restarting pods + +Q (solly): What is missing from kubernetes today - what is different vs generic HPA/VPA? Why can’t I reuse the existing HPA/VPA? How do I tell the difference from a UI perspective? +A (thockin): We could wrap the autoscaler; or we could have a ref pointing back up; or we could integrate back into HPA/VPA. The third option is mostly what we’ve encouraged - where things are proved out of tree and then merged back in. + +Some changes needed in HPA/VPA - pulling in metrics from a different source e.g. number of cores in cluster. (Do we need a “count” API) + +Q (clayton): Does this indicate that requests / limits are “broken”? Should we instead be thinking about e.g. overcommit to deal with smaller sizes. +A (thockin): No - the step function is defined by the author of e.g. kube-dns based on their knowledge of the requirements. And this is typically for “fundamental” components, not user applications, and so the broader question for users is separate. + +Q (gus): We need to consider architectures (e.g. 64 bit might use more memory). Are we wasting more memory running a pod to right-size the other pods? +A (thockin): That’s why we want one pod with a CRD, but yes … we want to fold it back in. but we have to prove it first. + +Q: Why can’t this be done in e.g. kubeadm +A: Because clusters change size - e.g. cluster autoscaler + +Q (jago): Focusing on small clusters - we should come up with clear guidance for resource requirements. Maybe we need T-shirt sizes which helps us figure out what we can run in various clusters. +A (thockin): We need to start collecting data - that’s part of this effort. + +Q (vish): Is kubernetes the right approach for small clusters +A: Yes! Yes! Yes! Lots of resounding yeses - devs want to run on their laptops, people happily run personal clusters on single nodes etc. + +Q (dawn): It is hard to come up with a single scale function - e.g. with docker on the node team the kernel options have a big impact. But we should still provide a scaling function as a guide, and to allow for operators to tweak it + +Q (chrislovecnm): Key point - we must collect data! + +Q: There is a lack of documentation on resource requirements e.g. memory requirements of apiserver at various sizes. + + +## Add-on Management + +It’s been the can that’s been kicked down the road +Questions: + What is an addon? + Kubeadm says kube-dns, kube-proxy + GKE adds dashboard & heapster + +Thockin: Addons are things we manage vs things you manage. We = administrator, you = User + +Luxas: kubeadm installs minimal addons to get conformance to pass + +solly: Important not to conflate addons with discussion of what is core vs not + +Clayton: Is Istio an addon? What if it becomes super-complicated... + +Mrubin: 3 categories of addons: + Istio, Service Catalog, Service Broker + Network policy + Kube-proxy and kube-dns +Sniff test: Does it have to be tied to a version of kubernetes? + +Robertbailey: Maybe addons should be the minimal set of things required to pass conformance + +Chrislovecnm: What’s our goal - we want something to install a manifest on the cluster? + +Clayton: the challenge isn’t installation - it’s upgrades / updates. Are we really ready to manage this? Addons should be things that we can actually manage, including addons. + +Thockin: we want a simple solution, not a full solution for arbitrarily complicated applications. + +Jago: We want a set of components that as a platform operator we install and upgrade/version together and validated together, and cluster-level. This is distinct from user applications or even application etcd. + +Bboreham: cross-grade between alternatives is a related aspect - e.g. swap out implementation of network policy. In practice today you have to reinstall the cluster, because otherwise it requires the network policies to cooperate. + +Clayton: comparison to runlevels - where it’s very possible to brick your cluster by doing things out of sequence. + +Gus: The installation tools do all differ in what addons they are installing (e.g. AWS and GCE have very different requirements). So the installation tool will have opinions here - what value are we adding? + +Thockin: There is an opportunity here to define something cross-cutting - “apt or yum for kubernetes”. Having each installer do something different is not great. + +Clayton: apt or yum doesn’t get you all the way - you still need to tweak configurations. + +Mrubin: Kubernetes is a bunch of small pods cooperating. The mechanism is the same for installing DNS as it is for network policy - which is good in that we have a harmonized approach. We want to figure out the mechanism by which components can be packaged by authors, discovered, and integrated by installers. + +Mrubin: There is a belief that some pods in kube-system are special + +Dawn: The kube-system namespace is known to be system-managed / updated and critical pods - even if it isn’t actually special in a technical sense. + +Jess: We want to avoid decision paralysis. And there’s odd incentives where people want to monetize their code or their approach. We have to limit the scope as much as possible. + +Robertbailey: Agreed - we have to scope this just to the minimal components needed to install stuff. And then users can install more opinionated things like helm. + +Thockin wrapping up: Still an open question of what we should do here, how the various distributions should avoid duplicating effort. diff --git a/events/2017/12-contributor-summit/schedule.png b/events/2017/12-contributor-summit/schedule.png Binary files differnew file mode 100644 index 00000000..03942e5d --- /dev/null +++ b/events/2017/12-contributor-summit/schedule.png diff --git a/events/2017/12-contributor-summit/should-k8s-use-feature-branches.md b/events/2017/12-contributor-summit/should-k8s-use-feature-branches.md new file mode 100644 index 00000000..f8af5427 --- /dev/null +++ b/events/2017/12-contributor-summit/should-k8s-use-feature-branches.md @@ -0,0 +1,113 @@ +# Should Kubernetes use feature branches? + +Lead: Brian Grant + +Note Taker(s): Aaron Crickenberger + +Focusing on our current release cadence: + +- We spend several weeks putting features in +- We spend several weeks stabilizing the week kinda/sortof + +The people who are managing the release process don’t have the ability to push back and say this can’t go in and we don’t want to slip the release so they’re left powerless + +In earlier session today over breaking up monolith: + +- We have over 90 repos +- The releases are getting bigger and more complex +- Not everything we’re moving out of the monorepo, not everything makes sense + +What do we do about those things +I think the only way we can get out of the release fire drills is to have an earlier gate + +Q: Aaron: Why not use the tests we have today and push back saying the tests have to be green? + +- A lot of the new features have inadequate testing or inadequate features +- We want to continue to release on a regular on cadence +- Use some model like gitflow that puts feature work into feature branches before they land into master +- I think we need to allow features to be developed in feature branches instead of landing in kubernetes/kubernetes master +- But we also can’t have large monolithic feature branches +- Because one thing going in might cause rebase hell for other things going in +- Test machinery is getting to a point where it can use branches other than master to spin up tests +- You can spin up a feature branch and get tests running on it without too many issues +- I think we have to do within either 1.10 or 1.11 release + +Q. I heard two 2 concerns: + +- How do you protect mainline development from these features going in? +- How do you know when they’re ready to go in? +- So nodejs claims to have something like 90-something% coverage, we don’t even have coverage measurement +- With the interdependence of everything in the system it’s basically impossible to back changes out + +Q: (thockin) +How do we think we can do this across repos? + +- Can we hear from people who have actually tried this for what the pain points are? +- There are lots of details to work out and people are just going to have to try it at small scale and then at large scale +- A lot of the times people try to get their foot in the door with tiny feature PR’s, and then docs, and then feature completion +- Jan commenting on experience with workload api’s +- Tried bumping workload api’s from v1beta2 to v1, tried it on feature branch +- Main blocker was tests, tests weren’t running against the feature branch, only the master branch automatically +- We decided we were just going to do it directly against the master branch + +Q (spiffxp) How do we prevent merge chicken, aka racing to get in so you don’t have to avoid rebase hell + +- With api machinery, one thing we’ve done is wait until after the release cycle to minimize the amount of time that people have to do rebasing +- We scheduled that massive change to land right after code freeze +- Re: how do we prevent once you’re in you’re in… that’s basically the way it goes right now + +Thockin: + +- the pattern that I see frequently is somebody sends me a 10k line PR and I see please break this up and they do, but then they forget the docs and somewhere along the way we miss it +- Bgrant: it’d be great if we could ask people to write the docs first (maybe this is a criteria for feature branches) +- Venezia: what about having a branch per KEP? Also, facebook does something like feature flags, is that something we could do or is that just impossible? + +Bgrant: + +- That can be used, but it requires a huge amount of discipline to ensure those flags are used correctly and consistently throughout the code +- Some features we can easily do that eg: if it’s an API, but it’s much harder to do with a feature that’s threaded through components +- A branch per KEP is roughly the right level of granularity +- What’s the difference between a feature branch and a massive PR? +- Review flow isn’t well suited to massive PR’s +(thockin) It’s easier on maintainers if we get things in and lock them in, but not on the main tree +(thockin) one thing I like about the linux kernel model is the lieutenant model + +Q: (maru) you’re still going to have to rebase feature branches the way that you rebase PR’s right? + +- A (jago) the difference is that a whole team could work on a feature branch +- I like the idea of trying a few projects and then reporting back +- I think roughly the level of granularity of having a branch per KEP sounds about right +- Multiple feature branches affecting the same area of code would be a very useful place to get some infromation + +Q (???) if someone needs to rebase on a feature branch, could it instead be resolved by a merge commit instead? + +- (thockin) I think we could solve that with technology and it could be better than building a better review tool +- Rebasing sucks, part of the reason we have that is generated code, part of it is poor modularity +- We could stand to expend some energy to improve modularity to reduce rebases +(bgrant) either feature branches or moving code to other repos are forcing functions to help us clean some of that up + +Q: how do we make sure that when issues are filed people know which commit/feature branch they should be filing against? +- (bgrant) I suspect most issues on kubernetes/kubernetes are filed by contributors +- (jdumars) imagine you have a KEP that represents three issues, three icecubes you’ve chipped off the iceberg, the feature branch would hold code relevant to all of those issues +- (luxas) feature branches should be protected + +Yes we’ll definitely want a bot that automatically +Q: (alex) do we want to have feature branches +Q: (thockin) why not just use a fork instead of branches? + +- I think it’s harder to get tests spun up for other users +- (jgrafton) worry about adding yet another dimension of feature branches to our already large matrix of kubernetes and release branches + +- Not so much for PR’s, but for CI/periodic jobs +- You need to run more than just the PR tests +- The experience with gated features is that tests do run less often, so racey bugs that pop up less frequently get exposed less often; so there is a concern that we wouldn’t get enough testing coverage on the code +- We’ll figure out what the right granularity is for feature branches is, probably shorter than a year, but probably shorter than a year +- One idea is instead of support forks to users, what if we forked to orgs owned by sigs, and just made sure the CI was well setup for those orgs? +(ed. I’m nodding my head vigorously that this is possible) +- Just another comment on branches on main repo vs. in other repos +- The branches in the main repo could be another potential for mistakes and fat fingers that we could be better solve by making the tooling work against arbitrary repos +- (jgrafton) related to the ownership of e2e tests and the explosion of stuff.. Do we expect sigs to own their tests across all feature branches? - Or should they only care about master + +Q: worry about tying feature ownership too closely to a single sig, because features could involve multiple sigs + +If you want to get started with feature branches, reach out to kubernetes-dev first and we’ll get you in touch with the right people to get this process started diff --git a/events/2017/12-contributor-summit/steering-committee-update.md b/events/2017/12-contributor-summit/steering-committee-update.md new file mode 100644 index 00000000..3c2f0a92 --- /dev/null +++ b/events/2017/12-contributor-summit/steering-committee-update.md @@ -0,0 +1,47 @@ +Steering Committee Update +-------------------------- + +Notes by @jberkus + +Showed list of Steering Committee backlog. + +We had a meeting, and the two big items we'd been pushing on hard were: + +* votining in the proposals in the bootstrap committee +* how we're going to handle incubator and contrib etc. + +Incubator/contrib: one of our big concerns are what the the consequences for projects and ecosystems. +We're still discussing it, please be patient. In the process of solving the incubator process, we have to answer +what is kubernetes, which is probably SIGs, but what's a SIG, and who decides, and ... we end up having to examine +everything. In terms of deciding what is and isn't kubernetes, we want to have that discussion in the open. +We also recognize that the project has big technical debt and is scary for new contributors. + +We're also trying to figure out how to bring in new people in order to have them add, instead of contributing +to chaos. So we need contributors to do mentorship. We also need some tools for project management, if anyone +wants to work on these. + +We're also going to be working on having a code of conduct for the project, if you +want to be part of that discussion you can join the meetings. For private concerns: steering-private@kubernetes.io. +One of the challenges is deciding how enforcement will work. We've had a few incidents over the last 2 years, +which were handled very quietly. But we're having more people show up on Slack who don't know norms, +so feel free to educate people on what the community standards are. As our project expands across various +media, we need to track the behavior of individuals. + +Q: "Can SIGs propose a decision for some of the stuff on the backlog list and bring it to the SC?" +A: "Yes, please!" + +The other big issue is who owns a SIG and how SIG leaders are chosen. The project needs to delegate more things to the +SIGs but first we need to have transparency around the SIG governance process. + +Q: "As for What Kubernetes Is: is that a living document we can reference?" +A: "There's an old one. I have a talk on Friday about updating it." + +There's also the question of whether we're involved in "ecosystem projects" like those hosted by the CNCF. Things will change +but the important thing is to have a good governance structure so that we can make transparent decisions as things change. + + + + + + + diff --git a/events/2017/12-contributor-summit/whats-up-with-sig-cluster-lifecycle.md b/events/2017/12-contributor-summit/whats-up-with-sig-cluster-lifecycle.md new file mode 100644 index 00000000..b2f202b3 --- /dev/null +++ b/events/2017/12-contributor-summit/whats-up-with-sig-cluster-lifecycle.md @@ -0,0 +1,62 @@ + +What's Up with SIG-Cluster-Lifecycle +------------------------------------ + +Notes by @jberkus + +Luxas talking: I can go over what we did last year, but I'd like to see your ideas about what we should be doing for the future, especially around hosting etc. + +How can we make Kubeadm beta for next year? +Opinions: +* HA + * etcd-multihost + * some solution for apiserver, controller +* Self-hosted Kubeadm + +Q: Can someone write a statement on purpose & scope of Kubeadm? + +To install a minimum viable, best-practice cluster for kubernetes. You have to install your own CNI provider. Kubeadm isn't to endorse any providers at any level of the stack. + +Joe: sub-goal (not required for GA), would be to break out components so that you can just install specific things. +Would also like documentation for what kubeadm does under the covers. + +Josh: requested documentation on "how do I modify my kubeadm install". Feels this is needed for GA. Another attendee felt the same thing. + +One of the other goals is to be a building block for higher-level installers. Talking to Kubespray people, etc. Enabling webhooks used as example. + +There was some additional discussion of various things people might want. One user wanted UI integration with Dashboard. The team says they want to keep the scope really narrow in order to be successful. UI would be a different project. GKE team may be working on some combination of Kubeadm + Cluster API. Amazon is not using Kubeadm, but Docker is. Docker for Mac and Windows will ship with embedded kubernetes in beta later this week. + +Kubeadm dilemma: we want humans to be able to run kubeadm and for it to be a good experience, and for automation to be able to run it. I don't think that can be the same tool. They've been targeting Kubeadm at people, we might want to make a slightly different UI for machines. Josh says that automating it works pretty well, it's just that error output is annoying to capture. + +HA definition: +* etcd should be in a quorum HA standard (3+ nodes) +* more than one master +* all core components: apiserver, scheduler, kcm, need to be on each master +* have to be able to add a master +* upgrades + +Or: we want to be able to survive a loss of one host/node, including a master node. This is different, if we want to survive the loss of any one master, we only need two then. Argument ensued. Also, what about recovery or replacement case? A new master needs to be able to join (manual command). + +What about HA upgrades? Are we going to support going from one master to three? Yes, we have to support that. + +Revised 4 requirements: +* 3+ etcd replicas +* all master components running in each master +* all TLS secured +* Upgrades for HA clusters + +Everyone says that we want a production environment, but it's hard to define what "production grade" means. We need to stop saying that. Over time, what matters is, "is it maintained". If it's still being worked on, over time it'll get better and better. + +CoreOS guy: trying to do self-hosted etcd. There's a lot of unexpected fragile moments. Just HA etcd isn't well tested upstream, there's not enough E2E tests. Self-hosting makes this worse. The etcd operator needs work. There needs to be a lot of work by various teams. Self-hosted control planes work really well, they host all of their customers that way. It's etcd that's special. + +There's some problems with how Kubernetes uses HA Etcd in general. Even if the etcd operator was perfect, and it worked, we couldn't necessarily convince people to use it. + +Should Kubeadm focus on stable installs, or should it focus on the most cutting-edge features? To date, it's been focused on the edge, but going to GA means slowing down. Does this mean that someone else will need to be forward-looking? Or do we do feature flags? + +SIG-Cluster-Lifecycle should also document recommendations on things like "how much memory do I need." But these requirements change all the time. We need more data, and testing by sig-scalability. + +For self-hosting, the single-master situation is different from multi-master. We can't require HA. Do we need to support non-self-hosted? We can't test all the paths, there's a cost of maintaining it. One case for non-self-hosted is security, in order to prevent subversion of nodes. + +Also, we need to support CA-signed Kubelet certs, but that's mostly done. + +So, is HA a necessity for GA? There are a bunch of automation things that already work really well. Maybe we should leave that to external controllers (like Kops etc) to use Kubeadm as a primitive. Now we're providing documentation for how you can provide HA by setting things up. But how would kubeadm upgrade work, then? diff --git a/events/OWNERS b/events/OWNERS new file mode 100644 index 00000000..bcc3f9b5 --- /dev/null +++ b/events/OWNERS @@ -0,0 +1,9 @@ +reviewers: + - parispittman + - castrojo +approvers: + - parispittman + - castrojo + - sig-contributor-experience-leads +labels: + - sig/contributor-experience diff --git a/events/elections/2017/BALLOTS.csv b/events/elections/2017/BALLOTS.csv new file mode 100644 index 00000000..9c67b4c1 --- /dev/null +++ b/events/elections/2017/BALLOTS.csv @@ -0,0 +1,310 @@ +Timothy St. Clair,Ilya Dmitrichenko,Caleb Miles,Doug Davis,Phillip Whitrock,Kris Nova,Derek Carr,Aaron Crickenberger,Jaice Singer DuMars,Michelle Noorali,Ihor Dvoretskyi,Alex Pollitt,Michael Rubin,Quinton Hoole,Rob Hirschfeld,Aaron Schlesinger,Sebastien Goasguen,Matt Farina,Adnan Abdulhussein,Justin Santa Barbara +20,20,10,20,2,20,15,1,1,2,20,20,2,20,20,20,20,15,20,10 +4,20,20,20,6,20,2,20,20,20,20,5,20,1,20,20,20,20,20,3 +7,16,11,11,11,2,10,5,1,2,7,No opinion,15,5,17,11,9,No opinion,No opinion,2 +20,1,20,20,20,20,20,20,20,20,2,20,20,20,20,20,20,20,20,20 +5,No opinion,8,No opinion,3,No opinion,1,7,No opinion,4,No opinion,No opinion,2,6,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion +3,20,5,6,7,2,7,7,7,7,7,1,4,16,18,7,14,19,7,17 +1,20,20,20,20,2,20,20,20,3,20,20,20,20,20,20,4,20,20,20 +4,1,20,20,20,20,20,20,20,3,20,20,20,20,20,20,20,20,2,20 +8,13,16,3,2,7,19,15,6,12,5,18,1,20,11,17,4,9,14,10 +20,20,20,20,20,20,1,20,20,20,20,20,20,20,20,20,20,20,20,20 +6,20,8,20,3,13,1,7,12,2,20,9,4,5,20,20,10,20,11,20 +20,20,3,20,9,1,4,7,2,5,20,20,9,9,20,6,20,20,20,20 +20,20,20,20,20,20,1,20,20,20,20,20,20,20,20,20,20,20,20,20 +2,3,8,20,20,5,1,20,20,6,20,20,20,20,7,20,20,9,20,4 +2,14,8,20,3,9,1,7,13,5,18,10,6,4,19,16,11,15,12,17 +20,20,20,6,20,2,20,4,3,1,20,20,20,5,20,20,20,20,20,20 +6,20,8,20,3,20,1,7,20,2,20,9,4,5,20,20,10,20,20,20 +13,No opinion,7,No opinion,1,11,12,5,10,4,6,No opinion,No opinion,2,No opinion,No opinion,8,No opinion,No opinion,3 +6,14,8,20,3,13,1,7,12,2,18,9,4,5,19,16,10,15,11,17 +4,20,8,5,7,6,14,14,3,14,14,1,2,17,15,9,18,14,19,16 +2,8,19,20,4,7,1,10,16,9,5,11,3,6,14,15,13,18,17,12 +5,18,16,11,6,1,12,20,17,4,9,8,7,19,13,14,2,10,15,3 +7,3,12,1,5,19,9,16,18,13,2,15,20,10,6,17,8,11,14,4 +1,14,11,12,7,2,3,6,5,4,17,10,9,16,19,No opinion,8,No opinion,15,20 +20,18,11,2,5,17,9,10,7,8,4,14,19,12,16,15,1,3,6,13 +3,20,7,20,2,20,20,10,9,4,6,20,1,20,20,20,12,8,5,11 +2,20,20,20,20,20,1,4,20,20,6,20,5,3,20,20,20,20,20,20 +2,18,5,12,8,6,1,7,16,9,14,15,3,4,20,13,10,17,19,11 +5,20,20,20,3,1,16,20,20,1,20,20,4,6,20,20,20,20,20,7 +20,11,2,20,1,12,5,11,2,9,6,20,1,20,8,11,9,12,11,6 +2,7,2,1,20,20,5,20,20,20,6,20,20,3,20,20,20,20,20,4 +20,20,20,20,20,20,20,20,20,20,20,20,20,1,20,20,20,20,20,20 +20,20,20,20,20,20,2,20,1,20,20,20,20,20,20,20,20,20,20,20 +7,12,8,17,3,11,1,6,10,4,18,15,5,2,20,19,13,14,9,16 +12,1,2,8,17,4,9,3,19,10,14,13,18,11,5,15,7,6,16,20 +7,18,20,4,3,14,1,9,12,2,15,8,5,6,19,17,10,16,11,13 +8,18,15,14,1,10,17,12,3,20,11,20,1,16,20,19,2,20,9,13 +6,20,8,20,7,6,3,4,8,2,20,20,1,9,5,20,20,20,20,4 +2,20,20,20,1,20,20,20,20,20,6,20,3,20,20,20,5,20,20,4 +3,20,20,20,2,20,20,20,20,20,20,20,1,20,20,20,20,20,20,20 +4,10,8,20,5,20,1,3,20,20,6,20,2,11,20,9,20,20,20,7 +20,20,1,20,7,3,20,20,20,2,20,4,6,20,5,20,20,20,20,20 +1,5,13,16,10,20,9,11,12,19,14,17,15,2,18,4,7,3,8,6 +20,20,20,20,20,20,20,20,20,20,20,20,20,1,20,20,20,20,20,20 +20,20,20,20,20,1,20,20,20,20,20,20,20,20,20,20,20,20,20,20 +20,20,1,20,3,20,20,6,5,2,20,8,4,7,20,20,20,20,20,9 +2,20,9,20,1,20,2,9,2,1,9,20,7,7,20,20,20,20,20,2 +20,20,2,20,20,20,20,5,20,1,3,20,20,20,6,20,20,20,20,4 +2,20,4,20,4,20,1,20,20,3,20,20,20,20,20,20,20,20,20,20 +2,20,20,20,20,20,1,20,19,3,20,20,20,20,20,20,20,20,20,20 +6,9,No opinion,No opinion,No opinion,5,No opinion,No opinion,4,1,2,No opinion,No opinion,No opinion,3,No opinion,8,No opinion,No opinion,7 +19,9,4,20,1,16,5,7,17,18,3,6,2,10,13,15,11,8,12,14 +8,16,18,12,13,6,15,10,4,19,7,3,9,2,5,11,1,14,17,20 +6,14,8,20,4,13,1,7,12,3,18,9,5,2,19,16,10,15,11,17 +20,20,20,20,2,20,4,20,20,20,20,20,1,20,20,20,20,20,20,4 +14,No opinion,3,13,2,12,No opinion,4,1,6,9,No opinion,11,5,10,8,7,No opinion,No opinion,16 +3,No opinion,No opinion,No opinion,No opinion,2,No opinion,No opinion,No opinion,1,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion +20,20,20,20,20,20,20,20,20,20,20,20,20,1,20,20,20,20,20,20 +1,18,8,10,9,11,4,5,6,3,17,14,12,2,20,13,16,19,15,7 +No opinion,3,No opinion,No opinion,No opinion,20,No opinion,No opinion,No opinion,4,No opinion,No opinion,No opinion,No opinion,No opinion,1,2,5,No opinion,No opinion +13,13,3,18,1,14,9,5,6,10,8,19,2,20,13,15,4,17,11,7 +No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,1 +7,18,11,14,1,9,8,12,5,2,20,4,10,19,17,6,3,13,16,15 +1,4,15,17,15,6,20,15,16,2,5,15,15,3,7,18,15,19,15,15 +20,20,20,20,20,2,20,20,20,5,20,4,3,1,20,20,20,6,20,20 +2,No opinion,No opinion,No opinion,No opinion,No opinion,1,No opinion,No opinion,6,No opinion,4,No opinion,7,8,No opinion,3,9,No opinion,5 +20,No opinion,2,3,1,20,2,2,5,20,3,5,1,20,No opinion,20,No opinion,3,No opinion,2 +3,No opinion,No opinion,No opinion,1,5,No opinion,No opinion,No opinion,4,No opinion,No opinion,2,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion +2,20,20,20,3,4,1,20,20,5,20,20,20,20,20,6,20,20,20,20 +3,11,8,15,2,12,1,7,10,4,13,9,5,6,13,11,9,11,9,14 +4,9,6,20,3,20,1,5,20,20,20,20,8,20,20,20,20,20,20,1 +8,19,11,19,6,5,10,1,3,7,4,19,2,19,19,12,9,19,19,20 +8,No opinion,16,No opinion,2,13,9,3,1,11,10,No opinion,7,4,5,15,14,12,No opinion,6 +3,7,4,20,10,1,2,20,12,9,20,6,5,11,20,20,20,8,20,20 +20,20,20,20,20,20,20,20,20,20,20,20,20,20,1,20,20,20,20,20 +5,20,20,20,2,20,3,20,20,7,4,20,1,6,20,20,20,20,20,20 +13,19,17,20,5,6,7,9,1,2,12,18,14,11,16,8,10,4,3,15 +5,20,19,19,19,2,19,19,7,19,8,1,3,19,6,19,19,19,19,19 +3,20,20,20,20,20,20,20,20,20,1,20,20,20,20,20,20,2,20,20 +4,20,20,20,5,20,2,1,6,20,20,20,20,20,20,20,20,20,20,3 +6,No opinion,8,No opinion,3,No opinion,1,7,No opinion,2,No opinion,9,4,5,No opinion,No opinion,10,No opinion,No opinion,No opinion +2,5,4,7,18,19,6,11,15,14,16,3,1,20,17,9,8,12,10,13 +20,20,1,20,1,1,20,1,1,1,20,20,20,20,20,20,20,20,20,20 +11,5,No opinion,4,3,No opinion,No opinion,9,No opinion,2,20,10,7,No opinion,No opinion,8,6,No opinion,No opinion,1 +2,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,3,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,1 +6,14,8,20,3,13,1,7,12,2,18,9,4,5,19,16,10,15,11,17 +20,20,1,20,3,20,20,6,5,2,20,8,4,7,20,20,20,20,20,9 +2,No opinion,8,13,3,12,1,9,7,4,No opinion,No opinion,5,6,10,No opinion,No opinion,No opinion,No opinion,11 +6,19,3,19,19,7,9,2,1,19,5,19,7,4,19,10,19,19,19,20 +20,20,20,20,20,20,20,20,20,20,20,20,20,1,20,20,20,20,20,20 +2,6,13,20,12,5,19,10,19,9,8,14,6,19,7,19,4,1,3,11 +10,10,8,10,1,10,1,10,1,10,1,1,8,20,10,10,1,10,10,1 +8,14,3,12,6,15,1,9,11,5,20,10,2,7,20,20,20,13,20,4 +6,4,5,13,7,1,17,17,10,3,11,8,17,18,20,9,17,19,12,2 +5,14,7,20,2,13,1,6,12,8,18,9,3,4,19,16,10,15,11,17 +4,18,20,18,1,19,14,17,2,18,13,18,15,18,12,18,16,18,18,3 +2,9,20,20,11,5,8,3,4,6,10,20,20,20,7,12,11,20,20,1 +13,7,11,16,1,20,3,6,5,19,14,9,4,17,15,12,8,10,18,2 +1,20,20,20,20,2,20,20,20,3,20,20,20,6,20,4,20,20,20,5 +4,14,7,20,3,12,1,8,19,2,18,10,6,5,16,17,15,11,9,13 +No opinion,5,2,No opinion,No opinion,4,No opinion,No opinion,2,No opinion,3,No opinion,No opinion,No opinion,No opinion,No opinion,1,No opinion,No opinion,No opinion +20,20,20,20,2,20,20,20,20,20,20,20,1,20,20,20,20,20,20,20 +6,10,7,12,5,16,3,13,19,9,8,11,2,1,18,20,15,17,14,4 +6,3,No opinion,No opinion,2,No opinion,No opinion,No opinion,No opinion,No opinion,7,No opinion,1,5,No opinion,No opinion,No opinion,No opinion,No opinion,4 +20,20,5,20,1,20,5,2,4,20,20,20,3,20,20,20,20,20,20,5 +3,20,8,20,4,20,1,20,20,5,20,20,2,7,20,20,20,20,20,6 +7,5,20,15,4,17,10,1,6,3,9,18,2,13,16,19,11,14,8,12 +4,7,No opinion,No opinion,No opinion,3,No opinion,6,8,2,9,No opinion,No opinion,5,No opinion,No opinion,No opinion,No opinion,10,1 +20,20,20,20,20,1,20,4,20,2,8,5,3,20,20,7,20,6,20,20 +18,16,15,20,5,2,17,19,9,1,3,11,10,12,13,7,4,8,6,14 +2,12,14,15,8,3,7,13,4,1,5,18,16,19,17,6,11,20,9,10 +3,20,4,20,1,20,1,2,2,20,2,20,1,20,20,20,20,20,20,2 +4,17,5,14,18,1,11,8,10,2,3,19,20,7,6,9,16,15,13,12 +5,10,10,20,2,5,3,20,10,5,10,20,1,10,20,20,20,10,20,5 +6,20,8,20,3,20,1,7,20,2,20,9,4,5,20,20,10,20,20,20 +3,20,5,20,9,20,1,7,20,8,20,20,2,20,20,20,20,20,20,4 +7,20,8,20,20,20,3,4,6,20,20,20,1,5,20,20,20,20,20,2 +20,20,20,3,7,4,20,5,2,20,6,20,20,20,20,20,20,20,20,1 +3,20,5,14,10,7,13,1,4,6,12,8,9,20,20,20,11,2,20,20 +11,2,18,20,16,3,10,17,1,7,19,12,15,5,14,4,13,9,6,8 +5,20,7,20,1,20,4,20,20,2,20,20,20,3,20,20,20,20,20,6 +15,No opinion,6,13,1,No opinion,14,3,4,10,7,No opinion,2,12,No opinion,No opinion,7,7,No opinion,5 +15,14,12,8,16,11,1,19,17,20,4,10,3,13,9,5,18,7,6,2 +7,19,10,10,1,20,9,11,16,3,18,6,2,4,8,13,5,15,14,17 +3,5,10,13,2,9,4,20,6,12,15,16,1,14,11,8,18,19,7,17 +3,18,5,20,12,6,1,10,13,2,14,8,7,4,16,17,9,19,11,15 +20,20,4,20,2,20,8,7,3,20,6,20,1,20,20,20,20,20,20,5 +20,20,20,20,20,20,20,20,20,20,20,20,20,1,20,20,20,20,20,2 +8,12,7,No opinion,4,11,1,13,3,10,6,No opinion,2,9,No opinion,14,No opinion,No opinion,No opinion,5 +2,20,8,20,11,20,1,20,9,7,20,20,6,5,12,4,10,20,20,3 +20,19,8,17,1,10,17,17,17,10,11,17,2,7,17,3,7,10,18,7 +20,20,20,20,1,20,5,4,20,20,20,20,3,6,20,20,20,20,20,2 +4,7,6,20,11,3,15,10,19,1,13,8,14,16,12,5,17,2,9,18 +19,20,20,20,20,20,20,20,20,20,1,2,20,20,18,20,20,20,20,20 +No opinion,No opinion,19,No opinion,12,18,15,20,11,10,13,No opinion,16,No opinion,14,17,8,No opinion,No opinion,9 +4,14,7,20,3,12,1,18,13,2,10,8,6,5,19,17,9,15,11,16 +2,20,20,20,20,20,1,20,4,20,20,20,3,20,20,20,20,20,20,5 +4,No opinion,3,12,6,15,13,11,10,13,10,20,2,18,5,10,1,10,10,14 +1,2,20,20,20,2,20,20,20,20,20,20,20,3,20,20,20,20,20,20 +20,16,17,19,8,18,6,2,4,5,3,12,9,10,15,14,13,1,7,11 +6,No opinion,11,No opinion,13,1,4,No opinion,3,2,5,No opinion,8,No opinion,No opinion,10,12,No opinion,9,7 +20,20,20,20,20,20,20,20,20,20,20,20,20,1,20,20,20,20,20,20 +17,20,18,20,2,20,5,18,18,10,4,20,1,3,20,18,7,20,20,6 +7,4,7,7,7,2,7,7,7,7,3,7,6,5,7,7,7,7,7,1 +6,9,9,4,6,1,7,8,4,2,10,7,5,5,9,10,8,5,3,6 +4,No opinion,No opinion,1,3,No opinion,No opinion,2,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,5,No opinion,No opinion,No opinion,No opinion +6,11,8,20,3,10,1,7,9,2,14,13,4,5,15,18,16,17,12,19 +11,No opinion,No opinion,No opinion,9,1,No opinion,No opinion,No opinion,7,6,8,5,10,4,No opinion,3,No opinion,12,2 +20,20,4,20,1,20,20,20,20,20,20,20,1,6,20,20,20,20,5,20 +14,1,13,12,7,20,6,8,1,15,16,4,3,5,2,10,19,17,11,9 +5,6,No opinion,No opinion,No opinion,1,No opinion,No opinion,4,2,5,5,No opinion,No opinion,4,3,No opinion,No opinion,No opinion,3 +19,6,4,20,8,13,7,11,10,9,14,2,1,20,3,17,12,15,16,5 +11,6,9,20,4,2,3,12,10,8,5,13,20,20,20,20,20,20,7,1 +3,20,4,20,7,4,20,1,20,20,4,20,20,20,20,8,20,2,20,20 +14,1,5,No opinion,6,3,11,2,7,9,10,No opinion,15,8,12,No opinion,4,16,13,No opinion +6,20,20,20,20,2,20,17,7,3,20,20,20,18,20,20,20,18,20,18 +5,10,9,17,6,20,15,2,1,4,11,15,8,12,18,3,8,15,16,19 +3,No opinion,No opinion,No opinion,No opinion,No opinion,1,No opinion,8,4,No opinion,No opinion,2,6,No opinion,No opinion,7,No opinion,No opinion,5 +6,14,8,20,3,13,1,7,12,2,18,9,4,5,19,16,10,15,11,17 +7,20,9,20,2,5,1,8,10,3,20,20,6,4,20,20,20,20,20,20 +6,14,8,20,3,12,1,7,13,2,19,9,4,5,17,18,10,15,11,16 +2,No opinion,No opinion,No opinion,No opinion,5,3,No opinion,7,6,No opinion,No opinion,No opinion,4,No opinion,No opinion,No opinion,No opinion,No opinion,1 +11,No opinion,10,No opinion,4,No opinion,1,2,8,5,13,No opinion,6,7,No opinion,No opinion,9,3,12,20 +12,14,1,20,7,15,4,3,9,10,11,5,6,18,19,16,2,8,17,13 +2,16,13,19,7,6,11,17,9,5,18,20,1,12,10,15,3,14,4,8 +20,20,20,20,20,20,20,20,20,20,20,20,20,1,20,20,20,20,20,20 +5,No opinion,No opinion,3,No opinion,2,1,No opinion,No opinion,No opinion,No opinion,4,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion +1,6,18,16,4,3,2,17,5,No opinion,19,No opinion,No opinion,6,No opinion,No opinion,No opinion,No opinion,No opinion,20 +20,11,7,6,1,16,3,12,5,15,2,18,1,4,19,17,13,14,10,8 +20,20,20,20,2,20,20,20,20,20,20,20,1,20,20,20,20,20,20,3 +9,19,3,2,6,16,10,1,5,No opinion,15,8,14,17,13,12,4,7,11,18 +2,20,20,19,20,5,1,19,19,3,4,20,20,20,20,20,20,19,20,19 +6,14,8,14,3,12,1,7,12,2,14,9,4,5,14,14,9,14,9,14 +2,20,1,20,3,20,20,20,20,20,20,20,20,4,20,20,20,20,20,20 +2,3,3,3,3,3,3,3,3,3,3,3,1,3,3,3,3,3,3,3 +6,14,8,20,3,13,1,7,12,2,18,9,4,5,19,16,10,15,11,17 +8,10,18,9,1,20,7,5,13,6,4,11,3,19,14,15,12,16,17,2 +2,8,5,18,1,No opinion,2,19,9,19,No opinion,5,6,19,18,19,20,No opinion,19,1 +3,20,8,20,20,5,20,1,4,6,20,7,20,20,20,20,20,2,20,20 +3,14,7,20,6,13,2,8,12,4,20,10,5,1,20,16,9,15,11,17 +6,20,10,20,3,20,1,10,20,2,20,15,4,5,20,20,15,20,15,20 +10,15,2,11,13,14,1,17,19,18,7,19,6,12,20,9,16,5,4,3 +7,8,10,No opinion,No opinion,11,2,No opinion,5,1,4,No opinion,9,6,No opinion,No opinion,No opinion,No opinion,No opinion,3 +20,20,20,6,2,4,20,20,20,5,20,20,1,20,20,3,20,20,20,20 +5,3,9,4,15,1,17,18,12,2,6,20,19,11,16,7,14,13,8,10 +4,No opinion,1,No opinion,No opinion,2,No opinion,No opinion,No opinion,5,No opinion,3,No opinion,6,No opinion,No opinion,No opinion,No opinion,No opinion,7 +1,11,4,15,5,6,8,9,2,7,20,17,3,10,13,12,19,14,16,18 +5,3,No opinion,No opinion,No opinion,4,No opinion,No opinion,No opinion,1,No opinion,No opinion,No opinion,No opinion,2,No opinion,6,No opinion,No opinion,No opinion +2,4,20,20,20,20,1,20,20,20,20,5,20,20,20,20,3,20,20,20 +4,20,5,6,12,7,8,11,3,9,14,1,2,18,10,13,17,16,15,19 +20,20,20,20,20,20,2,20,20,20,20,20,20,20,20,20,20,20,20,1 +No opinion,No opinion,No opinion,1,No opinion,No opinion,2,No opinion,No opinion,7,No opinion,No opinion,No opinion,6,No opinion,No opinion,No opinion,3,5,4 +5,20,20,20,20,20,20,20,1,2,20,20,3,20,20,20,20,20,20,4 +11,13,18,3,10,15,6,8,17,4,14,9,7,12,19,20,1,2,5,16 +1,19,19,19,19,4,20,5,2,19,19,19,3,19,19,19,19,19,19,19 +20,19,19,19,19,19,18,17,19,17,19,19,19,19,19,19,19,18,19,19 +6,14,8,20,3,13,1,7,12,2,18,9,4,5,19,16,10,15,11,17 +5,18,7,20,3,12,1,8,13,2,18,10,4,6,19,18,11,18,9,18 +2,No opinion,No opinion,No opinion,No opinion,No opinion,1,No opinion,No opinion,No opinion,3,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion +4,5,20,20,20,1,20,20,20,3,20,2,20,20,20,20,20,20,20,20 +3,12,6,17,1,15,13,14,9,5,19,7,2,20,10,18,11,8,16,4 +6,18,8,20,3,18,1,7,18,2,18,18,4,5,18,18,18,18,18,18 +1,No opinion,3,No opinion,No opinion,2,4,No opinion,No opinion,No opinion,No opinion,7,No opinion,5,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion +20,20,20,20,20,1,20,20,4,2,20,3,20,20,20,20,20,20,20,20 +10,3,4,No opinion,No opinion,12,No opinion,No opinion,No opinion,8,1,7,No opinion,No opinion,9,No opinion,1,5,6,11 +20,8,12,19,7,18,14,5,1,3,9,16,18,15,17,6,11,2,4,13 +5,15,6,20,12,10,16,7,18,2,1,14,19,17,8,11,13,4,3,9 +10,19,16,3,2,14,5,20,12,17,18,4,1,7,9,8,6,15,13,11 +6,20,5,20,4,20,20,1,8,7,20,20,3,20,20,20,9,2,20,20 +12,16,7,20,1,6,9,5,8,4,14,3,2,18,17,11,13,15,19,10 +8,20,20,20,1,20,3,20,20,4,20,20,2,6,20,7,5,20,10,9 +6,No opinion,No opinion,No opinion,2,No opinion,No opinion,20,No opinion,3,19,No opinion,1,18,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion +No opinion,1,No opinion,No opinion,No opinion,3,No opinion,No opinion,No opinion,6,5,No opinion,No opinion,4,8,No opinion,7,No opinion,No opinion,2 +4,8,18,15,5,6,1,12,13,14,16,19,3,10,17,11,7,20,2,9 +6,12,8,18,4,13,1,5,17,2,16,9,3,7,19,14,11,20,10,15 +5,20,8,20,3,12,1,7,12,2,20,9,4,6,20,20,9,20,9,20 +10,11,20,14,19,2,18,16,4,1,7,6,17,13,5,3,8,12,9,15 +No opinion,No opinion,No opinion,No opinion,1,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,1,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion +8,14,6,17,13,4,20,16,18,1,19,12,5,2,15,3,7,9,11,10 +1,20,8,20,7,16,13,20,4,5,6,14,2,10,11,3,9,12,20,15 +19,16,17,19,8,3,4,6,5,3,2,11,1,19,19,7,10,15,12,9 +3,20,20,20,2,20,20,20,20,20,4,7,1,6,5,20,20,20,20,20 +9,11,13,15,2,1,7,20,12,3,6,14,8,4,19,5,16,18,10,17 +6,20,8,20,3,9,1,10,20,2,20,7,4,5,20,20,20,20,20,20 +1,20,6,20,5,4,20,20,2,3,20,20,20,7,9,20,20,20,8,20 +19,19,4,19,2,19,19,5,3,19,19,19,1,19,7,19,19,19,19,6 +20,20,20,20,20,20,20,20,20,20,20,20,20,1,20,20,20,20,20,20 +20,1,20,20,20,20,4,20,20,20,1,20,20,20,20,20,1,20,20,20 +9,No opinion,6,13,3,18,1,16,8,2,16,14,10,7,19,18,17,11,20,4 +No opinion,No opinion,1,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,3,No opinion,2,No opinion,No opinion,No opinion,No opinion,No opinion,20 +4,20,3,20,5,20,20,20,20,20,20,6,20,20,2,20,20,20,20,1 +20,20,20,20,1,20,20,20,4,20,20,20,20,3,20,20,20,20,20,2 +6,8,1,No opinion,13,9,7,3,2,5,11,10,No opinion,14,4,No opinion,20,No opinion,No opinion,No opinion +10,7,12,18,17,8,15,4,2,5,6,3,9,19,1,16,11,15,15,20 +4,2,9,20,10,3,20,20,6,7,20,20,20,8,20,5,20,20,20,1 +5,19,6,20,9,1,No opinion,20,11,2,3,4,18,No opinion,20,12,10,7,20,8 +No opinion,No opinion,1,No opinion,No opinion,4,No opinion,No opinion,3,5,No opinion,No opinion,No opinion,2,6,7,No opinion,No opinion,No opinion,No opinion +5,20,5,20,3,20,4,20,2,20,20,20,1,6,20,20,20,20,20,20 +6,20,1,7,20,3,8,20,20,5,2,20,20,4,9,20,20,20,20,20 +12,11,1,14,2,10,20,3,4,7,9,13,6,8,15,19,16,18,17,5 +5,20,20,20,2,20,6,20,4,20,15,20,1,4,20,20,20,20,20,3 +8,15,10,19,7,6,1,2,3,12,17,5,4,9,18,20,14,16,11,13 +6,2,20,20,20,5,20,20,20,1,3,4,20,8,20,20,9,20,7,20 +5,8,6,9,2,18,4,20,15,16,20,7,1,12,11,17,10,19,13,3 +5,12,8,11,3,4,2,12,12,1,10,12,12,6,12,12,7,12,11,7 +2,9,12,20,5,20,4,11,8,6,20,20,3,7,10,20,20,20,20,1 +20,20,20,20,1,20,20,20,20,20,20,20,2,20,20,20,20,20,20,20 +5,20,6,20,3,20,2,20,20,20,8,20,1,7,20,20,20,20,20,4 +1,3,No opinion,No opinion,No opinion,1,3,No opinion,3,1,2,No opinion,No opinion,No opinion,2,No opinion,2,No opinion,No opinion,No opinion +1,20,15,11,5,4,13,7,8,6,16,2,3,9,10,12,18,14,17,17 +4,2,1,1,2,No opinion,2,1,2,1,1,No opinion,2,20,No opinion,No opinion,5,2,3,1 +20,20,20,20,20,20,20,20,20,20,20,20,20,1,20,20,20,20,20,20 +9,5,18,18,10,19,11,18,8,6,18,18,1,4,3,18,2,20,18,7 +5,9,6,9,2,8,1,6,8,3,9,7,4,4,9,9,7,9,7,9 +6,14,8,20,3,13,1,7,12,2,18,9,4,5,19,16,10,15,11,17 +19,19,5,19,1,6,19,19,3,19,19,19,2,20,19,19,19,19,19,4 +No opinion,No opinion,1,No opinion,No opinion,7,No opinion,No opinion,No opinion,2,6,No opinion,No opinion,No opinion,No opinion,4,No opinion,3,5,No opinion +6,14,4,11,5,16,2,1,8,3,17,20,18,19,15,13,10,12,9,7 +4,5,20,20,2,20,1,20,20,20,20,20,3,20,20,20,20,20,20,20 +10,20,20,20,20,1,20,20,20,2,20,20,20,20,20,20,20,20,20,10 +9,19,19,19,4,20,11,19,19,5,6,7,10,19,12,2,19,1,5,3 +3,No opinion,4,No opinion,3,No opinion,No opinion,2,1,2,5,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion +1,No opinion,2,No opinion,1,3,1,1,2,3,3,2,1,2,2,No opinion,No opinion,No opinion,No opinion,1 +2,20,20,20,20,20,20,20,20,20,20,20,3,4,20,20,20,20,20,1 +20,17,3,20,1,20,5,17,14,4,20,17,2,20,20,18,19,20,19,17 +6,12,8,20,3,7,1,5,9,2,13,16,10,4,20,15,11,14,17,20 +20,20,20,20,20,20,10,20,20,20,20,20,20,1,20,20,20,20,20,10 +18,11,3,16,4,7,1,9,6,10,2,5,19,20,17,15,13,12,8,14 +3,9,7,15,11,20,8,1,19,13,4,14,12,6,10,16,18,2,17,5 +2,20,3,2,2,20,20,20,20,20,1,1,2,2,3,20,20,20,20,20 +8,9,3,No opinion,1,10,4,7,10,10,6,No opinion,2,14,No opinion,10,No opinion,No opinion,No opinion,5 +20,1,11,20,15,3,6,8,5,7,15,9,20,4,15,2,13,10,16,12 +3,20,6,20,4,9,2,20,20,5,20,8,20,20,20,10,20,20,20,7 +6,20,20,20,2,20,3,20,7,7,20,5,4,20,20,8,20,20,7,1 +20,20,20,6,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,5 +3,13,8,20,2,16,1,6,12,4,18,9,7,5,19,15,10,14,11,17 +3,20,6,20,1,20,20,20,5,20,20,20,2,4,20,20,20,20,20,20 +6,19,19,1,19,4,19,5,19,2,19,3,19,19,20,19,19,19,19,19 +2,3,6,No opinion,No opinion,8,No opinion,1,7,9,No opinion,No opinion,No opinion,No opinion,No opinion,5,4,No opinion,No opinion,20 +3,4,No opinion,No opinion,8,9,No opinion,No opinion,11,5,10,6,15,1,7,No opinion,2,12,No opinion,13 +20,20,20,20,20,2,20,1,2,2,20,20,20,3,20,20,3,20,20,2 +5,10,3,12,6,14,9,1,7,4,13,17,19,11,15,20,8,2,16,18 +5,11,8,20,3,14,1,7,13,2,18,10,4,6,19,16,9,15,12,17 +12,8,14,No opinion,4,10,No opinion,9,7,2,11,No opinion,No opinion,No opinion,No opinion,6,5,1,3,13 +6,14,8,20,3,13,1,7,12,2,18,9,4,5,19,16,10,15,11,17 +6,20,20,20,3,20,1,20,20,2,20,20,4,5,20,20,20,20,20,20 +6,14,8,20,3,13,1,7,12,2,18,9,4,5,19,16,10,15,11,17 +6,20,8,20,2,20,1,5,12,3,19,20,3,6,20,18,20,20,20,17 +4,1,20,20,20,1,20,20,5,2,4,3,20,3,20,20,20,20,20,3 +3,4,No opinion,No opinion,1,No opinion,2,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,3,No opinion,No opinion,No opinion,No opinion,No opinion,3 +20,10,20,10,5,20,6,10,20,4,10,3,1,20,10,20,10,10,10,2 +10,20,1,20,1,2,10,1,15,15,4,4,1,20,10,20,15,15,20,15 +20,20,20,20,20,20,20,20,20,20,20,20,20,1,20,20,20,20,20,20 +2,5,19,7,3,9,1,6,10,11,5,5,5,12,20,10,17,4,8,9 +6,9,3,17,13,17,13,1,17,3,9,3,17,6,9,13,9,1,13,6 +2,No opinion,No opinion,No opinion,No opinion,No opinion,1,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion +6,14,8,20,3,13,1,7,12,2,18,9,4,5,19,16,10,15,11,17 +8,2,4,10,6,18,20,3,9,16,17,5,14,7,12,13,1,11,15,19 +5,7,20,20,2,20,3,5,20,20,20,20,1,20,20,20,20,20,20,3 +No opinion,No opinion,20,No opinion,No opinion,No opinion,20,No opinion,20,No opinion,No opinion,No opinion,20,1,No opinion,No opinion,No opinion,No opinion,No opinion,20 +20,14,13,16,1,12,3,17,8,4,19,10,2,18,18,11,7,15,6,5 +6,20,4,5,11,7,8,9,3,15,10,1,2,16,12,13,19,17,14,18 +1,No opinion,No opinion,No opinion,No opinion,2,4,5,No opinion,No opinion,No opinion,3,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion +4,No opinion,No opinion,No opinion,3,No opinion,1,No opinion,No opinion,2,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion,No opinion +19,17,19,6,20,19,20,1,19,19,16,17,20,3,17,2,17,18,17,16 +20,7,12,16,8,2,20,4,5,10,16,16,3,6,20,16,9,11,16,1 +3,4,5,20,20,20,2,20,20,3,20,20,20,1,20,20,20,3,2,4 +13,7,5,20,1,9,3,4,2,6,8,5,1,11,10,10,20,20,20,12 +7,20,20,1,20,5,4,8,20,20,6,7,2,10,9,20,20,20,20,3 diff --git a/events/elections/2017/README.md b/events/elections/2017/README.md new file mode 100644 index 00000000..0d17f68c --- /dev/null +++ b/events/elections/2017/README.md @@ -0,0 +1,77 @@ +# 2017 VOTERS GUIDE - KUBERNETES STEERING COMMITTEE ELECTION + +This election has been completed. + +[Results](RESULTS.md) + +# PURPOSE +The initial role of the steering committee is to instantiate the formal governance process for Kubernetes. In addition to defining the initial governance process, the [bootstrap committee](https://groups.google.com/forum/#!msg/kubernetes-dev/4e8WOnMvZC0/57GYmJKfDAAJ) strongly believes that it is important to provide a means for iterating the processes defined by the steering committee. We do not believe that we will get it right the first time, or possibly ever, and won’t even complete the governance development in a single shot. The role of the steering committee is to be a living, responsive body that can refactor and reform as necessary to adapt to a changing project and community. + +# VOTING +This election will shape the future of Kubernetes as a project and community. You will be voting on six (6) seats up for election, three with a two year term and three with an initial one year term. The motivation for these different terms is to set up an initial stagger so that the entire committee isn’t replaced every election cycle. After this first election, every elected community member will serve a two year term. + +The candidates you are voting on will be meeting regularly to strategically grow the project with contributors in mind. Most of the technical decisions, architecture planning, and the like come out of SIGs and other working groups. Some examples of responsibilities to consider as you are voting: +* Define, evolve, and defend the vision, values, mission, and scope of the project - to establish and maintain the soul of Kubernetes +* Charter and refine policy for defining new community groups (including Special Interest Groups, Working Groups, and Committees), and establish transparency and accountability policies for such groups. +* Define, evolve, and defend a Code of Conduct, which must include a neutral, unbiased process for resolving conflict + +Full scope of responsibilities, goals, and/or future election timelines, see [steering committee charter](https://github.com/kubernetes/steering/blob/master/charter.md). + +For more context, please see the [current issues](https://github.com/kubernetes/steering/blob/master/backlog.md) that need to be resolved or a previous governance [meeting video](https://www.youtube.com/watch?v=ltRKXLl0RaE&list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ&index=23) which lead to this whole process. + +## PROCESS +Elections will be held using time-limited [Condorcet](https://en.wikipedia.org/wiki/Condorcet_method) ranking on [CIVS](http://civs.cs.cornell.edu/) using the [Schulze method](https://en.wikipedia.org/wiki/Schulze_method). The top vote getters, modulo the corporate diversity requirement of no more than 1/3 of the committee from a single company, will be elected to the respective positions, with the top 3 filling the 2 year term seats, and the next 3 filling the 1 year term seats. + +You will be ranking your choices 1-20 with an option for no opinion. In the event of a tie, a coin will be flipped. + +The election will open for voting on September 19, 2017 at 09:00am PDT and end two weeks after on October 3, 2017 at 06:00pm PDT. You will receive an email to the address on file at the start of the election from Jorge Castro (jorge@heptio), Community Manager, please whitelist if necessary. Detailed voting instructions will be addressed in email and the CIVS polling page. + +## ELIGIBILITY +Members of Standing are defined by the union of: +* SIG leads +* Approvers and reviewers in any Kubernetes owned repositories +* Anyone with write access to a Kubernetes owned repository +* Active members of our community +If you believe you are a Member of Standing, please fill out [this form](https://docs.google.com/forms/d/e/1FAIpQLSeoeNSl9ufZ_jpp7OHgvtWm-GRFV6WUwTIqZ9W25eMd3xyyvg/viewform) <strong><em>before</strong></em> September 13, 2017. + +## DECISION +The newly elected body will be announced in the weekly Kubernetes Community Meeting on October 5, 2017 at 10:00am US Pacific Time. [Please join us](https://groups.google.com/forum/#!forum/kubernetes-community-video-chat). + +Following the meeting, the raw voting results and winners will be published on the [Kubernetes Blog](http://blog.kubernetes.io/). + +For more information, definitions, and/or detailed election process, see full [steering committee charter](https://github.com/kubernetes/steering/blob/master/charter.md). + +# NOMINEES +Don’t know someone on this list? We asked the nominees to provide a <300 word bio about their k8s experience that we'll link to their name in this table. + +Name | Organization/Company | GitHub +--- | --- | --- +[Aaron Crickenberger](aaroncrickenberger_bio.md) | Samsung SDS | [@spiffxp](https://github.com/spiffxp) +[Aaron Schlesinger](aaronschlesinger_bio.md) | Microsoft | [@arschles](https://github.com/arschles) +[Adnan Abdulhussein](adnanabdulhussein_bio.md) | Bitnami | [@prydonius](https://github.com/prydonius) +[Alex Pollitt](alexpollitt_bio.md) | Tigera | [@lxpollitt](https://github.com/lxpollitt) +[Caleb Miles](calebamiles_bio.md) | CoreOS | [@calebamiles](https://github.com/calebamiles) +[Derek Carr](derekcarr_bio.md) | Red Hat | [@derekwaynecarr](https://github.com/derekwaynecarr) +[Doug Davis](dougdavis_bio.md) | IBM | [@duglin](https://github.com/duglin) +[Ihor Dvoretskyi](idvoretskyi_bio.md) | CNCF | [@idvoretskyi](https://github.com/idvoretskyi) +[Ilya Dmitrichenko](errordeveloper_bio.md) | Weave | [@errordeveloper](https://github.com/errordeveloper) +[Jaice Singer DuMars](jaicesingerdumars_bio.md) | Microsoft | [@jdumars](https://github.com/jdumars) +[Justin Santa Barbara](vote_for_justinsb.md) | Independent | [@justinsb](https://github.com/justinsb) +[Kris Nova](kris-nova_bio.md) | Microsoft | [@kris-nova](https://github.com/kris-nova) +[Matt Farina](mattfarina_bio.md) | Samsung SDS | [@mattfarina](https://github.com/mattfarina) +[Michael Rubin](michaelrubin_bio.md) | Google | [@matchstick](https://github.com/matchstick) +[Michelle Noorali](michellenoorali_bio.md) | Microsoft | [@michelleN](https://github.com/michelleN) +[Phillip Wittrock](pwittrock_bio.md) | Google | [@pwittrock](https://github.com/pwittrock) +[Quinton Hoole](quintonhoole_bio.md) | Huawei | [@quinton-hoole](https://github.com/quinton-hoole) +[Rob Hirschfeld](rhirschfeld_bio.md) | RackN | [@zehicle](https://github.com/zehicle) +[Sebastien Goasguen](sebastiengoasguen_bio.md) | Bitnami | [@sebgoa](http://github.com/sebgoa) +[Timothy St. Clair](timothysc_bio.md) | Heptio | [@timothysc](https://github.com/timothysc) + + +<strong>Note:</strong>The bootstrap committee members have +recused themselves from any form of electioneering, including +campaigning, nominating, endorsing, or even asking people to run. + +Please direct any questions via email to <kubernetes-governance-committee@googlegroups.com><br> +[K8s Slack](http://slack.k8s.io/) + diff --git a/events/elections/2017/RESULTS.md b/events/elections/2017/RESULTS.md new file mode 100644 index 00000000..008ac2ff --- /dev/null +++ b/events/elections/2017/RESULTS.md @@ -0,0 +1,53 @@ +# Results of the 2017 Steering Committee Election + +Number of seats open: 3 (2 year term), 3 (1 year term) + +Number of eligible voters: 392 + +Number of votes cast: 309 + +Turnout: 78.8% + +[Raw ballot data](BALLOTS.csv) + +## Results + +The final ranking, using the "Schulze" Condorcet completion, is as follows: + +1. Derek Carr +2. Michelle Noorali +3. Phillip Wittrock +4. Michael Rubin +5. Timothy St. Clair +6. Quinton Hoole +7. Aaron Crickenberger +8. Caleb Miles +9. Jaice Singer DuMars +10. Kris Nova +11. Justin Santa Barbara +12. Alex Pollitt +13. Sebastien Goasguen +14. Ihor Dvoretskyi +15. Adnan Abdulhussein +16. Ilya Dmitrichenko +17. Matt Farina +18. Aaron Schlesinger +19. Rob Hirschfeld +20. Doug Davis + +According to the limits on company representation, Google would be +over-represented with this result, so Michael Rubin must be excluded. + +## Winners + +The winners of the open seats are as follows: + +Two year term: +1. Derek Carr +2. Michelle Noorali +3. Phillip Wittrock + +One year term: +1. Timothy St. Clair +2. Quinton Hoole +3. Aaron Crickenberger diff --git a/events/elections/2017/aaroncrickenberger_bio.md b/events/elections/2017/aaroncrickenberger_bio.md new file mode 100644 index 00000000..d3773272 --- /dev/null +++ b/events/elections/2017/aaroncrickenberger_bio.md @@ -0,0 +1,23 @@ +# Aaron Crickenberger + +I can be reached as [@spiffxp](https://github.com/spiffxp) on github, slack, gmail, linkedin, twitter, soundcloud, etc + +## What I've done + +I have been involved in open source projects since 2007, cloud related projects since 2009, and Kubernetes since 2015. + +I co-founded SIG Testing. I actively contribute in SIG Contributor Experience, Release, Scale. If you attend the weekly Kubernetes Community meetings, chances are you've seen me (or at least my beard.) + +I have participated in every Kubernetes release since v1.4. I drafted release notes for [v1.4](https://github.com/kubernetes/kubernetes/pull/33410) and [v1.5](https://github.com/kubernetes/features/pull/140). I am a member of of the [v1.8 release team](https://github.com/kubernetes/features/blob/master/release-1.8/release_team.md). + +I helped found [the kubernetes/community repo](https://github.com/kubernetes/community/pull/3). + +## What I'll do + +[The same thing we do every night Pinky...](https://www.youtube.com/watch?v=XJYmyYzuTa8) + +I will advocate for transparency and accountability in our decision making process. I will strive for simplicity and human-sized solutions to large-scale problems where possible. I will continue to push for community empowerment and ownership of key project responsibilities. Narf. + +## Where I work + +Samsung SDS, as part of the [Cloud Native Computing Team](https://samsung-cnct.github.io) diff --git a/events/elections/2017/aaronschlesinger_bio.md b/events/elections/2017/aaronschlesinger_bio.md new file mode 100644 index 00000000..84f043ec --- /dev/null +++ b/events/elections/2017/aaronschlesinger_bio.md @@ -0,0 +1,53 @@ +# Aaron Schlesinger + +- Github: [arschles](https://github.com/arschles) +- Twitter: [@arschles](https://twitter.com/arschles) + +# About + +I'm a Sr. Software Engineer, Microsoft Azure Containers Group. + +I'm a passionate engineer, teacher and leader with over 10 years software engineering and +architecture experience. + +I believe that Kubernetes will truly succeed when we improve the developer and +user experience. + +# Experience + +I've made contributions to [helm](https://github.com/kubernetes/helm), [helm charts](https://github.com/kubernetes/charts), [minikube](https://github.com/kubernetes/minikube), [service-catalog](https://github.com/kubernetes-incubator/service-catalog) and I am a co-lead of SIG-Service-Catalog. + +Outside of Kubernetes, I've spent over 10 years speaking at conferences, teaching Go & Scala, +building large systems, sitting on standards bodies, and contributing to other open source +projects. + +# Promises + +Here are my promises to the community should I be elected to the committee. + +Above all, I want Kubernetes to be a nice place to work. That means: + +- Engineers can build or improve things easily +- Engineers have clear expectations how & (generally) when they will get feedback on their work +- Engineers get constructive feedback on their work +- Everyone is treated with respect, and can resolve interpersonal problems amicably + +Holistically, most of these points are true most of the time, but Kubernetes is growing so +fast and is so diverse that all of them are not always guaranteed right now. + +I realize no community can achieve everything all the time, but I believe we can do better +together. + +I want to improve the following immediately: + +- Improve user _and_ developer documentation +- Create clearer guidelines on how reviews should happen +- Clarify how to rise as a contributor + +Additionally, I'll contribute best to the committee as a representative of end users +and developers. + +I will keep the same promises in the steering committee as I do in SIG-Service-Catalog: + +- **I will always solicit user and developer experiences before making decisions** +- **I will always tackle user and developer problems before anything else** diff --git a/events/elections/2017/adnanabdulhussein_bio.md b/events/elections/2017/adnanabdulhussein_bio.md new file mode 100644 index 00000000..81371e5e --- /dev/null +++ b/events/elections/2017/adnanabdulhussein_bio.md @@ -0,0 +1,46 @@ +# Adnan Abdulhussein + +- GitHub/Slack/Twitter: @prydonius +- Email: adnan@bitnami.com + +## Background + +I joined the Kubernetes community in 2016, and have contributed in a number of +capacities since: + +- Core contributor of Helm +- Co-leading the Kubernetes Charts project +- Kicked off the Helm incubation process and community docs +- Co-leading SIG Apps +- Regularly speaking and advocating for Kubernetes + +Prior to my involvement in Kubernetes, I have been an active maintainer of +Bitnami's open source container images. My work at Bitnami is heavily focused on +making containers and Kubernetes both accessible and easy to use. + +I am honoured to be amongst the amazing group of people nominated for the +Steering Committee. If elected, my goal will be to help Kubernetes grow into a +rich and diverse community. + +## Where I see myself contributing + +As someone who isn't completely focused on Kubernetes core, I think an important +part of the project is the ecosystem around it. I am deeply interested in being +part of discussions, as part of the Steering Committee, to define the vision of +the project, and to decide what relevant sub-projects should be part of +Kubernetes to continue building a strong ecosystem and brand. This also means +making sure sub-projects have the resources they need (i.e. CI, documentation +websites) to succeed and promote their roles in the overall project. + +The diversity and inclusivity is one of the things that I value the most about +the Kubernetes community. I think it's important that we uphold these values +when defining and practicing a Code of Coduct and refining the contributor +experience. + +Another area of the Steering Commitee I'm interested in is furthering the +transparency and accountability for SIGs. I believe improving the communication +and enabling more knowledge sharing will empower both community and consumers. + +## Where I work + +Bitnami diff --git a/events/elections/2017/alexpollitt_bio.md b/events/elections/2017/alexpollitt_bio.md new file mode 100644 index 00000000..2dbd3f37 --- /dev/null +++ b/events/elections/2017/alexpollitt_bio.md @@ -0,0 +1,27 @@ +# Alex Pollitt + +I am excited to have been nominated for the steering committee. I recognize that doing +the role justice will be a demanding and time consuming commitment, but I would like +to do my part to help maintain and foster the growth of this amazing community of +contributors and users. + +For those of you who don’t know me, I am one of the founders of Tigera, the company behind +Calico, and active contributors to flannel, CNI, Kubernetes, and Istio. I’ve been working +with the Kubernetes community since 2014 and have great respect for everyone I’ve had the +chance to get to know along the way. + +Since the early days of the project, when friends used to ask, “which container orchestrator +would you bet on?”, my answer was always Kubernetes – not because of the great design +principles, but because of how the founding team and those already contributing welcomed +others to take part and collaborate with an equal voice to move the project forward. For me, +“Kubernetes” means the people and how they work together, as much as technology. + +One example I experienced was the k8s-sig-net efforts to agree a network policy API. In +other communities, this might have descended into vendors trying to get one up on each +other. Instead there was true collaboration, focusing on technical merit and meeting the needs +of future users. It was awesome to play a part in that process and even though it was just one +small corner of Kubernetes it’s a great illustration of why I love working in this community. + +If elected, I promise to do my utmost to foster and facilitate this kind of diverse, vendor +neutral, merit based collaboration, empower and support contributors to shape the project, +and maintain the spirit of this awesome community as it grows. diff --git a/events/elections/2017/calebamiles_bio.md b/events/elections/2017/calebamiles_bio.md new file mode 100644 index 00000000..452035a8 --- /dev/null +++ b/events/elections/2017/calebamiles_bio.md @@ -0,0 +1,52 @@ +# caleb miles + +## Contact Information + +- GitHub/Slack: @calebamiles +- Email: caleb.miles@coreos.com + +### Problem solving style + +I am a huge fan of ideas which simplify and unify, possibly due to my +undergraduate study of physics and mathematics. I believe that collaboration is +incredibly important and that continuous feedback on incremental solutions are +likely to produce the best outcome. I therefore try spend a lot of time talking +with people about the problems that are important to them before stepping back +to try to introduce a minimal solution based on their concerns. + +### Problems that are important to me + +I believe that addressing the dual challenge of communicating what everyone is +working on to other developers, and then explaining that work to users is one +of the most pressing issues for Kubernetes today. + +I also believe that we need to invest in the contributor experience particularly +by better supporting new contributors with mentorship programs, especially for +the self taught, new graduates, and underrepresented communities. If we are +going to be a successful and sustainable project we need to ensure that we +maintain a healthy pipeline of new contributors while at the same time reducing +the workload for more established contributors. + +Scaling contributions is another serious challenge for the project and I am very +interested in working on helping to support the effort to rethink the Kubernetes +release process. + +### Roles held + +I joined the community in 2016 since then I have made shallow contributions to + +- SIG Contributor Experience +- SIG Release +- SIG PM +- SIG Testing + +as well as serving in a minor capacity in the 1.5, 1.6, 1.7, and 1.8 releases. I +currently help to facilitate + +- SIG PM, co lead and co founder +- SIG Release, co lead and co founder + +### Where I work + +CoreOS + diff --git a/events/elections/2017/derekcarr_bio.md b/events/elections/2017/derekcarr_bio.md new file mode 100644 index 00000000..4f0f757a --- /dev/null +++ b/events/elections/2017/derekcarr_bio.md @@ -0,0 +1,43 @@ +# Derek Carr + +- GitHub: @derekwaynecarr +- Employer: Red Hat + +## Roles Held + +- Co-founder, lead: sig-node, wg-resource-management +- Participate: api-machinery, autoscaling, federation, scheduling, + service-catalog + +## About me + +I joined in 2014 as an early external contributor. After the first developer +summit, I realized I had a lot to learn from the experience of others. My first +PR enabled people to contribute to the project by running Kubernetes in a VM. +Mentors in the community helped multiply the quality of my contributions. With +their help, I designed and implemented namespaces, quota, limits, and much of +the supporting internal machinery like admission control, indexers, and clients +prior to the 1.0 release. + +As the project grew, SIGs developed to support specialized collaboration. I +co-lead SIG node and advocate for expanded workload support and improvements to +node reliability. To improve cross-SIG collaboration, I co-founded the Resource +Management Working Group to define a multi-release roadmap across SIGs (node, +scheduling, storage, etc). The cross-SIG working group concept has been adopted +across the community. I participate in multiple incubator projects with code +and coaching on technology and process. As every maintainer acknowledges, it is +impossible to know everything about Kubernetes. I prioritize mentoring others to +help them grow their scope and sustain the project. + +My experience reflects both a breadth and depth of commitment across Kubernetes +that would benefit the committee. I have a pragmatic approach to problem +solving that builds from the expertise of others and adopts minimal solutions +that have the best potential outcome. I iterate and improve. As a member of +the community, I hope the steering committee adheres to the same general +approach. + +## Where I see myself contributing + +- Responsibly grow contributor pool across all disciplines +- Clarify scope, structure, and responsibilities of SIGs and roles within +- Enhance Code of Conduct to nurture an inclusive, diverse, and open community
\ No newline at end of file diff --git a/events/elections/2017/dougdavis_bio.md b/events/elections/2017/dougdavis_bio.md new file mode 100644 index 00000000..4a80fd85 --- /dev/null +++ b/events/elections/2017/dougdavis_bio.md @@ -0,0 +1,40 @@ +# Doug Davis + +- Github: [@duglin](https://github.com/duglin) +- Works for: IBM + +## Background + +I've been working on open source projects, and standards, for about 17 years, +starting with being a co-initiator of the Apache Axis project; working on +WS-* related projects and specifications, OpenStack, CloudFoundry, Docker +and now Kubernetes, including as a co-lead of the Service-Catalog Incubator. +I also founded the Web Services Testing Forum, a consortium of Web Service +providers and consumers designed for interop testing of SOAP/WS-* +specifications, as well as the [soaphub](http://soaphub.org) on-line +collaboration chat tool used by several communities. + +Throughout this time the importance of an open, fair and streamlined +collaboration process has really been a center piece of my activities and +goals for these projects. + +## Steering Committee + +The stated scope of the Steering Committee focuses on helping the +Kubernetes community define and optimize many of the (sometimes less than +glamorous, but necessary) processes and infrastructure that's critical +to keeping a community like Kubernetes healthy and successful. + +Kubernetes, as one of the fastest growing OSS projects, needs to navigate +and manage the complexities associated with this explosive growth very +carefully. The importance of balancing the need for continued enhancements, +managing stability of the code base, all while ensuring the community of +developers and users feel their contributions (from code, issues and feedback) +are welcome, respected and addressed in a timely fashion can not +be understated. This will require people who can consider the challenges +from multiple perspective, not just from a developer's. + +I believe that my background in the open communities I've been involved with +have given me a good perspective on what works, what doesn't and which might +be best for an organization like Kubernetes. + diff --git a/events/elections/2017/errordeveloper_bio.md b/events/elections/2017/errordeveloper_bio.md new file mode 100644 index 00000000..a667a2b4 --- /dev/null +++ b/events/elections/2017/errordeveloper_bio.md @@ -0,0 +1,33 @@ +# [Ilya Dmitrchenko](https://github.com/errordeveloper) + +## Manifesto + +Ilya has been contributing to Kubernetes since 2014. The focus of his contributions has been on solving +general cluster provisioning and bootstrap problems and making networking easier. For example in 2016, +he did the groundwork on the initial version of `kubeadm`. + +Ilya is a DX engineer at Weaveworks and is based in London. From there he travels (mostly in Europe) to +educate the wider community about the benefits of containers and orchestration with Kubernetes. Ilya +would be delighted to represent the European users as a member of the steering committee. + +Most recently, he started participating in organised community activities whose aim is to solve the +problems and the needs of end users who have Kubernetes in production or who are looking to put it in +production. To that end, he is a member of the London Production User Council, and together with other +contributors launched Kubernetes Office Hours. + +To learn more about Ilya's background and why he deeply cares about the community, please take a look at +his [personal blog](https://medium.com/@errordeveloper/a-little-more-about-me-b0b6238ba7f8). + + +## Key Priorities + + - Customer success with open-source Kubernetes + - Ease of use, in majority of general cases + - Growth of the community, and fairness + - Social and technical diversity + +## More Info + + - [Personal projects](https://github.com/errordeveloper) + - [Tweets](https://github.com/errordeveloper) + - [Personal blog](https://medium.com/@errordeveloper/a-little-more-about-me-b0b6238ba7f8) diff --git a/events/elections/2017/idvoretskyi_bio.md b/events/elections/2017/idvoretskyi_bio.md new file mode 100644 index 00000000..c4e8c002 --- /dev/null +++ b/events/elections/2017/idvoretskyi_bio.md @@ -0,0 +1,24 @@ +# Ihor Dvoretskyi + +Twitter: https://twitter.com/idvoretskyi + +GitHub: https://github.com/idvoretskyi + +Anywhere else: @idvoretskyi + +## About me + +I'm a [Developer Advocate for Cloud Native Computing Foundation](https://www.cncf.io/blog/2017/09/18/meet-cncfs-newest-developer-advocate/), open source lover, passionate about the technologies and open source adoption; with the deep technical background, together with program and product management experience in the open source world. In a previous life, I’ve been a System Administrator and DevOps Engineer, passionate about the Cloud technologies. Later, I’ve joined Mirantis, one of the largest players in the OpenStack world, where I’ve started my collaboration with OpenStack community. + +I've enjoyed Kubernetes since the early days. I've started researching it in mid-2014 when it was publicly announced; I've been using for my projects it in mid-2015 when it had reached its 1.0 milestone; and I've started contributing (as an individual contributor) to Kubernetes in late-2015, when I felt that my knowledge and experience might be useful for the project. + +As a Kubernetes Community member, I'm mostly focused on Product- and Project-management areas, with the primary interest in the features and roadmap scope. +For Kubernetes 1.3, 1.4 and 1.5 I've been unofficially involved in the release process with the features tracking and management. +Since Kubernetes 1.6 release cycle (and continued these efforts during 1.7 and 1.8), I've joined the release teams as the release manager, responsible for the features. +Finally, I've been one of the co-founders of [SIG-PM](https://github.com/kubernetes/community/tree/master/sig-product-management), responsible for managing Kubernetes-as-a-project and Kubernetes-as-a-product. + +Besides that, as an individual with OpenStack community background, I'm co-leading [SIG-OpenStack](https://github.com/kubernetes/community/blob/master/sig-openstack/README.md), building the relationships between two major open source communities. + +## What's next? + +As a vendor-neutral Kubernetes contributor, I'm going to continue my work on enhancing Kubernetes as an open source project, technology stack, and one of the largest open source communities in the world. diff --git a/events/elections/2017/jaicesingerdumars_bio.md b/events/elections/2017/jaicesingerdumars_bio.md new file mode 100644 index 00000000..90ba0ee3 --- /dev/null +++ b/events/elections/2017/jaicesingerdumars_bio.md @@ -0,0 +1,20 @@ +# Jaice Singer DuMars + +## Contact Information + +- GitHub: [@jdumars](https://github.com/jdumars) +- Twitter: [@jaydumars](https://twitter.com/jaydumars) +- LinkedIn: [Jaice Singer DuMars](https://www.linkedin.com/in/jasondumars/) +- Works for: Microsoft + +## Why me? + +The steering committee represents a unique convergence of my passion for servant leadership, organizational dynamics, inclusion, advocacy, and process optimization. It's always my highest purpose to be a voice and advocate for those I serve. And, if elected, I will faithfully serve the ever-changing, sometimes divergent, and always important needs of the Kubernetes community. + +## Servant leadership + +Since my first moments in the community, I have sought out every opportunity I can to make things better. I wrote an article about this called [What Kubernetes means to me](http://bit.ly/k8s2me) that will shed some light on my journey to this point. My contributions to the community currently include leading the 1.8 release team, co-leading SIG Azure, SIG Architecture, and SIG Cluster Ops (although Rob Hirschfeld has been covering for me while I work on the release). I have facilitated every release retrospective since 1.3, and have improved my typing skills by note taking at countless SIG meetings. I also work behind the scenes on Community things with Sarah and Jorge. + +## Career + +I've been in love with systems, networks, and the human elements of interconnectivity since I was a SysOp on a BBS as a teenager. My first hack was in grade school, making the Oregon Trail game say naughty things. In the time since, I have been a systems and network engineer, and held several engineering management directorships at companies large and small. At Microsoft, all I do is Kubernetes, so I definitely have the organizational support to carry out my duties on the committee. diff --git a/events/elections/2017/kris-nova_bio.md b/events/elections/2017/kris-nova_bio.md new file mode 100644 index 00000000..371379d4 --- /dev/null +++ b/events/elections/2017/kris-nova_bio.md @@ -0,0 +1,36 @@ +# Kris Nova + +## What is important to me + +Mental, philosophical, and design diversity. Period. + +I truly believe that the best teams and projects are those that embrace unique and sometimes conflicting ways of thinking. + +Having a leadership board of people who all think the same way, and all are moving in the right direction is stale and stagnant. + + - I want to help foster diversity in Kubernetes in both our design process, but also our goals and philosophies. + - I want to encourage my peers to put themselves into difficult, or testing situations in order to help grow and mature. + - I want to let others know that it's okay to try something and fail as we all learn as a scientific community. + + +## Work in the community + + - I spend as much of my free time as humanly possible contributing to Kubernetes. + - I take impromptu video calls with total strangers. + - I helped run *sig-aws* even though it conflicted with my dayjob. + - I created [kubicorn](https://github.com/kris-nova/kubicorn) in my free time and it has been reviewed as one of the best open source projects to contribute to in the space. + +## Reasons to pick me + +## I think differently. + +I question the world around me, and have been in a ton of different roles in engineering space. Including time in both genders. + +## I believe that software comes first. + +I love kubernetes regardless of my employer, and will always contribute because I believe in it. +I believe the way we continue to keep the project great is to focus on community, as well as writing kick ass software with a great user interface. + +## tldr + +I embrace being different and want to clean up our code and improve user experience.
\ No newline at end of file diff --git a/events/elections/2017/mattfarina_bio.md b/events/elections/2017/mattfarina_bio.md new file mode 100644 index 00000000..2f4e6db9 --- /dev/null +++ b/events/elections/2017/mattfarina_bio.md @@ -0,0 +1,23 @@ +# [Matt Farina](https://www.mattfarina.com) +Technical leader, coder, author, presenter + +## Governance and Organization + +The steering committee is [chartered](https://github.com/kubernetes/steering/blob/master/charter.md) with figuring out the governance and organization of the Kubernetes community. Matt has years of experience in both non-profit and open source organizations. He has served on the board for two non-profits along with being involved in open source projects with mature governance models (e.g., Drupal and OpenStack). + +Matt believes that governance should enable people to easily navigate an organization and be empowered to be part of the community. That groups within an organization, such as special interest groups (SIG) and working groups, should have clearly communicated scope. This helps everyone understand how the organization is laid out while helping to highlight gaps. + +He also believes in diversity. This includes diversity of ideas and uses for Kubernetes. Diversity brings out ideas that can often be missing in an echo chamber of similar people with similar uses cases. Diversity leads to a stronger Kubernetes that's more capable of meeting real world needs. + +Matt wants to see Kubernetes as a professional community with an enjoyable experience for both contributors and consumers. + +## Technical Background + +Matt has been a contributor of open source for 15 years and a consumer of open source for over 20 years. In that time he has contributed to projects such as [Glide](https://glide.sh) (package manager), [Drupal](https://drupal.org), [OpenStack](https://www.openstack.org), and many others. While working in and out of open source Matt has had a focus on both software development and _user experience_. + +He has also authored technical books, hosted podcasts, and presented at numerous conferences. This has taught him how to clearly communicate with a wide variety of people. + +## Quick Details + +* Co-founded and co-leads SIG Apps +* Works for [Samsung SDS](https://samsung-cnct.github.io) diff --git a/events/elections/2017/michaelrubin_bio.md b/events/elections/2017/michaelrubin_bio.md new file mode 100644 index 00000000..09a4c5f7 --- /dev/null +++ b/events/elections/2017/michaelrubin_bio.md @@ -0,0 +1,36 @@ +# Michael Rubin Bio + +GitHub: @matchstick + +## About Me + +I have been a software engineer, technical manager, and leader for +~20 years. I am passionate about helping projects and people thrive. +Today I lead a number of Kubernetes teams at Google, including Storage, +Networking, Multi-cluster, Node, and more. + +I love coding and technology, but even more than that I love mentoring +others and helping them to have impact and grow professionally. This is +why I choose to support teams and projects as a technical manager. + +My leadership style is to bring sharp focus to technical efforts, and +to help others become successful leaders, themselves. I find that many +technology problems are not purely technical, and one of my biggest +strengths is my ability communicate, connect with people, and get to +the root of a problem. People are not machines, and they need different +skills than just pure engineering. + +During my 1.5 years with Kubernetes, I have worked directly and +indirectly with several SIGs - Storage, Multi-cluster, Networking, +Scheduling, and Node. I think I have brought clarity and structure to +the efforts, helped organize priorities and roadmaps, and helped the +teams and individuals grow. I have built lasting relationships with +the people of Kubernete, and not just the code. + +As a member of the Steering Committee, I would focus on: + + * Cross SIG efforts and how to co-ordinate them + * Getting more things out of core, but keeping them part of Kubernetes + * Maintain the cool, open and inviting environment we have today + * Delegate and empower others by adding structure (not process) + diff --git a/events/elections/2017/michellenoorali_bio.md b/events/elections/2017/michellenoorali_bio.md new file mode 100644 index 00000000..6d9faa0e --- /dev/null +++ b/events/elections/2017/michellenoorali_bio.md @@ -0,0 +1,35 @@ +# Michelle Noorali + +GitHub: [@michelleN](https://github.com/michelleN) + +Twitter: [@michellenoorali](https://twitter.com/michellenoorali) + +LinkedIn: [Michelle Noorali](https://www.linkedin.com/in/michelle-noorali-a588b516) + +## History & Roles Held + +- Working with Kubernetes and have been part of the community since mid-2015 +- Co-founded and Co-lead Kubernetes SIG Apps for focusing on defining and managing applications in Kubernetes +- Core Maintainer of the Kubernetes Helm project +- Co-chair CloudNativeCon & KubeCon 2017 + +## Why I'm here + +The Kubernetes community has been a special place for me, and I'm glad to be part of it. I have found it exciting to work collaboratively with folks from different organizations, backgrounds, parts of the world, and industries on problems that matter to all of us sometimes even for different reasons, so I enjoy and place great value on communicating effectively and balancing different perspectives for technical solutions. + +Because Kubernetes is such an empowering piece of technology, I am obsessed with helping make sure that it is usable across audiences. This is where the bulk of my work with [SIG-Apps](https://github.com/kubernetes/community/tree/master/sig-apps) comes in. We've been working for almost a year and a half on making the process of defining and managing applications in Kubernetes easier and the best experience possible. Throughout this process, we've repeatedly re-defined how our SIG functions and how it can best serve the community. We focused on creating a tight feedback loop with the folks who consume apps features in Kubernetes and those who help create and maintain those features. We share solutions to common sets of problems, and we ensure that the solutions presented are set up for success by helping our presenters set the right context for the audience. We even focus on the flow of our meetings to make sure that people are able to follow along making the SIG content digestable, empowering newcomers in the space, and helping funnel more opinions into feedback loops. I believe SIGs are a fundamental part of the Kubernetes community and one of the big reasons the community is so successful. I am particularly passionate about supporting SIGs and making them even stronger than they are today using my experience with SIG Apps. + +Lastly, it has been rewarding for me to help people begin their journey in the Kubernetes community by guiding them to the right place to get what they need. I've enjoyed being part of the growth of this community and want to ensure that it remains an approachable one as well as an inclusive place to work. + +## Key priorities + +- Making sure people are heard +- Making sure opinions are respected +- Helping ensure the right context is set when discussing concerns and ideas +- Helping people be successful in whatever area they are in +- Empowering newcomers +- Empowering SIGs and SIG leads + +## Where I work + +Microsoft Azure, previously Engine Yard/Deis diff --git a/events/elections/2017/pwittrock_bio.md b/events/elections/2017/pwittrock_bio.md new file mode 100644 index 00000000..a6defb9b --- /dev/null +++ b/events/elections/2017/pwittrock_bio.md @@ -0,0 +1,55 @@ +# Phillip Wittrock Bio + +GitHub: @pwittrock + +## Problem solving style + +- Incremental approach to problem solving +- Try new things - build and iterate on minimalist solutions + +## What I find important + +I care deeply about helping people feel positive about what they are doing and empowered to do more. +Within the Kubernetes community, I would like to focus on helping ensure contributors': + +- time and contributions are impactful +- time and contributions are recognized and appreciated +- opinions are listened to and valued + +## Where I see myself contributing + +I believe it is important for the processes for contribution and maintenance of the project +to be well communicated and understood within the community. +I would like to be involved in areas owned by the SC such as - helping to develop a +SIG charter template, simplifying the contributor ladder, documenting resources the community has, +and helping to find a home for unowned areas of the project. + +## Roles held + +Since joining the community in 2015, I have made contributions in various roles +(developer, SIG-lead, docs, release manager) across the project. Working +in these capacities has given me insight into the challenges faced throughout the project. + +Areas of the project I have contributed to include: node, kubectl, service catalog, docs and release. + +### Current SIG lead positions I hold + +- Co-founded SIG-cli +- Co-founded SIG-release + +### Emeritus SIG lead positions I have held (brief tenure) + +- Bootstrap SIG-contribX +- Bootstrap SIG-docs + +### Relevant non-technical contributions + +- (release) Changed management role from an individual to a team with defined roles +- (release) Proposals for streamlining and automating various processes +- Published community membership ladder developed from draft (draft by Brian Grant) +- 1.4 release czar +- Participated in 1.5, 1.6 and 1.7 releases + +## Where I work + +Google
\ No newline at end of file diff --git a/events/elections/2017/quintonhoole_bio.md b/events/elections/2017/quintonhoole_bio.md new file mode 100644 index 00000000..d72d6aa0 --- /dev/null +++ b/events/elections/2017/quintonhoole_bio.md @@ -0,0 +1,66 @@ +# Quinton Hoole + +- Github: [quinton-hoole](https://github.com/quinton-hoole) + +# About Me + +Currently I'm Technical Vice President of Cloud Computing At Huawei +Technologies (185,000 employees globally, US$75BN annual revenue +(2016), 32% annual revenue growth). + +I was the founding engineer of Amazon EC2 (2005-2010). + +I was the lead engineer at Nimbula.com (2010-2012), a Cloud IaaS +startup which was acquired by Oracle in 2013. + +I was at Google from 2012-2016, first as Technical Lead and Manager of +Ads Serving SRE, and then Engineering Lead on the Kubernetes team +(2014-2016). + +Prior to 2005, I co-founded two startups (one in the online travel +industry, the other in financial services), on both of which I served +as lead techie until they were acquired by public companies. I have +also worked as a consultant, contractor and technical leader in the +telco, financial and retail industries. + +I have an M.Sc in Computer Science. + +# Highlights of What I've done on Kubernetes thus far + +Initially I led the effort to get our CI Testing stuff initiated and +[working effectively](https://github.com/kubernetes/community/blob/master/contributors/devel/writing-good-e2e-tests.md), which was critical to being able to launch v1.0 successfully, +way back in July 2015. Around that time I handed it over to the now-famous SIG-Testing. + +After that I initiated and led the [Cluster +Federation](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/federation/federation.md) +effort, which has spilled over and touched many aspects of Kubernetes +in one way or another (API, Networking, Security, Scalability, Resiliency, +Testing etc). + +Along the way I also helped the [Scalability folks to set some key +objectives](https://github.com/kubernetes/community/blob/master/sig-scalability/goals.md). + +# How I Hope to Serve You on the Steering Committee + +I am a strong believer that Cloud Computing will change the world +(even more so than it already has), and that Kubernetes is the next +big step in this exciting journey. + +To realize this potential, I believe that we need to create both a +thriving, collaborative community, and freaking awesome software that +people love to use, everywhere. We're doing pretty well in both of +these areas, but there's still a lot of room for improvement. So I'd +like to offer my broad and deep experience, time and passion to help +us to get there. Hopefully my track record will help to convince you +that I'm fairly good at identifying good things for us to do, figuring +out how best to do them, and getting them done. Along the way I have +accumulated more than a few scuffs and bruises, as well as a few +victories, so hopefully I can help us to avoid some of the former. + +More specifically, I'd like to help us to (amongst others): + +- ensure that Kubernetes remains a fun place to develop stuff that + people love to use. +- create effective incentives to motivate constructive, productive + and sustainable progress. +- protect what we have built (and will build) from bad things happening. diff --git a/events/elections/2017/rhirschfeld_bio.md b/events/elections/2017/rhirschfeld_bio.md new file mode 100644 index 00000000..c3994be7 --- /dev/null +++ b/events/elections/2017/rhirschfeld_bio.md @@ -0,0 +1,27 @@ +# Rob Hirschfeld Bio + +* GitHub: [@zehicle](https://github.com/zehicle) +* Twitter: [@zehicle](https://twitter.com/zehicle) +* Blog: [RobHirschfeld.com](http://robhirschfeld.com) +* I work at [RackN.com](https://RackN.com) and live in Austin, Texas. + +## Governance Perspective: it's about people, not tech + +Open source infrastructure automation is a critical foundation for the Internet and, thus, advancing society as a whole. In a very practical way, protecting open software is essential to building a better world. I am seeking a seat on the Kubernetes Steering Committee because I bring special perspectives to governing the project. + +## Leadership: Technical, Open Source and Business + +I’ve demonstrated practical and relevant experience to building open governance at this stage in our development. + +* **Open source project founder ([Digital Rebar](http://rebar.digital), Crowbar)**: I know the challenges of sustaining contribution +* **Operator**: I’ve run infrastructure and founded the SIG Cluster Ops group to give a voice for operators. I focus professionally on helping promote SRE and DevOps practices. +* **Founding OpenStack board member (4 terms)**: I’ve built open governance processes and done the hard work to build conformance and core definition (#DefCore) patterns. +* **Start-up founder (RackN.com)**: I’ve done every role and am willing to get my hands dirty. I understand the needs for small companies trying to support large projects. +* **Dell Executive**: I’ve been part of large organizations and know how to explain things in corporate speak to protect project interests. +* **Technologist**: I still write code and build applications as needed so I understand the very specific concerns of delivering working platforms. + +## TL;DR: Collaborative and Principled + +Most importantly, I’m collaborative by nature. I know how to stand my ground and fight without being a jerk or excluding people. I can be one of the advocates that the Kubernetes community needs as we build formal governance. + +_Governance means being able to say no and keep consensus._ diff --git a/events/elections/2017/sebastiengoasguen_bio.md b/events/elections/2017/sebastiengoasguen_bio.md new file mode 100644 index 00000000..065bf8eb --- /dev/null +++ b/events/elections/2017/sebastiengoasguen_bio.md @@ -0,0 +1,30 @@ +# Sebastien Goasguen + +## Contact + +* GitHub: [@sebgoa](https://github.com/sebgoa) +* Twitter: [@sebgoa](https://twitter/sebgoa) +* Linkedin [Profile](https://www.linkedin.com/in/sebastien-goasguen-382b7b21/) +* Employer: Bitnami + +## Vision + +Being a member of the steering committee is not a technical role, it is a temporary position to help the community grow and ensure that the project (at large) succeeds in the long term. + +The steering committee already has a long backlog of things to do and I submitted a [PR](https://github.com/kubernetes/steering/pull/2) some time ago to show my take on it. Within that backlog, I see three main areas that need a lot of clarity. + +* Membership. Our membership and roles have grown organically, driven by our velocity and the limitations of GitHub. I would like to see us step back a bit and clearly define our membership ladder and ways by which contributors navigate it. This would provide clear criteria for example to define who is a "person of standing" and how you become one. + +* Decision making. We have a lot of slack channels, hangouts and mailing lists. However we do not know where decisions are made, when and how. We need to bring clarity to our decision making processes so that everyone feels that they own the decision. It is also a major issue of inclusiveness which can lead to fragmentation of the community. + +* Incubator. We have an incubator but I believe we need to revise its "charter" now that several projects have graduated. We need more mentoring and support of the incubating projects. Moreover, the graduation criteria, need to encompass a more visible decision process (currently only 2 people decide if a project graduates). + +My experience at the Apache Software Foundation (member and part of several project management committee) can be extremely helpful. Not to apply the exact same principles and processes but to find a balance and bring a perspective from an established open source foundation that is totally vendor neutral, based on meritocracy, has an incubator and clear decision making processes. + +## Diversity + +My name is in a French dialect (Breton), it literally means "white man". However Kubernetes is a worldwide community and I believe it needs representation from different nationalities, cultural background and from people who do not necessarily speak English as their mother tongue. I believe I can be that little voice which brings a bit of balance in an otherwise heavily US dominated steering committee. + +## Kubernetes Background + +I have been involved with Kubernetes since [Sept 2014](https://github.com/kubernetes/kubernetes/pull/1411) and focused on tools in the ecosystem. My startup [Skippbox](https://github.com/skippbox) created several tools including: kompose, kmachine, Cabin and [kubeless](http://kubeless.io). We joined Bitnami in March 2017 where I lead all our Kubernetes efforts. I helped very early on with Helm and Charts. I try to help with the Python client and mentored a GSoC student this summer. I also do a fair share of advocacy and training and I am the co-author of the upcoming Kubernetes cookbook. diff --git a/events/elections/2017/timothysc_bio.md b/events/elections/2017/timothysc_bio.md new file mode 100644 index 00000000..9daeee9f --- /dev/null +++ b/events/elections/2017/timothysc_bio.md @@ -0,0 +1,26 @@ +# Timothy St. Clair + +## Contact Information + +- GitHub/Slack: [@timothysc](https://github.com/timothysc) +- Email: tstclair@heptio.com +- Twitter: [@timothysc](https://twitter.com/timothysc) + +## Background + +I've been involved in the Kubernetes project since 2014, and prior to that I was an active member in the Mesos community, with over 10+ years of experience working on open source projects across a plethora of communities. During my tenure, I've been lucky enough to meet a number of great folks working on various SIGs: + +- SIG Scheduling & Testing (Lead) +- SIG Scale, Cluster Life-cycle, API Machinery, Node (Contributor) + +## Why I want to be on the Steering Committee (Optimize, Simplify, and Empower) + +Time is our most precious commodity, and few things can be more frustrating than wanting to contribute only to watch patches bit-rot, have legitimate issues sit unaddressed, or type `/retest` for the 10th time. My primary goal in running for a position on the steering committee is optimize processes such that no one feels like their time is wasted working on this project. As of today, the process that a contributor needs to follow is byzantine, and can be daunting for folks who are new to the community. (Optimize) + +However, it doesn't stop there. In order for the community to thrive, we need to continually reevaluate our processes to ensure what we are doing makes sense, and ensure those processes are simple and concise. (Simplify) + +Lastly, I want to help promote and empower folks in the community to take on some of the roles that have traditionally been filled by google. Once core aspects of the test and release process can be done by non-googlers I think we've reached a point where we have empowered the broader community. (Empower) + +## Where I work + +Heptio diff --git a/events/elections/2017/vote_for_justinsb.md b/events/elections/2017/vote_for_justinsb.md new file mode 100644 index 00000000..bae8ece2 --- /dev/null +++ b/events/elections/2017/vote_for_justinsb.md @@ -0,0 +1,43 @@ +## Vote for justinsb! + +You can see my full "manifesto" [here](https://groups.google.com/d/msg/kubernetes-dev/YawXYxGHWEg/IlLN-iD6CgAJ); we've been asked to provide a shorter summary here. + +This election is critical. The steering committee is not an honorary position; +it has effectively unlimited powers. Serving is a duty, not a privilege, and I +humbly ask for your vote. + +--- + +I have been involved with Kubernetes since before 1.0, primarily on AWS support +(I am a lead of sig-aws), but also contributed the original multi-zone & +NodePort support. I also started the kops project, to produce an open-source +and consistent Kubernetes installation tool. I'm an independent, working with +Kubernetes in my day job but not employed to contribute to it, so I contribute +instead where I see a problem or an unmet need. + +If elected, I will serve wearing my own two hats: that of a developer on +the project contributing because it is a positive personal experience, and that +of an end-user of Kubernetes valuing a stable and straightforward product. + +I believe I bring particularly strong experience on non-GCE, non-Redhat +platforms, and on the realities of maintaining projects outside the core repo. +I think these are likely to be some of the most challenging areas for the +steering committee. + +## My manifesto: + +(Abbreviated from [here](https://groups.google.com/d/msg/kubernetes-dev/YawXYxGHWEg/IlLN-iD6CgAJ) ) + +* Reach decisions quickly & consistently; communicate them clearly. + +* Value a clear and efficient developer process. + +* Empower the release team with the goal of producing a more stable release. + +* Figure out how to delegate to SIGs without creating fiefdoms. + +* Streamline our processes. + +* Promote experimentation with alternative processes amongst our many projects, +balancing against the need for a consistent experience, allow the best approaches to "bubble-up". + diff --git a/events/elections/README.md b/events/elections/README.md new file mode 100644 index 00000000..14a1ea3f --- /dev/null +++ b/events/elections/README.md @@ -0,0 +1,96 @@ +## Kubernetes Elections + +This document will outline how to conduct a Kubernetes Steering Committee Election. See the [Steering Committee Charter](https://git.k8s.io/steering/charter.md) for more information of how the committee decides when to have an election, the method, and the maximal representation. + +## Steering Committee chooses Election Deadlines and Officers + +- Steering Committee selects the Election Officers +- Dates should be in UTC time, use a [world clock service](https://www.timeanddate.com/worldclock/fixedtime.html?msg=Election+Test&iso=20181101T00&p1=%3A&ah=10) in documentation and email announcements so that end users see the correct time and date based on where they live. + + +### SC Selects the following dates: + +- Recommend the month of October to not collide with a release or end of a quarter. +- Nomination and Voter Registration period start +- Nomination period end (At least a two week period) +- Voter Registration Deadline +- Link to voter registration process (doesn’t exist yet) +- Election period start + - It takes time to create the poll in CIVS, so don’t give a specific hour, instead say “Morning of the 10th” or something vague. +- Election period stop + - CIVS needs to be manually stopped, so an actual person needs to click for the poll to stop, so this needs to be a human friendly time. +- Results announcement date + +## Process + +1. Election officers prepare the election repository + - Make github.com/kubernetes/community/elections/$YEAR + - Make github.com/kubernetes/community/elections/$YEAR/README.md, this is the voter’s guide. + - Copy over the voter’s guide from the previous year. The voter’s guide is the single source of truth for the election that year! All announcements and notices should link to this document. + - Update with new dates, candidates, and procedures (if necessary). + - Announce to the candidates to submit PRs with their platform statement (if they desire), 300 word limit. Each platform document lives in the elections/$YEAR directory, with the voter’s guide (README.md) acting as the index. + +2. Announce voting schedule to community + +- Should mostly be links to the voter guide and the Steering Committee Charter +- On kubernetes-dev list, kubernetes-dev slack, and twitter + +3. Executing the Election in CIVS + +- Use [CIVS](http://civs.cs.cornell.edu/civs_create.html) to create the election, which CIVS calls a poll. Once you send out the ballots you cannot UNSEND the emails, ensure everything in the form is correct! +- Name of the poll - “Kubernetes Steering Committee Election for $YEAR” +- Name of supervisor - “Kubernetes Steering Committee” +- Email - elections@kubernetes.io : Googlegroups doesn’t work here. This mail should resolve to members of the steering committee AND the election officers. +- Date and Time: Write in the date and time the election will stop. This field is not programmatic, the election is stopped by hand, so you can write this in plain text. +- Description: This election is to nominate the steering committee for the Kubernetes project. Select the three(3) candidates, by order of preference. Please see the voter's guide for more information. PLEASE NOTE: "No opinion" is also a voting option if you do not feel comfortable ranking every single candidate. +- Add the candidate list to the form +- How many choices will win: This number needs to be set to the amount of open seats of a given election +- More options, check the boxes for: + - Do not release results to all voters. + - Enable detailed ballot reporting. + - Allow voters to select “no opinion” for some choices. +- Click create poll, this will send elections@kubernetes.io an email with instructions. +- It will send you a link to “Poll Control”, bookmark this generated page as this is where you will add voters and also resend ballots to people if their ballot gets lost or filtered. +- This page is where the “Start Poll” and “Stop Poll” buttons are, start the poll. +- Paste in the registered voters and click add voters. + - It will mail the ballots to the participants. + - It does duplicate detection so multiple entries are fine. +- Leave the poll open for the duration of voting. + - Remember to send a 24 hour reminder before closing the poll. + +## Roles and Responsibilities: + +### Steering Committee + +- Select election dates +- Select Election Officers +- Select criteria for Kubernetes Members of Standing +- Must refrain from endorsing or otherwise advocating for any candidate +- Is allowed to vote in the election +- Announces results of the election to the community +- Commit the results of the election to the Kubernetes Community repository + +### Election Officers + +- Must be a Kubernetes Member of Standing +- Cannot be running for office in the current election +- Cannot be a current member of the steering committee +- Generates the voter guide +- Tracks candidates +- Monitors kubernetes-dev for nominations + - Keeps track of nominees in a spreadsheet + - Ensures that each nominee has the required nominations from three different employers (as stated in the charter) + - All nominations are conducted in the public, so sharing this sheet during the nomination process is encouraged +- Accepts/Reviews pull requests for the candidate platforms + - The community generally assists in helping with PRs to give the candidates a quick response time +- Update the community regularly via the community meeting +- Post on behalf of the steering committee if necessary +- Posting deadlines and reminders to kubernetes-dev, twitter, and slack. +- Reissues ballots from CIVS to voters who might have not received their ballot. +- Miscellaneous election related tasks as decided by the steering committee. +- Must refrain from endorsing or otherwise advocating for any candidate. +- Must refrain from discussing the election specifics during the election period. +- Guard the privacy of the email addresses of voters +- It is impossible for the election officers to see the results of the election until the election ends; for purposes of transparency with the community it is encouraged to release some statistics during the election (ie. “65% of the community has voted so far!”) +- Ensure that the election results are handed over to the steering committee. +- Is allowed to vote in the election
\ No newline at end of file diff --git a/events/office-hours.md b/events/office-hours.md new file mode 100644 index 00000000..fa6249fe --- /dev/null +++ b/events/office-hours.md @@ -0,0 +1,61 @@ +# Kubernetes Community Office Hours + +Office Hours is a live stream where we answer live questions about Kubernetes from **users** on the [YouTube channel](https://www.youtube.com/c/KubernetesCommunity/). Office hours are a regularly scheduled meeting where people can bring topics to discuss with the greater community. They are great for answering questions, getting feedback on how you’re using Kubernetes, or to just passively learn by following along. + +### When and Where + +Third Wednesday of every month, there are two sessions: + +- European Edition: [2pm UTC](https://www.timeanddate.com/worldclock/fixedtime.html?msg=Kubernetes+Office+Hours+%28European+Edition%29&iso=20171115T14&p1=136&ah=1) +- Western Edition: [9pm UTC](https://www.timeanddate.com/worldclock/fixedtime.html?msg=Kubernetes+Office+Hours+%28Western+Edition%29&iso=20171115T13&p1=1241) + +Tune into the [Kubernetes YouTube Channel](https://www.youtube.com/c/KubernetesCommunity/live) to follow along. + +You can post questions on the [#office-hours channel](https://kubernetes.slack.com/messages/office-hours) on Slack, or if you like you can submit your question to Stack Overflow and have us take a look. + +### How it Works + +#### Bringing your own Stack Overflow Question + +If you submit a [SO question](https://stackoverflow.com/questions/tagged/kubernetes) we can prepare ahead of time and check important details. + +As a thanks to the community the person asking the question can then ensure that a well written answer makes it way to the question afterwards so that we can build a knowledge base and help maintain the incoming questions. We can also use the video archive of each meeting to bring context to each SO question we answer. + +Questions that aren’t addressed or need work can be punted to the next week or we can encourage other people to give them a look, at a bare minimum we can at least help socialize the difficult questions. We keep a backlog of open questions in the [meeting notes](http://bit.ly/k8s-office-hours-notes). + +The hosts will do a shout out of thanks to the [current leaderboard](https://stackoverflow.com/tags/kubernetes/topusers) of SO answerers each meeting so the people helping answer questions can get some recognition. + +#### What’s Ontopic + +Specific questions about Kubernetes as pertaining to the topic. Since this is a Q&A format, we’d like questions that can be answered generally. So for example: + +- Bad: My ingress configuration is broken please fix it. (Dependent on local configuration) +- Good: My ingress configuration is broken can you discuss what problems you usually see and common mistakes that people make so I can avoid them? (Generalized and can communicate things that can help multiple people) +- Bad: How do I deploy my application in Kubernetes? (Easy to look up, tons of content on this) +- Good: What best practice techniques are there for deploying applications in Kubernetes? (Nuanced version lets us cover RBAC, application lifecycle) +- Bad: My pods aren’t coming up on a node. (Vague) +- Good: My pods aren’t coming up or misbehaving, what are the debugging steps to find out the root cause? (Helps us communicate the thought process for debugging to a general audience) + +#### What’s Offtopic + +Local installation and debugging: The participants don’t have access to your network or your hardware. The host can/should help the user transform a vague question into something answerable and reusable. + +Let’s try to not just dismiss bad questions outright, but use it as an opportunity for the answer to be a teaching tool as opposed to just answering “It’s in /var/log/foo, next question.” If the question is about logging then the developers might as well share their experiences in that area, recommend tools, share an anecdote, things of that nature. + + +### Archives + +Archives of all the sessions are kept here: + +- [Office Hours playlist](https://www.youtube.com/watch?v=D0Q7wwljN30&list=PL69nYSiGNLP3azFUvYJjGn45YbF6C-uIg) +- [Office Hours meeting notes](http://bit.ly/k8s-office-hours-notes) + + +### Volunteering + +We're always looking for volunteers to host and answer questions. Volunteers looking to host hours in +more time zones are also welcome to join in and help us expand. + +- [Volunteer working sheet](http://bit.ly/k8s-office-hours-volunteers) + +If you have any questions ping [@castrojo](https://github.com/castrojo). diff --git a/generator/Dockerfile b/generator/Dockerfile deleted file mode 100644 index deffa8a9..00000000 --- a/generator/Dockerfile +++ /dev/null @@ -1,7 +0,0 @@ -FROM golang:1.8 - -WORKDIR /go/src/app -COPY . . - -RUN go-wrapper download -RUN go-wrapper install diff --git a/generator/Gopkg.lock b/generator/Gopkg.lock new file mode 100644 index 00000000..eecd1bec --- /dev/null +++ b/generator/Gopkg.lock @@ -0,0 +1,15 @@ +# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'. + + +[[projects]] + branch = "v2" + name = "gopkg.in/yaml.v2" + packages = ["."] + revision = "eb3733d160e74a9c7e442f435eb3bea458e1d19f" + +[solve-meta] + analyzer-name = "dep" + analyzer-version = 1 + inputs-digest = "c39e9119cc91080f9178c39214d6ca06156205351dec2523319554ee3669537e" + solver-name = "gps-cdcl" + solver-version = 1 diff --git a/generator/Gopkg.toml b/generator/Gopkg.toml new file mode 100644 index 00000000..aabec216 --- /dev/null +++ b/generator/Gopkg.toml @@ -0,0 +1,26 @@ + +# Gopkg.toml example +# +# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md +# for detailed Gopkg.toml documentation. +# +# required = ["github.com/user/thing/cmd/thing"] +# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"] +# +# [[constraint]] +# name = "github.com/user/project" +# version = "1.0.0" +# +# [[constraint]] +# name = "github.com/user/project2" +# branch = "dev" +# source = "github.com/myfork/project2" +# +# [[override]] +# name = "github.com/x/y" +# version = "2.4.0" + + +[[constraint]] + branch = "v2" + name = "gopkg.in/yaml.v2" diff --git a/generator/README.md b/generator/README.md index d2382aa5..0f09eb0f 100644 --- a/generator/README.md +++ b/generator/README.md @@ -5,40 +5,55 @@ different Special Interest Groups (SIGs) of Kubernetes. The authoritative source for SIG information is the `sigs.yaml` file in the project root. All updates must be done there. +The schema for this file should be self explanatory. However, if you need to see all the options check out the generator code in `app.go`. + +**Time Zone gotcha**: +Time zones make everything complicated. +And Daylight Savings time makes it even more complicated. +Meetings are specified with a time zone and we generate a link to http://www.thetimezoneconverter.com/ so that people can easily convert it to their local time zone. +To make this work you need to specify the time zone in a way that that web site recognizes. +Practically, that means US pacific time must be `PT (Pacific Time)`. +`PT` isn't good enough, unfortunately. + When an update happens to the this file, the next step is generate the accompanying documentation. This takes the format of two types of doc file: ``` -./sig-<sig-name>/README.md -./wg-<working-group-name>/README.md -./sig-list.md +sig-<sig-name>/README.md +wg-<working-group-name>/README.md +sig-list.md ``` For example, if a contributor has updated `sig-cluster-lifecycle`, the following files will be generated: ``` -./sig-cluster-lifecycle/README.md -./sig-list.md +sig-cluster-lifecycle/README.md +sig-list.md ``` ## How to use -To (re)build documentation for all the SIGs, run these commands: +To (re)build documentation for all the SIGs in a go environment, run: ```bash -make all +make generate +``` +or to run this inside a docker container: +```bash +make generate-dockerized ``` -To build docs for one SIG, run these commands: +To build docs for one SIG, run one of these commands: ```bash -make SIG=sig-apps gen-docs -make SIG=sig-testing gen-docs -make WG=resource-management gen-docs +make WHAT=sig-apps +make WHAT=cluster-lifecycle +make WHAT=wg-resource-management +make WHAT=container-identity ``` -where the `SIG` or `WG` var refers to the directory being built. +where the `WHAT` var refers to the directory being built. ## Adding custom content to your README diff --git a/generator/app.go b/generator/app.go index 99336c0d..da75b16c 100644 --- a/generator/app.go +++ b/generator/app.go @@ -20,6 +20,7 @@ import ( "fmt" "io/ioutil" "log" + "net/url" "os" "path/filepath" "sort" @@ -29,7 +30,7 @@ import ( "gopkg.in/yaml.v2" ) -var ( +const ( readmeTemplate = "readme.tmpl" listTemplate = "list.tmpl" headerTemplate = "header.tmpl" @@ -37,11 +38,14 @@ var ( sigsYamlFile = "sigs.yaml" sigListOutput = "sig-list.md" indexFilename = "README.md" - baseOutputDir = "generated" - githubTeamNames = []string{"misc", "test-failures", "bugs", "feature-requests", "proposals", "pr-reviews", "api-reviews"} - beginMarker = "<!-- BEGIN CUSTOM CONTENT -->" - endMarker = "<!-- END CUSTOM CONTENT -->" + beginMarker = "<!-- BEGIN CUSTOM CONTENT -->" + endMarker = "<!-- END CUSTOM CONTENT -->" +) + +var ( + baseGeneratorDir = "" + templateDir = "generator" ) // Lead represents a lead engineer for a particular group. There are usually @@ -55,18 +59,22 @@ type Lead struct { // Meeting represents a regular meeting for a group. type Meeting struct { Day string - UTC string - PST string + Time string + TZ string `yaml:"tz"` Frequency string } // Contact represents the various contact points for a group. type Contact struct { - Slack string - MailingList string `yaml:"mailing_list"` - FullGitHubTeams bool `yaml:"full_github_teams"` - GithubTeamPrefix string `yaml:"github_team_prefix"` - GithubTeamNames []string + Slack string + MailingList string `yaml:"mailing_list"` + GithubTeams []GithubTeams `yaml:"teams"` +} + +// GithubTeams represents a specific Github Team. +type GithubTeams struct { + Name string + Description string } // Group represents either a Special Interest Group (SIG) or a Working Group (WG) @@ -74,6 +82,7 @@ type Group struct { Name string Dir string MissionStatement string `yaml:"mission_statement"` + Label string Leads []Lead Meetings []Meeting MeetingURL string `yaml:"meeting_url"` @@ -89,20 +98,6 @@ func (e *Group) DirName(prefix string) string { return fmt.Sprintf("%s-%s", prefix, strings.ToLower(strings.Replace(e.Name, " ", "-", -1))) } -// SetupGitHubTeams will iterate over all the possible teams available to a -// group (these are defined by the Kubernetes organisation) and populate a -// list using the group's prefix. -func (e *Group) SetupGitHubTeams(prefix string) { - ghPrefix := e.Contact.GithubTeamPrefix - if ghPrefix == "" { - ghPrefix = e.DirName(prefix) - } - - for _, gtn := range githubTeamNames { - e.Contact.GithubTeamNames = append(e.Contact.GithubTeamNames, fmt.Sprintf("%s-%s", ghPrefix, gtn)) - } -} - // Context is the context for the sigs.yaml file. type Context struct { Sigs []Group @@ -147,9 +142,22 @@ func getExistingContent(path string) (string, error) { return strings.Join(captured, "\n"), nil } +var funcMap template.FuncMap = template.FuncMap{ + "tzUrlEncode": tzUrlEncode, +} + +// tzUrlEncode returns an url encoded string without the + shortcut. This is +// required as the timezone conversion site we are using doesn't recognize + as +// a valid url escape character. +func tzUrlEncode(tz string) string { + return strings.Replace(url.QueryEscape(tz), "+", "%20", -1) +} + func writeTemplate(templatePath, outputPath string, data interface{}) error { // set up template - t, err := template.ParseFiles(templatePath, headerTemplate) + t, err := template.New(filepath.Base(templatePath)). + Funcs(funcMap). + ParseFiles(templatePath, filepath.Join(baseGeneratorDir, templateDir, headerTemplate)) if err != nil { return err } @@ -200,29 +208,27 @@ func writeCustomContentBlock(f *os.File, content string) { func createGroupReadme(groups []Group, prefix string) error { // figure out if the user wants to generate one group var selectedGroupName *string - if envVal, ok := os.LookupEnv(strings.ToUpper(prefix)); ok { + if envVal, ok := os.LookupEnv("WHAT"); ok { selectedGroupName = &envVal } for _, group := range groups { group.Dir = group.DirName(prefix) // skip generation if the user specified only one group - if selectedGroupName != nil && *selectedGroupName != group.Dir { + if selectedGroupName != nil && strings.HasSuffix(group.Dir, *selectedGroupName) == false { fmt.Printf("Skipping %s/README.md\n", group.Dir) continue } fmt.Printf("Generating %s/README.md\n", group.Dir) - outputDir := filepath.Join(baseOutputDir, group.Dir) + outputDir := filepath.Join(baseGeneratorDir, group.Dir) if err := createDirIfNotExists(outputDir); err != nil { return err } - group.SetupGitHubTeams(prefix) - outputPath := filepath.Join(outputDir, indexFilename) - readmePath := fmt.Sprintf("%s_%s", prefix, readmeTemplate) + readmePath := filepath.Join(baseGeneratorDir, templateDir, fmt.Sprintf("%s_%s", prefix, readmeTemplate)) if err := writeTemplate(readmePath, outputPath, group); err != nil { return err } @@ -232,7 +238,7 @@ func createGroupReadme(groups []Group, prefix string) error { } func main() { - yamlData, err := ioutil.ReadFile(filepath.Join(baseOutputDir, sigsYamlFile)) + yamlData, err := ioutil.ReadFile(filepath.Join(baseGeneratorDir, sigsYamlFile)) if err != nil { log.Fatal(err) } @@ -262,8 +268,8 @@ func main() { } fmt.Println("Generating sig-list.md") - outputPath := filepath.Join(baseOutputDir, sigListOutput) - err = writeTemplate(listTemplate, outputPath, ctx) + outputPath := filepath.Join(baseGeneratorDir, sigListOutput) + err = writeTemplate(filepath.Join(baseGeneratorDir, templateDir, listTemplate), outputPath, ctx) if err != nil { log.Fatal(err) } diff --git a/generator/app_test.go b/generator/app_test.go index c71471c7..965e0d47 100644 --- a/generator/app_test.go +++ b/generator/app_test.go @@ -17,11 +17,9 @@ limitations under the License. package main import ( - "fmt" "io/ioutil" "os" "path/filepath" - "reflect" "strings" "testing" ) @@ -83,6 +81,9 @@ func TestGetExistingData(t *testing.T) { } func TestWriteTemplate(t *testing.T) { + baseGeneratorDir = "generated" + templateDir = "../../generator" + customContent := ` <!-- BEGIN CUSTOM CONTENT --> Example @@ -147,38 +148,13 @@ func TestGroupDirName(t *testing.T) { } } -func TestSetupGithubTeams(t *testing.T) { - group := Group{Name: "Foo Bar"} - group.SetupGitHubTeams("sig") - - var expected []string - for _, ght := range githubTeamNames { - expected = append(expected, fmt.Sprintf("sig-foo-bar-%s", ght)) - } - - if !reflect.DeepEqual(group.Contact.GithubTeamNames, expected) { - t.Fatalf("%v does not match %v", group.Contact.GithubTeamNames, expected) - } -} - -func TestCustomPrefixSetupGithubTeams(t *testing.T) { - group := Group{Contact: Contact{GithubTeamPrefix: "foo"}} - group.SetupGitHubTeams("") - - var expected []string - for _, ght := range githubTeamNames { - expected = append(expected, fmt.Sprintf("foo-%s", ght)) - } - - if !reflect.DeepEqual(group.Contact.GithubTeamNames, expected) { - t.Fatalf("%v does not match %v", group.Contact.GithubTeamNames, expected) - } -} - func TestCreateGroupReadmes(t *testing.T) { + baseGeneratorDir = "generated" + templateDir = "../../generator" + groups := []Group{ - Group{Name: "Foo"}, - Group{Name: "Bar"}, + {Name: "Foo"}, + {Name: "Bar"}, } err := createGroupReadme(groups, "sig") @@ -187,7 +163,7 @@ func TestCreateGroupReadmes(t *testing.T) { } for _, group := range groups { - path := filepath.Join(baseOutputDir, group.DirName("sig"), "README.md") + path := filepath.Join(baseGeneratorDir, group.DirName("sig"), "README.md") if !pathExists(path) { t.Fatalf("%s should exist", path) } @@ -195,11 +171,14 @@ func TestCreateGroupReadmes(t *testing.T) { } func TestReadmesAreSkipped(t *testing.T) { + baseGeneratorDir = "generated" + templateDir = "../../generator" + os.Setenv("SIG", "sig-foo") groups := []Group{ - Group{Name: "Foo"}, - Group{Name: "Bar"}, + {Name: "Foo"}, + {Name: "Bar"}, } err := createGroupReadme(groups, "sig") @@ -208,7 +187,7 @@ func TestReadmesAreSkipped(t *testing.T) { } for _, group := range groups[1:] { - path := filepath.Join(baseOutputDir, group.DirName("sig"), "README.md") + path := filepath.Join(baseGeneratorDir, group.DirName("sig"), "README.md") if !pathExists(path) { t.Fatalf("%s should exist", path) } @@ -232,6 +211,9 @@ func copyFile(src, dst string) error { } func TestFullGeneration(t *testing.T) { + baseGeneratorDir = "generated" + templateDir = "../../generator" + err := copyFile("testdata/sigs.yaml", "generated/sigs.yaml") if err != nil { t.Fatalf("Error received: %v", err) @@ -241,7 +223,7 @@ func TestFullGeneration(t *testing.T) { expectedDirs := []string{"sig-foo", "sig-bar", "wg-baz"} for _, ed := range expectedDirs { - path := filepath.Join(baseOutputDir, ed, "README.md") + path := filepath.Join(baseGeneratorDir, ed, "README.md") if !pathExists(path) { t.Fatalf("%s should exist", path) } diff --git a/generator/header.tmpl b/generator/header.tmpl index 9f53887a..c84f4b7b 100644 --- a/generator/header.tmpl +++ b/generator/header.tmpl @@ -5,6 +5,6 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> {{- end -}} diff --git a/generator/list.tmpl b/generator/list.tmpl index f911529e..38ddc0e1 100644 --- a/generator/list.tmpl +++ b/generator/list.tmpl @@ -2,7 +2,7 @@ # SIGs and Working Groups Most community activity is organized into Special Interest Groups (SIGs), -time bounded Working Groups, and the [community meeting](communication.md#Meeting). +time bounded Working Groups, and the [community meeting](communication.md#weekly-meeting). SIGs follow these [guidelines](governance.md) although each of these groups may operate a little differently depending on their needs and workflow. @@ -13,10 +13,10 @@ When the need arises, a [new SIG can be created](sig-creation-procedure.md) ### Master SIG List -| Name | Leads | Contact | Meetings | -|------|-------|---------|----------| +| Name | Label | Leads | Contact | Meetings | +|------|--------|-------|---------|----------| {{- range .Sigs}} -|[{{.Name}}]({{.Dir}}/README.md)|{{range .Leads}}* [{{.Name}}](https://github.com/{{.GitHub}}){{if .Company}}, {{.Company}}{{end}}<br>{{end}}|* [Slack](https://kubernetes.slack.com/messages/{{.Contact.Slack}})<br>* [Mailing List]({{.Contact.MailingList}})|{{ $save := . }}{{range .Meetings}}* [{{.Day}}s at {{.UTC}} UTC ({{.Frequency}})]({{$save.MeetingURL}})<br>{{end}} +|[{{.Name}}]({{.Dir}}/README.md)|{{.Label}}|{{range .Leads}}* [{{.Name}}](https://github.com/{{.GitHub}}){{if .Company}}, {{.Company}}{{end}}<br>{{end}}|* [Slack](https://kubernetes.slack.com/messages/{{.Contact.Slack}})<br>* [Mailing List]({{.Contact.MailingList}})|{{ $save := . }}{{range .Meetings}}* [{{.Day}}s at {{.Time}} {{.TZ}} ({{.Frequency}})]({{$save.MeetingURL}})<br>{{end}} {{- end }} ### Master Working Group List @@ -24,5 +24,5 @@ When the need arises, a [new SIG can be created](sig-creation-procedure.md) | Name | Organizers | Contact | Meetings | |------|------------|---------|----------| {{- range .WorkingGroups}} -|[{{.Name}}]({{.Dir}}/README.md)|{{range .Leads}}* [{{.Name}}](https://github.com/{{.GitHub}}){{if .Company}}, {{.Company}}{{end}}<br>{{end}}|* [Slack](https://kubernetes.slack.com/messages/{{.Contact.Slack}})<br>* [Mailing List]({{.Contact.MailingList}})|{{ $save := . }}{{range .Meetings}}* [{{.Day}}s at {{.UTC}} UTC ({{.Frequency}})]({{$save.MeetingURL}})<br>{{end}} +|[{{.Name}}]({{.Dir}}/README.md)|{{range .Leads}}* [{{.Name}}](https://github.com/{{.GitHub}}){{if .Company}}, {{.Company}}{{end}}<br>{{end}}|* [Slack](https://kubernetes.slack.com/messages/{{.Contact.Slack}})<br>* [Mailing List]({{.Contact.MailingList}})|{{ $save := . }}{{range .Meetings}}* [{{.Day}}s at {{.Time}} {{.TZ}} ({{.Frequency}})]({{$save.MeetingURL}})<br>{{end}} {{- end }} diff --git a/generator/sig_readme.tmpl b/generator/sig_readme.tmpl index 8e5605bf..2af46a52 100644 --- a/generator/sig_readme.tmpl +++ b/generator/sig_readme.tmpl @@ -4,23 +4,36 @@ {{ .MissionStatement }} ## Meetings {{- range .Meetings }} -* [{{.Day}}s at {{.UTC}} UTC]({{$.MeetingURL}}) ({{.Frequency}}). [Convert to your timezone](http://www.thetimezoneconverter.com/?t={{.UTC}}&tz=UTC). +* [{{.Day}}s at {{.Time}} {{.TZ}}]({{$.MeetingURL}}) ({{.Frequency}}). [Convert to your timezone](http://www.thetimezoneconverter.com/?t={{.Time}}&tz={{.TZ | tzUrlEncode}}). {{- end }} +{{ if .MeetingArchiveURL -}} Meeting notes and Agenda can be found [here]({{.MeetingArchiveURL}}). +{{- end }} +{{ if .MeetingRecordingsURL -}} Meeting recordings can be found [here]({{.MeetingRecordingsURL}}). +{{- end }} ## Leads {{- range .Leads }} -* [{{.Name}}](https://github.com/{{.GitHub}}){{if .Company}}, {{.Company}}{{end}} +* {{.Name}} (**[@{{.GitHub}}](https://github.com/{{.GitHub}})**){{if .Company}}, {{.Company}}{{end}} {{- end }} ## Contact * [Slack](https://kubernetes.slack.com/messages/{{.Contact.Slack}}) * [Mailing list]({{.Contact.MailingList}}) -{{if .Contact.FullGitHubTeams}} -## GitHub Teams -{{range .Contact.GithubTeamNames -}} -* [@{{.}}](https://github.com/kubernetes/teams/{{.}}) -{{end}} -{{end}} +{{- if .Label }} +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2F{{.Label}}) +{{- end }} +{{ if .Contact.GithubTeams }} +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +{{- range .Contact.GithubTeams }} +| @kubernetes/{{.Name}} | [link](https://github.com/orgs/kubernetes/teams/{{.Name}}) | {{.Description}} | +{{- end }} +{{ end }} diff --git a/generator/vendor/gopkg.in/yaml.v2/.travis.yml b/generator/vendor/gopkg.in/yaml.v2/.travis.yml new file mode 100644 index 00000000..004172a2 --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/.travis.yml @@ -0,0 +1,9 @@ +language: go + +go: + - 1.4 + - 1.5 + - 1.6 + - tip + +go_import_path: gopkg.in/yaml.v2 diff --git a/generator/vendor/gopkg.in/yaml.v2/LICENSE b/generator/vendor/gopkg.in/yaml.v2/LICENSE new file mode 100644 index 00000000..8dada3ed --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/generator/vendor/gopkg.in/yaml.v2/LICENSE.libyaml b/generator/vendor/gopkg.in/yaml.v2/LICENSE.libyaml new file mode 100644 index 00000000..8da58fbf --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/LICENSE.libyaml @@ -0,0 +1,31 @@ +The following files were ported to Go from C files of libyaml, and thus +are still covered by their original copyright and license: + + apic.go + emitterc.go + parserc.go + readerc.go + scannerc.go + writerc.go + yamlh.go + yamlprivateh.go + +Copyright (c) 2006 Kirill Simonov + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/generator/vendor/gopkg.in/yaml.v2/README.md b/generator/vendor/gopkg.in/yaml.v2/README.md new file mode 100644 index 00000000..7a512d67 --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/README.md @@ -0,0 +1,133 @@ +# YAML support for the Go language + +Introduction +------------ + +The yaml package enables Go programs to comfortably encode and decode YAML +values. It was developed within [Canonical](https://www.canonical.com) as +part of the [juju](https://juju.ubuntu.com) project, and is based on a +pure Go port of the well-known [libyaml](http://pyyaml.org/wiki/LibYAML) +C library to parse and generate YAML data quickly and reliably. + +Compatibility +------------- + +The yaml package supports most of YAML 1.1 and 1.2, including support for +anchors, tags, map merging, etc. Multi-document unmarshalling is not yet +implemented, and base-60 floats from YAML 1.1 are purposefully not +supported since they're a poor design and are gone in YAML 1.2. + +Installation and usage +---------------------- + +The import path for the package is *gopkg.in/yaml.v2*. + +To install it, run: + + go get gopkg.in/yaml.v2 + +API documentation +----------------- + +If opened in a browser, the import path itself leads to the API documentation: + + * [https://gopkg.in/yaml.v2](https://gopkg.in/yaml.v2) + +API stability +------------- + +The package API for yaml v2 will remain stable as described in [gopkg.in](https://gopkg.in). + + +License +------- + +The yaml package is licensed under the Apache License 2.0. Please see the LICENSE file for details. + + +Example +------- + +Some more examples can be found in the "examples" folder. + +```Go +package main + +import ( + "fmt" + "log" + + "gopkg.in/yaml.v2" +) + +var data = ` +a: Easy! +b: + c: 2 + d: [3, 4] +` + +type T struct { + A string + B struct { + RenamedC int `yaml:"c"` + D []int `yaml:",flow"` + } +} + +func main() { + t := T{} + + err := yaml.Unmarshal([]byte(data), &t) + if err != nil { + log.Fatalf("error: %v", err) + } + fmt.Printf("--- t:\n%v\n\n", t) + + d, err := yaml.Marshal(&t) + if err != nil { + log.Fatalf("error: %v", err) + } + fmt.Printf("--- t dump:\n%s\n\n", string(d)) + + m := make(map[interface{}]interface{}) + + err = yaml.Unmarshal([]byte(data), &m) + if err != nil { + log.Fatalf("error: %v", err) + } + fmt.Printf("--- m:\n%v\n\n", m) + + d, err = yaml.Marshal(&m) + if err != nil { + log.Fatalf("error: %v", err) + } + fmt.Printf("--- m dump:\n%s\n\n", string(d)) +} +``` + +This example will generate the following output: + +``` +--- t: +{Easy! {2 [3 4]}} + +--- t dump: +a: Easy! +b: + c: 2 + d: [3, 4] + + +--- m: +map[a:Easy! b:map[c:2 d:[3 4]]] + +--- m dump: +a: Easy! +b: + c: 2 + d: + - 3 + - 4 +``` + diff --git a/generator/vendor/gopkg.in/yaml.v2/apic.go b/generator/vendor/gopkg.in/yaml.v2/apic.go new file mode 100644 index 00000000..95ec014e --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/apic.go @@ -0,0 +1,742 @@ +package yaml + +import ( + "io" + "os" +) + +func yaml_insert_token(parser *yaml_parser_t, pos int, token *yaml_token_t) { + //fmt.Println("yaml_insert_token", "pos:", pos, "typ:", token.typ, "head:", parser.tokens_head, "len:", len(parser.tokens)) + + // Check if we can move the queue at the beginning of the buffer. + if parser.tokens_head > 0 && len(parser.tokens) == cap(parser.tokens) { + if parser.tokens_head != len(parser.tokens) { + copy(parser.tokens, parser.tokens[parser.tokens_head:]) + } + parser.tokens = parser.tokens[:len(parser.tokens)-parser.tokens_head] + parser.tokens_head = 0 + } + parser.tokens = append(parser.tokens, *token) + if pos < 0 { + return + } + copy(parser.tokens[parser.tokens_head+pos+1:], parser.tokens[parser.tokens_head+pos:]) + parser.tokens[parser.tokens_head+pos] = *token +} + +// Create a new parser object. +func yaml_parser_initialize(parser *yaml_parser_t) bool { + *parser = yaml_parser_t{ + raw_buffer: make([]byte, 0, input_raw_buffer_size), + buffer: make([]byte, 0, input_buffer_size), + } + return true +} + +// Destroy a parser object. +func yaml_parser_delete(parser *yaml_parser_t) { + *parser = yaml_parser_t{} +} + +// String read handler. +func yaml_string_read_handler(parser *yaml_parser_t, buffer []byte) (n int, err error) { + if parser.input_pos == len(parser.input) { + return 0, io.EOF + } + n = copy(buffer, parser.input[parser.input_pos:]) + parser.input_pos += n + return n, nil +} + +// File read handler. +func yaml_file_read_handler(parser *yaml_parser_t, buffer []byte) (n int, err error) { + return parser.input_file.Read(buffer) +} + +// Set a string input. +func yaml_parser_set_input_string(parser *yaml_parser_t, input []byte) { + if parser.read_handler != nil { + panic("must set the input source only once") + } + parser.read_handler = yaml_string_read_handler + parser.input = input + parser.input_pos = 0 +} + +// Set a file input. +func yaml_parser_set_input_file(parser *yaml_parser_t, file *os.File) { + if parser.read_handler != nil { + panic("must set the input source only once") + } + parser.read_handler = yaml_file_read_handler + parser.input_file = file +} + +// Set the source encoding. +func yaml_parser_set_encoding(parser *yaml_parser_t, encoding yaml_encoding_t) { + if parser.encoding != yaml_ANY_ENCODING { + panic("must set the encoding only once") + } + parser.encoding = encoding +} + +// Create a new emitter object. +func yaml_emitter_initialize(emitter *yaml_emitter_t) bool { + *emitter = yaml_emitter_t{ + buffer: make([]byte, output_buffer_size), + raw_buffer: make([]byte, 0, output_raw_buffer_size), + states: make([]yaml_emitter_state_t, 0, initial_stack_size), + events: make([]yaml_event_t, 0, initial_queue_size), + } + return true +} + +// Destroy an emitter object. +func yaml_emitter_delete(emitter *yaml_emitter_t) { + *emitter = yaml_emitter_t{} +} + +// String write handler. +func yaml_string_write_handler(emitter *yaml_emitter_t, buffer []byte) error { + *emitter.output_buffer = append(*emitter.output_buffer, buffer...) + return nil +} + +// File write handler. +func yaml_file_write_handler(emitter *yaml_emitter_t, buffer []byte) error { + _, err := emitter.output_file.Write(buffer) + return err +} + +// Set a string output. +func yaml_emitter_set_output_string(emitter *yaml_emitter_t, output_buffer *[]byte) { + if emitter.write_handler != nil { + panic("must set the output target only once") + } + emitter.write_handler = yaml_string_write_handler + emitter.output_buffer = output_buffer +} + +// Set a file output. +func yaml_emitter_set_output_file(emitter *yaml_emitter_t, file io.Writer) { + if emitter.write_handler != nil { + panic("must set the output target only once") + } + emitter.write_handler = yaml_file_write_handler + emitter.output_file = file +} + +// Set the output encoding. +func yaml_emitter_set_encoding(emitter *yaml_emitter_t, encoding yaml_encoding_t) { + if emitter.encoding != yaml_ANY_ENCODING { + panic("must set the output encoding only once") + } + emitter.encoding = encoding +} + +// Set the canonical output style. +func yaml_emitter_set_canonical(emitter *yaml_emitter_t, canonical bool) { + emitter.canonical = canonical +} + +//// Set the indentation increment. +func yaml_emitter_set_indent(emitter *yaml_emitter_t, indent int) { + if indent < 2 || indent > 9 { + indent = 2 + } + emitter.best_indent = indent +} + +// Set the preferred line width. +func yaml_emitter_set_width(emitter *yaml_emitter_t, width int) { + if width < 0 { + width = -1 + } + emitter.best_width = width +} + +// Set if unescaped non-ASCII characters are allowed. +func yaml_emitter_set_unicode(emitter *yaml_emitter_t, unicode bool) { + emitter.unicode = unicode +} + +// Set the preferred line break character. +func yaml_emitter_set_break(emitter *yaml_emitter_t, line_break yaml_break_t) { + emitter.line_break = line_break +} + +///* +// * Destroy a token object. +// */ +// +//YAML_DECLARE(void) +//yaml_token_delete(yaml_token_t *token) +//{ +// assert(token); // Non-NULL token object expected. +// +// switch (token.type) +// { +// case YAML_TAG_DIRECTIVE_TOKEN: +// yaml_free(token.data.tag_directive.handle); +// yaml_free(token.data.tag_directive.prefix); +// break; +// +// case YAML_ALIAS_TOKEN: +// yaml_free(token.data.alias.value); +// break; +// +// case YAML_ANCHOR_TOKEN: +// yaml_free(token.data.anchor.value); +// break; +// +// case YAML_TAG_TOKEN: +// yaml_free(token.data.tag.handle); +// yaml_free(token.data.tag.suffix); +// break; +// +// case YAML_SCALAR_TOKEN: +// yaml_free(token.data.scalar.value); +// break; +// +// default: +// break; +// } +// +// memset(token, 0, sizeof(yaml_token_t)); +//} +// +///* +// * Check if a string is a valid UTF-8 sequence. +// * +// * Check 'reader.c' for more details on UTF-8 encoding. +// */ +// +//static int +//yaml_check_utf8(yaml_char_t *start, size_t length) +//{ +// yaml_char_t *end = start+length; +// yaml_char_t *pointer = start; +// +// while (pointer < end) { +// unsigned char octet; +// unsigned int width; +// unsigned int value; +// size_t k; +// +// octet = pointer[0]; +// width = (octet & 0x80) == 0x00 ? 1 : +// (octet & 0xE0) == 0xC0 ? 2 : +// (octet & 0xF0) == 0xE0 ? 3 : +// (octet & 0xF8) == 0xF0 ? 4 : 0; +// value = (octet & 0x80) == 0x00 ? octet & 0x7F : +// (octet & 0xE0) == 0xC0 ? octet & 0x1F : +// (octet & 0xF0) == 0xE0 ? octet & 0x0F : +// (octet & 0xF8) == 0xF0 ? octet & 0x07 : 0; +// if (!width) return 0; +// if (pointer+width > end) return 0; +// for (k = 1; k < width; k ++) { +// octet = pointer[k]; +// if ((octet & 0xC0) != 0x80) return 0; +// value = (value << 6) + (octet & 0x3F); +// } +// if (!((width == 1) || +// (width == 2 && value >= 0x80) || +// (width == 3 && value >= 0x800) || +// (width == 4 && value >= 0x10000))) return 0; +// +// pointer += width; +// } +// +// return 1; +//} +// + +// Create STREAM-START. +func yaml_stream_start_event_initialize(event *yaml_event_t, encoding yaml_encoding_t) bool { + *event = yaml_event_t{ + typ: yaml_STREAM_START_EVENT, + encoding: encoding, + } + return true +} + +// Create STREAM-END. +func yaml_stream_end_event_initialize(event *yaml_event_t) bool { + *event = yaml_event_t{ + typ: yaml_STREAM_END_EVENT, + } + return true +} + +// Create DOCUMENT-START. +func yaml_document_start_event_initialize(event *yaml_event_t, version_directive *yaml_version_directive_t, + tag_directives []yaml_tag_directive_t, implicit bool) bool { + *event = yaml_event_t{ + typ: yaml_DOCUMENT_START_EVENT, + version_directive: version_directive, + tag_directives: tag_directives, + implicit: implicit, + } + return true +} + +// Create DOCUMENT-END. +func yaml_document_end_event_initialize(event *yaml_event_t, implicit bool) bool { + *event = yaml_event_t{ + typ: yaml_DOCUMENT_END_EVENT, + implicit: implicit, + } + return true +} + +///* +// * Create ALIAS. +// */ +// +//YAML_DECLARE(int) +//yaml_alias_event_initialize(event *yaml_event_t, anchor *yaml_char_t) +//{ +// mark yaml_mark_t = { 0, 0, 0 } +// anchor_copy *yaml_char_t = NULL +// +// assert(event) // Non-NULL event object is expected. +// assert(anchor) // Non-NULL anchor is expected. +// +// if (!yaml_check_utf8(anchor, strlen((char *)anchor))) return 0 +// +// anchor_copy = yaml_strdup(anchor) +// if (!anchor_copy) +// return 0 +// +// ALIAS_EVENT_INIT(*event, anchor_copy, mark, mark) +// +// return 1 +//} + +// Create SCALAR. +func yaml_scalar_event_initialize(event *yaml_event_t, anchor, tag, value []byte, plain_implicit, quoted_implicit bool, style yaml_scalar_style_t) bool { + *event = yaml_event_t{ + typ: yaml_SCALAR_EVENT, + anchor: anchor, + tag: tag, + value: value, + implicit: plain_implicit, + quoted_implicit: quoted_implicit, + style: yaml_style_t(style), + } + return true +} + +// Create SEQUENCE-START. +func yaml_sequence_start_event_initialize(event *yaml_event_t, anchor, tag []byte, implicit bool, style yaml_sequence_style_t) bool { + *event = yaml_event_t{ + typ: yaml_SEQUENCE_START_EVENT, + anchor: anchor, + tag: tag, + implicit: implicit, + style: yaml_style_t(style), + } + return true +} + +// Create SEQUENCE-END. +func yaml_sequence_end_event_initialize(event *yaml_event_t) bool { + *event = yaml_event_t{ + typ: yaml_SEQUENCE_END_EVENT, + } + return true +} + +// Create MAPPING-START. +func yaml_mapping_start_event_initialize(event *yaml_event_t, anchor, tag []byte, implicit bool, style yaml_mapping_style_t) bool { + *event = yaml_event_t{ + typ: yaml_MAPPING_START_EVENT, + anchor: anchor, + tag: tag, + implicit: implicit, + style: yaml_style_t(style), + } + return true +} + +// Create MAPPING-END. +func yaml_mapping_end_event_initialize(event *yaml_event_t) bool { + *event = yaml_event_t{ + typ: yaml_MAPPING_END_EVENT, + } + return true +} + +// Destroy an event object. +func yaml_event_delete(event *yaml_event_t) { + *event = yaml_event_t{} +} + +///* +// * Create a document object. +// */ +// +//YAML_DECLARE(int) +//yaml_document_initialize(document *yaml_document_t, +// version_directive *yaml_version_directive_t, +// tag_directives_start *yaml_tag_directive_t, +// tag_directives_end *yaml_tag_directive_t, +// start_implicit int, end_implicit int) +//{ +// struct { +// error yaml_error_type_t +// } context +// struct { +// start *yaml_node_t +// end *yaml_node_t +// top *yaml_node_t +// } nodes = { NULL, NULL, NULL } +// version_directive_copy *yaml_version_directive_t = NULL +// struct { +// start *yaml_tag_directive_t +// end *yaml_tag_directive_t +// top *yaml_tag_directive_t +// } tag_directives_copy = { NULL, NULL, NULL } +// value yaml_tag_directive_t = { NULL, NULL } +// mark yaml_mark_t = { 0, 0, 0 } +// +// assert(document) // Non-NULL document object is expected. +// assert((tag_directives_start && tag_directives_end) || +// (tag_directives_start == tag_directives_end)) +// // Valid tag directives are expected. +// +// if (!STACK_INIT(&context, nodes, INITIAL_STACK_SIZE)) goto error +// +// if (version_directive) { +// version_directive_copy = yaml_malloc(sizeof(yaml_version_directive_t)) +// if (!version_directive_copy) goto error +// version_directive_copy.major = version_directive.major +// version_directive_copy.minor = version_directive.minor +// } +// +// if (tag_directives_start != tag_directives_end) { +// tag_directive *yaml_tag_directive_t +// if (!STACK_INIT(&context, tag_directives_copy, INITIAL_STACK_SIZE)) +// goto error +// for (tag_directive = tag_directives_start +// tag_directive != tag_directives_end; tag_directive ++) { +// assert(tag_directive.handle) +// assert(tag_directive.prefix) +// if (!yaml_check_utf8(tag_directive.handle, +// strlen((char *)tag_directive.handle))) +// goto error +// if (!yaml_check_utf8(tag_directive.prefix, +// strlen((char *)tag_directive.prefix))) +// goto error +// value.handle = yaml_strdup(tag_directive.handle) +// value.prefix = yaml_strdup(tag_directive.prefix) +// if (!value.handle || !value.prefix) goto error +// if (!PUSH(&context, tag_directives_copy, value)) +// goto error +// value.handle = NULL +// value.prefix = NULL +// } +// } +// +// DOCUMENT_INIT(*document, nodes.start, nodes.end, version_directive_copy, +// tag_directives_copy.start, tag_directives_copy.top, +// start_implicit, end_implicit, mark, mark) +// +// return 1 +// +//error: +// STACK_DEL(&context, nodes) +// yaml_free(version_directive_copy) +// while (!STACK_EMPTY(&context, tag_directives_copy)) { +// value yaml_tag_directive_t = POP(&context, tag_directives_copy) +// yaml_free(value.handle) +// yaml_free(value.prefix) +// } +// STACK_DEL(&context, tag_directives_copy) +// yaml_free(value.handle) +// yaml_free(value.prefix) +// +// return 0 +//} +// +///* +// * Destroy a document object. +// */ +// +//YAML_DECLARE(void) +//yaml_document_delete(document *yaml_document_t) +//{ +// struct { +// error yaml_error_type_t +// } context +// tag_directive *yaml_tag_directive_t +// +// context.error = YAML_NO_ERROR // Eliminate a compliler warning. +// +// assert(document) // Non-NULL document object is expected. +// +// while (!STACK_EMPTY(&context, document.nodes)) { +// node yaml_node_t = POP(&context, document.nodes) +// yaml_free(node.tag) +// switch (node.type) { +// case YAML_SCALAR_NODE: +// yaml_free(node.data.scalar.value) +// break +// case YAML_SEQUENCE_NODE: +// STACK_DEL(&context, node.data.sequence.items) +// break +// case YAML_MAPPING_NODE: +// STACK_DEL(&context, node.data.mapping.pairs) +// break +// default: +// assert(0) // Should not happen. +// } +// } +// STACK_DEL(&context, document.nodes) +// +// yaml_free(document.version_directive) +// for (tag_directive = document.tag_directives.start +// tag_directive != document.tag_directives.end +// tag_directive++) { +// yaml_free(tag_directive.handle) +// yaml_free(tag_directive.prefix) +// } +// yaml_free(document.tag_directives.start) +// +// memset(document, 0, sizeof(yaml_document_t)) +//} +// +///** +// * Get a document node. +// */ +// +//YAML_DECLARE(yaml_node_t *) +//yaml_document_get_node(document *yaml_document_t, index int) +//{ +// assert(document) // Non-NULL document object is expected. +// +// if (index > 0 && document.nodes.start + index <= document.nodes.top) { +// return document.nodes.start + index - 1 +// } +// return NULL +//} +// +///** +// * Get the root object. +// */ +// +//YAML_DECLARE(yaml_node_t *) +//yaml_document_get_root_node(document *yaml_document_t) +//{ +// assert(document) // Non-NULL document object is expected. +// +// if (document.nodes.top != document.nodes.start) { +// return document.nodes.start +// } +// return NULL +//} +// +///* +// * Add a scalar node to a document. +// */ +// +//YAML_DECLARE(int) +//yaml_document_add_scalar(document *yaml_document_t, +// tag *yaml_char_t, value *yaml_char_t, length int, +// style yaml_scalar_style_t) +//{ +// struct { +// error yaml_error_type_t +// } context +// mark yaml_mark_t = { 0, 0, 0 } +// tag_copy *yaml_char_t = NULL +// value_copy *yaml_char_t = NULL +// node yaml_node_t +// +// assert(document) // Non-NULL document object is expected. +// assert(value) // Non-NULL value is expected. +// +// if (!tag) { +// tag = (yaml_char_t *)YAML_DEFAULT_SCALAR_TAG +// } +// +// if (!yaml_check_utf8(tag, strlen((char *)tag))) goto error +// tag_copy = yaml_strdup(tag) +// if (!tag_copy) goto error +// +// if (length < 0) { +// length = strlen((char *)value) +// } +// +// if (!yaml_check_utf8(value, length)) goto error +// value_copy = yaml_malloc(length+1) +// if (!value_copy) goto error +// memcpy(value_copy, value, length) +// value_copy[length] = '\0' +// +// SCALAR_NODE_INIT(node, tag_copy, value_copy, length, style, mark, mark) +// if (!PUSH(&context, document.nodes, node)) goto error +// +// return document.nodes.top - document.nodes.start +// +//error: +// yaml_free(tag_copy) +// yaml_free(value_copy) +// +// return 0 +//} +// +///* +// * Add a sequence node to a document. +// */ +// +//YAML_DECLARE(int) +//yaml_document_add_sequence(document *yaml_document_t, +// tag *yaml_char_t, style yaml_sequence_style_t) +//{ +// struct { +// error yaml_error_type_t +// } context +// mark yaml_mark_t = { 0, 0, 0 } +// tag_copy *yaml_char_t = NULL +// struct { +// start *yaml_node_item_t +// end *yaml_node_item_t +// top *yaml_node_item_t +// } items = { NULL, NULL, NULL } +// node yaml_node_t +// +// assert(document) // Non-NULL document object is expected. +// +// if (!tag) { +// tag = (yaml_char_t *)YAML_DEFAULT_SEQUENCE_TAG +// } +// +// if (!yaml_check_utf8(tag, strlen((char *)tag))) goto error +// tag_copy = yaml_strdup(tag) +// if (!tag_copy) goto error +// +// if (!STACK_INIT(&context, items, INITIAL_STACK_SIZE)) goto error +// +// SEQUENCE_NODE_INIT(node, tag_copy, items.start, items.end, +// style, mark, mark) +// if (!PUSH(&context, document.nodes, node)) goto error +// +// return document.nodes.top - document.nodes.start +// +//error: +// STACK_DEL(&context, items) +// yaml_free(tag_copy) +// +// return 0 +//} +// +///* +// * Add a mapping node to a document. +// */ +// +//YAML_DECLARE(int) +//yaml_document_add_mapping(document *yaml_document_t, +// tag *yaml_char_t, style yaml_mapping_style_t) +//{ +// struct { +// error yaml_error_type_t +// } context +// mark yaml_mark_t = { 0, 0, 0 } +// tag_copy *yaml_char_t = NULL +// struct { +// start *yaml_node_pair_t +// end *yaml_node_pair_t +// top *yaml_node_pair_t +// } pairs = { NULL, NULL, NULL } +// node yaml_node_t +// +// assert(document) // Non-NULL document object is expected. +// +// if (!tag) { +// tag = (yaml_char_t *)YAML_DEFAULT_MAPPING_TAG +// } +// +// if (!yaml_check_utf8(tag, strlen((char *)tag))) goto error +// tag_copy = yaml_strdup(tag) +// if (!tag_copy) goto error +// +// if (!STACK_INIT(&context, pairs, INITIAL_STACK_SIZE)) goto error +// +// MAPPING_NODE_INIT(node, tag_copy, pairs.start, pairs.end, +// style, mark, mark) +// if (!PUSH(&context, document.nodes, node)) goto error +// +// return document.nodes.top - document.nodes.start +// +//error: +// STACK_DEL(&context, pairs) +// yaml_free(tag_copy) +// +// return 0 +//} +// +///* +// * Append an item to a sequence node. +// */ +// +//YAML_DECLARE(int) +//yaml_document_append_sequence_item(document *yaml_document_t, +// sequence int, item int) +//{ +// struct { +// error yaml_error_type_t +// } context +// +// assert(document) // Non-NULL document is required. +// assert(sequence > 0 +// && document.nodes.start + sequence <= document.nodes.top) +// // Valid sequence id is required. +// assert(document.nodes.start[sequence-1].type == YAML_SEQUENCE_NODE) +// // A sequence node is required. +// assert(item > 0 && document.nodes.start + item <= document.nodes.top) +// // Valid item id is required. +// +// if (!PUSH(&context, +// document.nodes.start[sequence-1].data.sequence.items, item)) +// return 0 +// +// return 1 +//} +// +///* +// * Append a pair of a key and a value to a mapping node. +// */ +// +//YAML_DECLARE(int) +//yaml_document_append_mapping_pair(document *yaml_document_t, +// mapping int, key int, value int) +//{ +// struct { +// error yaml_error_type_t +// } context +// +// pair yaml_node_pair_t +// +// assert(document) // Non-NULL document is required. +// assert(mapping > 0 +// && document.nodes.start + mapping <= document.nodes.top) +// // Valid mapping id is required. +// assert(document.nodes.start[mapping-1].type == YAML_MAPPING_NODE) +// // A mapping node is required. +// assert(key > 0 && document.nodes.start + key <= document.nodes.top) +// // Valid key id is required. +// assert(value > 0 && document.nodes.start + value <= document.nodes.top) +// // Valid value id is required. +// +// pair.key = key +// pair.value = value +// +// if (!PUSH(&context, +// document.nodes.start[mapping-1].data.mapping.pairs, pair)) +// return 0 +// +// return 1 +//} +// +// diff --git a/generator/vendor/gopkg.in/yaml.v2/decode.go b/generator/vendor/gopkg.in/yaml.v2/decode.go new file mode 100644 index 00000000..db1f5f20 --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/decode.go @@ -0,0 +1,685 @@ +package yaml + +import ( + "encoding" + "encoding/base64" + "fmt" + "math" + "reflect" + "strconv" + "time" +) + +const ( + documentNode = 1 << iota + mappingNode + sequenceNode + scalarNode + aliasNode +) + +type node struct { + kind int + line, column int + tag string + value string + implicit bool + children []*node + anchors map[string]*node +} + +// ---------------------------------------------------------------------------- +// Parser, produces a node tree out of a libyaml event stream. + +type parser struct { + parser yaml_parser_t + event yaml_event_t + doc *node +} + +func newParser(b []byte) *parser { + p := parser{} + if !yaml_parser_initialize(&p.parser) { + panic("failed to initialize YAML emitter") + } + + if len(b) == 0 { + b = []byte{'\n'} + } + + yaml_parser_set_input_string(&p.parser, b) + + p.skip() + if p.event.typ != yaml_STREAM_START_EVENT { + panic("expected stream start event, got " + strconv.Itoa(int(p.event.typ))) + } + p.skip() + return &p +} + +func (p *parser) destroy() { + if p.event.typ != yaml_NO_EVENT { + yaml_event_delete(&p.event) + } + yaml_parser_delete(&p.parser) +} + +func (p *parser) skip() { + if p.event.typ != yaml_NO_EVENT { + if p.event.typ == yaml_STREAM_END_EVENT { + failf("attempted to go past the end of stream; corrupted value?") + } + yaml_event_delete(&p.event) + } + if !yaml_parser_parse(&p.parser, &p.event) { + p.fail() + } +} + +func (p *parser) fail() { + var where string + var line int + if p.parser.problem_mark.line != 0 { + line = p.parser.problem_mark.line + } else if p.parser.context_mark.line != 0 { + line = p.parser.context_mark.line + } + if line != 0 { + where = "line " + strconv.Itoa(line) + ": " + } + var msg string + if len(p.parser.problem) > 0 { + msg = p.parser.problem + } else { + msg = "unknown problem parsing YAML content" + } + failf("%s%s", where, msg) +} + +func (p *parser) anchor(n *node, anchor []byte) { + if anchor != nil { + p.doc.anchors[string(anchor)] = n + } +} + +func (p *parser) parse() *node { + switch p.event.typ { + case yaml_SCALAR_EVENT: + return p.scalar() + case yaml_ALIAS_EVENT: + return p.alias() + case yaml_MAPPING_START_EVENT: + return p.mapping() + case yaml_SEQUENCE_START_EVENT: + return p.sequence() + case yaml_DOCUMENT_START_EVENT: + return p.document() + case yaml_STREAM_END_EVENT: + // Happens when attempting to decode an empty buffer. + return nil + default: + panic("attempted to parse unknown event: " + strconv.Itoa(int(p.event.typ))) + } +} + +func (p *parser) node(kind int) *node { + return &node{ + kind: kind, + line: p.event.start_mark.line, + column: p.event.start_mark.column, + } +} + +func (p *parser) document() *node { + n := p.node(documentNode) + n.anchors = make(map[string]*node) + p.doc = n + p.skip() + n.children = append(n.children, p.parse()) + if p.event.typ != yaml_DOCUMENT_END_EVENT { + panic("expected end of document event but got " + strconv.Itoa(int(p.event.typ))) + } + p.skip() + return n +} + +func (p *parser) alias() *node { + n := p.node(aliasNode) + n.value = string(p.event.anchor) + p.skip() + return n +} + +func (p *parser) scalar() *node { + n := p.node(scalarNode) + n.value = string(p.event.value) + n.tag = string(p.event.tag) + n.implicit = p.event.implicit + p.anchor(n, p.event.anchor) + p.skip() + return n +} + +func (p *parser) sequence() *node { + n := p.node(sequenceNode) + p.anchor(n, p.event.anchor) + p.skip() + for p.event.typ != yaml_SEQUENCE_END_EVENT { + n.children = append(n.children, p.parse()) + } + p.skip() + return n +} + +func (p *parser) mapping() *node { + n := p.node(mappingNode) + p.anchor(n, p.event.anchor) + p.skip() + for p.event.typ != yaml_MAPPING_END_EVENT { + n.children = append(n.children, p.parse(), p.parse()) + } + p.skip() + return n +} + +// ---------------------------------------------------------------------------- +// Decoder, unmarshals a node into a provided value. + +type decoder struct { + doc *node + aliases map[string]bool + mapType reflect.Type + terrors []string + strict bool +} + +var ( + mapItemType = reflect.TypeOf(MapItem{}) + durationType = reflect.TypeOf(time.Duration(0)) + defaultMapType = reflect.TypeOf(map[interface{}]interface{}{}) + ifaceType = defaultMapType.Elem() +) + +func newDecoder(strict bool) *decoder { + d := &decoder{mapType: defaultMapType, strict: strict} + d.aliases = make(map[string]bool) + return d +} + +func (d *decoder) terror(n *node, tag string, out reflect.Value) { + if n.tag != "" { + tag = n.tag + } + value := n.value + if tag != yaml_SEQ_TAG && tag != yaml_MAP_TAG { + if len(value) > 10 { + value = " `" + value[:7] + "...`" + } else { + value = " `" + value + "`" + } + } + d.terrors = append(d.terrors, fmt.Sprintf("line %d: cannot unmarshal %s%s into %s", n.line+1, shortTag(tag), value, out.Type())) +} + +func (d *decoder) callUnmarshaler(n *node, u Unmarshaler) (good bool) { + terrlen := len(d.terrors) + err := u.UnmarshalYAML(func(v interface{}) (err error) { + defer handleErr(&err) + d.unmarshal(n, reflect.ValueOf(v)) + if len(d.terrors) > terrlen { + issues := d.terrors[terrlen:] + d.terrors = d.terrors[:terrlen] + return &TypeError{issues} + } + return nil + }) + if e, ok := err.(*TypeError); ok { + d.terrors = append(d.terrors, e.Errors...) + return false + } + if err != nil { + fail(err) + } + return true +} + +// d.prepare initializes and dereferences pointers and calls UnmarshalYAML +// if a value is found to implement it. +// It returns the initialized and dereferenced out value, whether +// unmarshalling was already done by UnmarshalYAML, and if so whether +// its types unmarshalled appropriately. +// +// If n holds a null value, prepare returns before doing anything. +func (d *decoder) prepare(n *node, out reflect.Value) (newout reflect.Value, unmarshaled, good bool) { + if n.tag == yaml_NULL_TAG || n.kind == scalarNode && n.tag == "" && (n.value == "null" || n.value == "" && n.implicit) { + return out, false, false + } + again := true + for again { + again = false + if out.Kind() == reflect.Ptr { + if out.IsNil() { + out.Set(reflect.New(out.Type().Elem())) + } + out = out.Elem() + again = true + } + if out.CanAddr() { + if u, ok := out.Addr().Interface().(Unmarshaler); ok { + good = d.callUnmarshaler(n, u) + return out, true, good + } + } + } + return out, false, false +} + +func (d *decoder) unmarshal(n *node, out reflect.Value) (good bool) { + switch n.kind { + case documentNode: + return d.document(n, out) + case aliasNode: + return d.alias(n, out) + } + out, unmarshaled, good := d.prepare(n, out) + if unmarshaled { + return good + } + switch n.kind { + case scalarNode: + good = d.scalar(n, out) + case mappingNode: + good = d.mapping(n, out) + case sequenceNode: + good = d.sequence(n, out) + default: + panic("internal error: unknown node kind: " + strconv.Itoa(n.kind)) + } + return good +} + +func (d *decoder) document(n *node, out reflect.Value) (good bool) { + if len(n.children) == 1 { + d.doc = n + d.unmarshal(n.children[0], out) + return true + } + return false +} + +func (d *decoder) alias(n *node, out reflect.Value) (good bool) { + an, ok := d.doc.anchors[n.value] + if !ok { + failf("unknown anchor '%s' referenced", n.value) + } + if d.aliases[n.value] { + failf("anchor '%s' value contains itself", n.value) + } + d.aliases[n.value] = true + good = d.unmarshal(an, out) + delete(d.aliases, n.value) + return good +} + +var zeroValue reflect.Value + +func resetMap(out reflect.Value) { + for _, k := range out.MapKeys() { + out.SetMapIndex(k, zeroValue) + } +} + +func (d *decoder) scalar(n *node, out reflect.Value) (good bool) { + var tag string + var resolved interface{} + if n.tag == "" && !n.implicit { + tag = yaml_STR_TAG + resolved = n.value + } else { + tag, resolved = resolve(n.tag, n.value) + if tag == yaml_BINARY_TAG { + data, err := base64.StdEncoding.DecodeString(resolved.(string)) + if err != nil { + failf("!!binary value contains invalid base64 data") + } + resolved = string(data) + } + } + if resolved == nil { + if out.Kind() == reflect.Map && !out.CanAddr() { + resetMap(out) + } else { + out.Set(reflect.Zero(out.Type())) + } + return true + } + if s, ok := resolved.(string); ok && out.CanAddr() { + if u, ok := out.Addr().Interface().(encoding.TextUnmarshaler); ok { + err := u.UnmarshalText([]byte(s)) + if err != nil { + fail(err) + } + return true + } + } + switch out.Kind() { + case reflect.String: + if tag == yaml_BINARY_TAG { + out.SetString(resolved.(string)) + good = true + } else if resolved != nil { + out.SetString(n.value) + good = true + } + case reflect.Interface: + if resolved == nil { + out.Set(reflect.Zero(out.Type())) + } else { + out.Set(reflect.ValueOf(resolved)) + } + good = true + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + switch resolved := resolved.(type) { + case int: + if !out.OverflowInt(int64(resolved)) { + out.SetInt(int64(resolved)) + good = true + } + case int64: + if !out.OverflowInt(resolved) { + out.SetInt(resolved) + good = true + } + case uint64: + if resolved <= math.MaxInt64 && !out.OverflowInt(int64(resolved)) { + out.SetInt(int64(resolved)) + good = true + } + case float64: + if resolved <= math.MaxInt64 && !out.OverflowInt(int64(resolved)) { + out.SetInt(int64(resolved)) + good = true + } + case string: + if out.Type() == durationType { + d, err := time.ParseDuration(resolved) + if err == nil { + out.SetInt(int64(d)) + good = true + } + } + } + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: + switch resolved := resolved.(type) { + case int: + if resolved >= 0 && !out.OverflowUint(uint64(resolved)) { + out.SetUint(uint64(resolved)) + good = true + } + case int64: + if resolved >= 0 && !out.OverflowUint(uint64(resolved)) { + out.SetUint(uint64(resolved)) + good = true + } + case uint64: + if !out.OverflowUint(uint64(resolved)) { + out.SetUint(uint64(resolved)) + good = true + } + case float64: + if resolved <= math.MaxUint64 && !out.OverflowUint(uint64(resolved)) { + out.SetUint(uint64(resolved)) + good = true + } + } + case reflect.Bool: + switch resolved := resolved.(type) { + case bool: + out.SetBool(resolved) + good = true + } + case reflect.Float32, reflect.Float64: + switch resolved := resolved.(type) { + case int: + out.SetFloat(float64(resolved)) + good = true + case int64: + out.SetFloat(float64(resolved)) + good = true + case uint64: + out.SetFloat(float64(resolved)) + good = true + case float64: + out.SetFloat(resolved) + good = true + } + case reflect.Ptr: + if out.Type().Elem() == reflect.TypeOf(resolved) { + // TODO DOes this make sense? When is out a Ptr except when decoding a nil value? + elem := reflect.New(out.Type().Elem()) + elem.Elem().Set(reflect.ValueOf(resolved)) + out.Set(elem) + good = true + } + } + if !good { + d.terror(n, tag, out) + } + return good +} + +func settableValueOf(i interface{}) reflect.Value { + v := reflect.ValueOf(i) + sv := reflect.New(v.Type()).Elem() + sv.Set(v) + return sv +} + +func (d *decoder) sequence(n *node, out reflect.Value) (good bool) { + l := len(n.children) + + var iface reflect.Value + switch out.Kind() { + case reflect.Slice: + out.Set(reflect.MakeSlice(out.Type(), l, l)) + case reflect.Interface: + // No type hints. Will have to use a generic sequence. + iface = out + out = settableValueOf(make([]interface{}, l)) + default: + d.terror(n, yaml_SEQ_TAG, out) + return false + } + et := out.Type().Elem() + + j := 0 + for i := 0; i < l; i++ { + e := reflect.New(et).Elem() + if ok := d.unmarshal(n.children[i], e); ok { + out.Index(j).Set(e) + j++ + } + } + out.Set(out.Slice(0, j)) + if iface.IsValid() { + iface.Set(out) + } + return true +} + +func (d *decoder) mapping(n *node, out reflect.Value) (good bool) { + switch out.Kind() { + case reflect.Struct: + return d.mappingStruct(n, out) + case reflect.Slice: + return d.mappingSlice(n, out) + case reflect.Map: + // okay + case reflect.Interface: + if d.mapType.Kind() == reflect.Map { + iface := out + out = reflect.MakeMap(d.mapType) + iface.Set(out) + } else { + slicev := reflect.New(d.mapType).Elem() + if !d.mappingSlice(n, slicev) { + return false + } + out.Set(slicev) + return true + } + default: + d.terror(n, yaml_MAP_TAG, out) + return false + } + outt := out.Type() + kt := outt.Key() + et := outt.Elem() + + mapType := d.mapType + if outt.Key() == ifaceType && outt.Elem() == ifaceType { + d.mapType = outt + } + + if out.IsNil() { + out.Set(reflect.MakeMap(outt)) + } + l := len(n.children) + for i := 0; i < l; i += 2 { + if isMerge(n.children[i]) { + d.merge(n.children[i+1], out) + continue + } + k := reflect.New(kt).Elem() + if d.unmarshal(n.children[i], k) { + kkind := k.Kind() + if kkind == reflect.Interface { + kkind = k.Elem().Kind() + } + if kkind == reflect.Map || kkind == reflect.Slice { + failf("invalid map key: %#v", k.Interface()) + } + e := reflect.New(et).Elem() + if d.unmarshal(n.children[i+1], e) { + out.SetMapIndex(k, e) + } + } + } + d.mapType = mapType + return true +} + +func (d *decoder) mappingSlice(n *node, out reflect.Value) (good bool) { + outt := out.Type() + if outt.Elem() != mapItemType { + d.terror(n, yaml_MAP_TAG, out) + return false + } + + mapType := d.mapType + d.mapType = outt + + var slice []MapItem + var l = len(n.children) + for i := 0; i < l; i += 2 { + if isMerge(n.children[i]) { + d.merge(n.children[i+1], out) + continue + } + item := MapItem{} + k := reflect.ValueOf(&item.Key).Elem() + if d.unmarshal(n.children[i], k) { + v := reflect.ValueOf(&item.Value).Elem() + if d.unmarshal(n.children[i+1], v) { + slice = append(slice, item) + } + } + } + out.Set(reflect.ValueOf(slice)) + d.mapType = mapType + return true +} + +func (d *decoder) mappingStruct(n *node, out reflect.Value) (good bool) { + sinfo, err := getStructInfo(out.Type()) + if err != nil { + panic(err) + } + name := settableValueOf("") + l := len(n.children) + + var inlineMap reflect.Value + var elemType reflect.Type + if sinfo.InlineMap != -1 { + inlineMap = out.Field(sinfo.InlineMap) + inlineMap.Set(reflect.New(inlineMap.Type()).Elem()) + elemType = inlineMap.Type().Elem() + } + + for i := 0; i < l; i += 2 { + ni := n.children[i] + if isMerge(ni) { + d.merge(n.children[i+1], out) + continue + } + if !d.unmarshal(ni, name) { + continue + } + if info, ok := sinfo.FieldsMap[name.String()]; ok { + var field reflect.Value + if info.Inline == nil { + field = out.Field(info.Num) + } else { + field = out.FieldByIndex(info.Inline) + } + d.unmarshal(n.children[i+1], field) + } else if sinfo.InlineMap != -1 { + if inlineMap.IsNil() { + inlineMap.Set(reflect.MakeMap(inlineMap.Type())) + } + value := reflect.New(elemType).Elem() + d.unmarshal(n.children[i+1], value) + inlineMap.SetMapIndex(name, value) + } else if d.strict { + d.terrors = append(d.terrors, fmt.Sprintf("line %d: field %s not found in struct %s", n.line+1, name.String(), out.Type())) + } + } + return true +} + +func failWantMap() { + failf("map merge requires map or sequence of maps as the value") +} + +func (d *decoder) merge(n *node, out reflect.Value) { + switch n.kind { + case mappingNode: + d.unmarshal(n, out) + case aliasNode: + an, ok := d.doc.anchors[n.value] + if ok && an.kind != mappingNode { + failWantMap() + } + d.unmarshal(n, out) + case sequenceNode: + // Step backwards as earlier nodes take precedence. + for i := len(n.children) - 1; i >= 0; i-- { + ni := n.children[i] + if ni.kind == aliasNode { + an, ok := d.doc.anchors[ni.value] + if ok && an.kind != mappingNode { + failWantMap() + } + } else if ni.kind != mappingNode { + failWantMap() + } + d.unmarshal(ni, out) + } + default: + failWantMap() + } +} + +func isMerge(n *node) bool { + return n.kind == scalarNode && n.value == "<<" && (n.implicit == true || n.tag == yaml_MERGE_TAG) +} diff --git a/generator/vendor/gopkg.in/yaml.v2/decode_test.go b/generator/vendor/gopkg.in/yaml.v2/decode_test.go new file mode 100644 index 00000000..713b1ee9 --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/decode_test.go @@ -0,0 +1,1017 @@ +package yaml_test + +import ( + "errors" + . "gopkg.in/check.v1" + "gopkg.in/yaml.v2" + "math" + "net" + "reflect" + "strings" + "time" +) + +var unmarshalIntTest = 123 + +var unmarshalTests = []struct { + data string + value interface{} +}{ + { + "", + &struct{}{}, + }, { + "{}", &struct{}{}, + }, { + "v: hi", + map[string]string{"v": "hi"}, + }, { + "v: hi", map[string]interface{}{"v": "hi"}, + }, { + "v: true", + map[string]string{"v": "true"}, + }, { + "v: true", + map[string]interface{}{"v": true}, + }, { + "v: 10", + map[string]interface{}{"v": 10}, + }, { + "v: 0b10", + map[string]interface{}{"v": 2}, + }, { + "v: 0xA", + map[string]interface{}{"v": 10}, + }, { + "v: 4294967296", + map[string]int64{"v": 4294967296}, + }, { + "v: 0.1", + map[string]interface{}{"v": 0.1}, + }, { + "v: .1", + map[string]interface{}{"v": 0.1}, + }, { + "v: .Inf", + map[string]interface{}{"v": math.Inf(+1)}, + }, { + "v: -.Inf", + map[string]interface{}{"v": math.Inf(-1)}, + }, { + "v: -10", + map[string]interface{}{"v": -10}, + }, { + "v: -.1", + map[string]interface{}{"v": -0.1}, + }, + + // Simple values. + { + "123", + &unmarshalIntTest, + }, + + // Floats from spec + { + "canonical: 6.8523e+5", + map[string]interface{}{"canonical": 6.8523e+5}, + }, { + "expo: 685.230_15e+03", + map[string]interface{}{"expo": 685.23015e+03}, + }, { + "fixed: 685_230.15", + map[string]interface{}{"fixed": 685230.15}, + }, { + "neginf: -.inf", + map[string]interface{}{"neginf": math.Inf(-1)}, + }, { + "fixed: 685_230.15", + map[string]float64{"fixed": 685230.15}, + }, + //{"sexa: 190:20:30.15", map[string]interface{}{"sexa": 0}}, // Unsupported + //{"notanum: .NaN", map[string]interface{}{"notanum": math.NaN()}}, // Equality of NaN fails. + + // Bools from spec + { + "canonical: y", + map[string]interface{}{"canonical": true}, + }, { + "answer: NO", + map[string]interface{}{"answer": false}, + }, { + "logical: True", + map[string]interface{}{"logical": true}, + }, { + "option: on", + map[string]interface{}{"option": true}, + }, { + "option: on", + map[string]bool{"option": true}, + }, + // Ints from spec + { + "canonical: 685230", + map[string]interface{}{"canonical": 685230}, + }, { + "decimal: +685_230", + map[string]interface{}{"decimal": 685230}, + }, { + "octal: 02472256", + map[string]interface{}{"octal": 685230}, + }, { + "hexa: 0x_0A_74_AE", + map[string]interface{}{"hexa": 685230}, + }, { + "bin: 0b1010_0111_0100_1010_1110", + map[string]interface{}{"bin": 685230}, + }, { + "bin: -0b101010", + map[string]interface{}{"bin": -42}, + }, { + "decimal: +685_230", + map[string]int{"decimal": 685230}, + }, + + //{"sexa: 190:20:30", map[string]interface{}{"sexa": 0}}, // Unsupported + + // Nulls from spec + { + "empty:", + map[string]interface{}{"empty": nil}, + }, { + "canonical: ~", + map[string]interface{}{"canonical": nil}, + }, { + "english: null", + map[string]interface{}{"english": nil}, + }, { + "~: null key", + map[interface{}]string{nil: "null key"}, + }, { + "empty:", + map[string]*bool{"empty": nil}, + }, + + // Flow sequence + { + "seq: [A,B]", + map[string]interface{}{"seq": []interface{}{"A", "B"}}, + }, { + "seq: [A,B,C,]", + map[string][]string{"seq": []string{"A", "B", "C"}}, + }, { + "seq: [A,1,C]", + map[string][]string{"seq": []string{"A", "1", "C"}}, + }, { + "seq: [A,1,C]", + map[string][]int{"seq": []int{1}}, + }, { + "seq: [A,1,C]", + map[string]interface{}{"seq": []interface{}{"A", 1, "C"}}, + }, + // Block sequence + { + "seq:\n - A\n - B", + map[string]interface{}{"seq": []interface{}{"A", "B"}}, + }, { + "seq:\n - A\n - B\n - C", + map[string][]string{"seq": []string{"A", "B", "C"}}, + }, { + "seq:\n - A\n - 1\n - C", + map[string][]string{"seq": []string{"A", "1", "C"}}, + }, { + "seq:\n - A\n - 1\n - C", + map[string][]int{"seq": []int{1}}, + }, { + "seq:\n - A\n - 1\n - C", + map[string]interface{}{"seq": []interface{}{"A", 1, "C"}}, + }, + + // Literal block scalar + { + "scalar: | # Comment\n\n literal\n\n \ttext\n\n", + map[string]string{"scalar": "\nliteral\n\n\ttext\n"}, + }, + + // Folded block scalar + { + "scalar: > # Comment\n\n folded\n line\n \n next\n line\n * one\n * two\n\n last\n line\n\n", + map[string]string{"scalar": "\nfolded line\nnext line\n * one\n * two\n\nlast line\n"}, + }, + + // Map inside interface with no type hints. + { + "a: {b: c}", + map[interface{}]interface{}{"a": map[interface{}]interface{}{"b": "c"}}, + }, + + // Structs and type conversions. + { + "hello: world", + &struct{ Hello string }{"world"}, + }, { + "a: {b: c}", + &struct{ A struct{ B string } }{struct{ B string }{"c"}}, + }, { + "a: {b: c}", + &struct{ A *struct{ B string } }{&struct{ B string }{"c"}}, + }, { + "a: {b: c}", + &struct{ A map[string]string }{map[string]string{"b": "c"}}, + }, { + "a: {b: c}", + &struct{ A *map[string]string }{&map[string]string{"b": "c"}}, + }, { + "a:", + &struct{ A map[string]string }{}, + }, { + "a: 1", + &struct{ A int }{1}, + }, { + "a: 1", + &struct{ A float64 }{1}, + }, { + "a: 1.0", + &struct{ A int }{1}, + }, { + "a: 1.0", + &struct{ A uint }{1}, + }, { + "a: [1, 2]", + &struct{ A []int }{[]int{1, 2}}, + }, { + "a: 1", + &struct{ B int }{0}, + }, { + "a: 1", + &struct { + B int "a" + }{1}, + }, { + "a: y", + &struct{ A bool }{true}, + }, + + // Some cross type conversions + { + "v: 42", + map[string]uint{"v": 42}, + }, { + "v: -42", + map[string]uint{}, + }, { + "v: 4294967296", + map[string]uint64{"v": 4294967296}, + }, { + "v: -4294967296", + map[string]uint64{}, + }, + + // int + { + "int_max: 2147483647", + map[string]int{"int_max": math.MaxInt32}, + }, + { + "int_min: -2147483648", + map[string]int{"int_min": math.MinInt32}, + }, + { + "int_overflow: 9223372036854775808", // math.MaxInt64 + 1 + map[string]int{}, + }, + + // int64 + { + "int64_max: 9223372036854775807", + map[string]int64{"int64_max": math.MaxInt64}, + }, + { + "int64_max_base2: 0b111111111111111111111111111111111111111111111111111111111111111", + map[string]int64{"int64_max_base2": math.MaxInt64}, + }, + { + "int64_min: -9223372036854775808", + map[string]int64{"int64_min": math.MinInt64}, + }, + { + "int64_neg_base2: -0b111111111111111111111111111111111111111111111111111111111111111", + map[string]int64{"int64_neg_base2": -math.MaxInt64}, + }, + { + "int64_overflow: 9223372036854775808", // math.MaxInt64 + 1 + map[string]int64{}, + }, + + // uint + { + "uint_min: 0", + map[string]uint{"uint_min": 0}, + }, + { + "uint_max: 4294967295", + map[string]uint{"uint_max": math.MaxUint32}, + }, + { + "uint_underflow: -1", + map[string]uint{}, + }, + + // uint64 + { + "uint64_min: 0", + map[string]uint{"uint64_min": 0}, + }, + { + "uint64_max: 18446744073709551615", + map[string]uint64{"uint64_max": math.MaxUint64}, + }, + { + "uint64_max_base2: 0b1111111111111111111111111111111111111111111111111111111111111111", + map[string]uint64{"uint64_max_base2": math.MaxUint64}, + }, + { + "uint64_maxint64: 9223372036854775807", + map[string]uint64{"uint64_maxint64": math.MaxInt64}, + }, + { + "uint64_underflow: -1", + map[string]uint64{}, + }, + + // float32 + { + "float32_max: 3.40282346638528859811704183484516925440e+38", + map[string]float32{"float32_max": math.MaxFloat32}, + }, + { + "float32_nonzero: 1.401298464324817070923729583289916131280e-45", + map[string]float32{"float32_nonzero": math.SmallestNonzeroFloat32}, + }, + { + "float32_maxuint64: 18446744073709551615", + map[string]float32{"float32_maxuint64": float32(math.MaxUint64)}, + }, + { + "float32_maxuint64+1: 18446744073709551616", + map[string]float32{"float32_maxuint64+1": float32(math.MaxUint64 + 1)}, + }, + + // float64 + { + "float64_max: 1.797693134862315708145274237317043567981e+308", + map[string]float64{"float64_max": math.MaxFloat64}, + }, + { + "float64_nonzero: 4.940656458412465441765687928682213723651e-324", + map[string]float64{"float64_nonzero": math.SmallestNonzeroFloat64}, + }, + { + "float64_maxuint64: 18446744073709551615", + map[string]float64{"float64_maxuint64": float64(math.MaxUint64)}, + }, + { + "float64_maxuint64+1: 18446744073709551616", + map[string]float64{"float64_maxuint64+1": float64(math.MaxUint64 + 1)}, + }, + + // Overflow cases. + { + "v: 4294967297", + map[string]int32{}, + }, { + "v: 128", + map[string]int8{}, + }, + + // Quoted values. + { + "'1': '\"2\"'", + map[interface{}]interface{}{"1": "\"2\""}, + }, { + "v:\n- A\n- 'B\n\n C'\n", + map[string][]string{"v": []string{"A", "B\nC"}}, + }, + + // Explicit tags. + { + "v: !!float '1.1'", + map[string]interface{}{"v": 1.1}, + }, { + "v: !!null ''", + map[string]interface{}{"v": nil}, + }, { + "%TAG !y! tag:yaml.org,2002:\n---\nv: !y!int '1'", + map[string]interface{}{"v": 1}, + }, + + // Non-specific tag (Issue #75) + { + "v: ! test", + map[string]interface{}{"v": "test"}, + }, + + // Anchors and aliases. + { + "a: &x 1\nb: &y 2\nc: *x\nd: *y\n", + &struct{ A, B, C, D int }{1, 2, 1, 2}, + }, { + "a: &a {c: 1}\nb: *a", + &struct { + A, B struct { + C int + } + }{struct{ C int }{1}, struct{ C int }{1}}, + }, { + "a: &a [1, 2]\nb: *a", + &struct{ B []int }{[]int{1, 2}}, + }, { + "b: *a\na: &a {c: 1}", + &struct { + A, B struct { + C int + } + }{struct{ C int }{1}, struct{ C int }{1}}, + }, + + // Bug #1133337 + { + "foo: ''", + map[string]*string{"foo": new(string)}, + }, { + "foo: null", + map[string]string{"foo": ""}, + }, { + "foo: null", + map[string]interface{}{"foo": nil}, + }, + + // Ignored field + { + "a: 1\nb: 2\n", + &struct { + A int + B int "-" + }{1, 0}, + }, + + // Bug #1191981 + { + "" + + "%YAML 1.1\n" + + "--- !!str\n" + + `"Generic line break (no glyph)\n\` + "\n" + + ` Generic line break (glyphed)\n\` + "\n" + + ` Line separator\u2028\` + "\n" + + ` Paragraph separator\u2029"` + "\n", + "" + + "Generic line break (no glyph)\n" + + "Generic line break (glyphed)\n" + + "Line separator\u2028Paragraph separator\u2029", + }, + + // Struct inlining + { + "a: 1\nb: 2\nc: 3\n", + &struct { + A int + C inlineB `yaml:",inline"` + }{1, inlineB{2, inlineC{3}}}, + }, + + // Map inlining + { + "a: 1\nb: 2\nc: 3\n", + &struct { + A int + C map[string]int `yaml:",inline"` + }{1, map[string]int{"b": 2, "c": 3}}, + }, + + // bug 1243827 + { + "a: -b_c", + map[string]interface{}{"a": "-b_c"}, + }, + { + "a: +b_c", + map[string]interface{}{"a": "+b_c"}, + }, + { + "a: 50cent_of_dollar", + map[string]interface{}{"a": "50cent_of_dollar"}, + }, + + // Duration + { + "a: 3s", + map[string]time.Duration{"a": 3 * time.Second}, + }, + + // Issue #24. + { + "a: <foo>", + map[string]string{"a": "<foo>"}, + }, + + // Base 60 floats are obsolete and unsupported. + { + "a: 1:1\n", + map[string]string{"a": "1:1"}, + }, + + // Binary data. + { + "a: !!binary gIGC\n", + map[string]string{"a": "\x80\x81\x82"}, + }, { + "a: !!binary |\n " + strings.Repeat("kJCQ", 17) + "kJ\n CQ\n", + map[string]string{"a": strings.Repeat("\x90", 54)}, + }, { + "a: !!binary |\n " + strings.Repeat("A", 70) + "\n ==\n", + map[string]string{"a": strings.Repeat("\x00", 52)}, + }, + + // Ordered maps. + { + "{b: 2, a: 1, d: 4, c: 3, sub: {e: 5}}", + &yaml.MapSlice{{"b", 2}, {"a", 1}, {"d", 4}, {"c", 3}, {"sub", yaml.MapSlice{{"e", 5}}}}, + }, + + // Issue #39. + { + "a:\n b:\n c: d\n", + map[string]struct{ B interface{} }{"a": {map[interface{}]interface{}{"c": "d"}}}, + }, + + // Custom map type. + { + "a: {b: c}", + M{"a": M{"b": "c"}}, + }, + + // Support encoding.TextUnmarshaler. + { + "a: 1.2.3.4\n", + map[string]net.IP{"a": net.IPv4(1, 2, 3, 4)}, + }, + { + "a: 2015-02-24T18:19:39Z\n", + map[string]time.Time{"a": time.Unix(1424801979, 0).In(time.UTC)}, + }, + + // Encode empty lists as zero-length slices. + { + "a: []", + &struct{ A []int }{[]int{}}, + }, + + // UTF-16-LE + { + "\xff\xfe\xf1\x00o\x00\xf1\x00o\x00:\x00 \x00v\x00e\x00r\x00y\x00 \x00y\x00e\x00s\x00\n\x00", + M{"ñoño": "very yes"}, + }, + // UTF-16-LE with surrogate. + { + "\xff\xfe\xf1\x00o\x00\xf1\x00o\x00:\x00 \x00v\x00e\x00r\x00y\x00 \x00y\x00e\x00s\x00 \x00=\xd8\xd4\xdf\n\x00", + M{"ñoño": "very yes 🟔"}, + }, + + // UTF-16-BE + { + "\xfe\xff\x00\xf1\x00o\x00\xf1\x00o\x00:\x00 \x00v\x00e\x00r\x00y\x00 \x00y\x00e\x00s\x00\n", + M{"ñoño": "very yes"}, + }, + // UTF-16-BE with surrogate. + { + "\xfe\xff\x00\xf1\x00o\x00\xf1\x00o\x00:\x00 \x00v\x00e\x00r\x00y\x00 \x00y\x00e\x00s\x00 \xd8=\xdf\xd4\x00\n", + M{"ñoño": "very yes 🟔"}, + }, + + // YAML Float regex shouldn't match this + { + "a: 123456e1\n", + M{"a": "123456e1"}, + }, { + "a: 123456E1\n", + M{"a": "123456E1"}, + }, +} + +type M map[interface{}]interface{} + +type inlineB struct { + B int + inlineC `yaml:",inline"` +} + +type inlineC struct { + C int +} + +func (s *S) TestUnmarshal(c *C) { + for i, item := range unmarshalTests { + c.Logf("test %d: %q", i, item.data) + t := reflect.ValueOf(item.value).Type() + var value interface{} + switch t.Kind() { + case reflect.Map: + value = reflect.MakeMap(t).Interface() + case reflect.String: + value = reflect.New(t).Interface() + case reflect.Ptr: + value = reflect.New(t.Elem()).Interface() + default: + c.Fatalf("missing case for %s", t) + } + err := yaml.Unmarshal([]byte(item.data), value) + if _, ok := err.(*yaml.TypeError); !ok { + c.Assert(err, IsNil) + } + if t.Kind() == reflect.String { + c.Assert(*value.(*string), Equals, item.value) + } else { + c.Assert(value, DeepEquals, item.value) + } + } +} + +func (s *S) TestUnmarshalNaN(c *C) { + value := map[string]interface{}{} + err := yaml.Unmarshal([]byte("notanum: .NaN"), &value) + c.Assert(err, IsNil) + c.Assert(math.IsNaN(value["notanum"].(float64)), Equals, true) +} + +var unmarshalErrorTests = []struct { + data, error string +}{ + {"v: !!float 'error'", "yaml: cannot decode !!str `error` as a !!float"}, + {"v: [A,", "yaml: line 1: did not find expected node content"}, + {"v:\n- [A,", "yaml: line 2: did not find expected node content"}, + {"a: *b\n", "yaml: unknown anchor 'b' referenced"}, + {"a: &a\n b: *a\n", "yaml: anchor 'a' value contains itself"}, + {"value: -", "yaml: block sequence entries are not allowed in this context"}, + {"a: !!binary ==", "yaml: !!binary value contains invalid base64 data"}, + {"{[.]}", `yaml: invalid map key: \[\]interface \{\}\{"\."\}`}, + {"{{.}}", `yaml: invalid map key: map\[interface\ \{\}\]interface \{\}\{".":interface \{\}\(nil\)\}`}, + {"%TAG !%79! tag:yaml.org,2002:\n---\nv: !%79!int '1'", "yaml: did not find expected whitespace"}, +} + +func (s *S) TestUnmarshalErrors(c *C) { + for _, item := range unmarshalErrorTests { + var value interface{} + err := yaml.Unmarshal([]byte(item.data), &value) + c.Assert(err, ErrorMatches, item.error, Commentf("Partial unmarshal: %#v", value)) + } +} + +var unmarshalerTests = []struct { + data, tag string + value interface{} +}{ + {"_: {hi: there}", "!!map", map[interface{}]interface{}{"hi": "there"}}, + {"_: [1,A]", "!!seq", []interface{}{1, "A"}}, + {"_: 10", "!!int", 10}, + {"_: null", "!!null", nil}, + {`_: BAR!`, "!!str", "BAR!"}, + {`_: "BAR!"`, "!!str", "BAR!"}, + {"_: !!foo 'BAR!'", "!!foo", "BAR!"}, + {`_: ""`, "!!str", ""}, +} + +var unmarshalerResult = map[int]error{} + +type unmarshalerType struct { + value interface{} +} + +func (o *unmarshalerType) UnmarshalYAML(unmarshal func(v interface{}) error) error { + if err := unmarshal(&o.value); err != nil { + return err + } + if i, ok := o.value.(int); ok { + if result, ok := unmarshalerResult[i]; ok { + return result + } + } + return nil +} + +type unmarshalerPointer struct { + Field *unmarshalerType "_" +} + +type unmarshalerValue struct { + Field unmarshalerType "_" +} + +func (s *S) TestUnmarshalerPointerField(c *C) { + for _, item := range unmarshalerTests { + obj := &unmarshalerPointer{} + err := yaml.Unmarshal([]byte(item.data), obj) + c.Assert(err, IsNil) + if item.value == nil { + c.Assert(obj.Field, IsNil) + } else { + c.Assert(obj.Field, NotNil, Commentf("Pointer not initialized (%#v)", item.value)) + c.Assert(obj.Field.value, DeepEquals, item.value) + } + } +} + +func (s *S) TestUnmarshalerValueField(c *C) { + for _, item := range unmarshalerTests { + obj := &unmarshalerValue{} + err := yaml.Unmarshal([]byte(item.data), obj) + c.Assert(err, IsNil) + c.Assert(obj.Field, NotNil, Commentf("Pointer not initialized (%#v)", item.value)) + c.Assert(obj.Field.value, DeepEquals, item.value) + } +} + +func (s *S) TestUnmarshalerWholeDocument(c *C) { + obj := &unmarshalerType{} + err := yaml.Unmarshal([]byte(unmarshalerTests[0].data), obj) + c.Assert(err, IsNil) + value, ok := obj.value.(map[interface{}]interface{}) + c.Assert(ok, Equals, true, Commentf("value: %#v", obj.value)) + c.Assert(value["_"], DeepEquals, unmarshalerTests[0].value) +} + +func (s *S) TestUnmarshalerTypeError(c *C) { + unmarshalerResult[2] = &yaml.TypeError{[]string{"foo"}} + unmarshalerResult[4] = &yaml.TypeError{[]string{"bar"}} + defer func() { + delete(unmarshalerResult, 2) + delete(unmarshalerResult, 4) + }() + + type T struct { + Before int + After int + M map[string]*unmarshalerType + } + var v T + data := `{before: A, m: {abc: 1, def: 2, ghi: 3, jkl: 4}, after: B}` + err := yaml.Unmarshal([]byte(data), &v) + c.Assert(err, ErrorMatches, ""+ + "yaml: unmarshal errors:\n"+ + " line 1: cannot unmarshal !!str `A` into int\n"+ + " foo\n"+ + " bar\n"+ + " line 1: cannot unmarshal !!str `B` into int") + c.Assert(v.M["abc"], NotNil) + c.Assert(v.M["def"], IsNil) + c.Assert(v.M["ghi"], NotNil) + c.Assert(v.M["jkl"], IsNil) + + c.Assert(v.M["abc"].value, Equals, 1) + c.Assert(v.M["ghi"].value, Equals, 3) +} + +type proxyTypeError struct{} + +func (v *proxyTypeError) UnmarshalYAML(unmarshal func(interface{}) error) error { + var s string + var a int32 + var b int64 + if err := unmarshal(&s); err != nil { + panic(err) + } + if s == "a" { + if err := unmarshal(&b); err == nil { + panic("should have failed") + } + return unmarshal(&a) + } + if err := unmarshal(&a); err == nil { + panic("should have failed") + } + return unmarshal(&b) +} + +func (s *S) TestUnmarshalerTypeErrorProxying(c *C) { + type T struct { + Before int + After int + M map[string]*proxyTypeError + } + var v T + data := `{before: A, m: {abc: a, def: b}, after: B}` + err := yaml.Unmarshal([]byte(data), &v) + c.Assert(err, ErrorMatches, ""+ + "yaml: unmarshal errors:\n"+ + " line 1: cannot unmarshal !!str `A` into int\n"+ + " line 1: cannot unmarshal !!str `a` into int32\n"+ + " line 1: cannot unmarshal !!str `b` into int64\n"+ + " line 1: cannot unmarshal !!str `B` into int") +} + +type failingUnmarshaler struct{} + +var failingErr = errors.New("failingErr") + +func (ft *failingUnmarshaler) UnmarshalYAML(unmarshal func(interface{}) error) error { + return failingErr +} + +func (s *S) TestUnmarshalerError(c *C) { + err := yaml.Unmarshal([]byte("a: b"), &failingUnmarshaler{}) + c.Assert(err, Equals, failingErr) +} + +type sliceUnmarshaler []int + +func (su *sliceUnmarshaler) UnmarshalYAML(unmarshal func(interface{}) error) error { + var slice []int + err := unmarshal(&slice) + if err == nil { + *su = slice + return nil + } + + var intVal int + err = unmarshal(&intVal) + if err == nil { + *su = []int{intVal} + return nil + } + + return err +} + +func (s *S) TestUnmarshalerRetry(c *C) { + var su sliceUnmarshaler + err := yaml.Unmarshal([]byte("[1, 2, 3]"), &su) + c.Assert(err, IsNil) + c.Assert(su, DeepEquals, sliceUnmarshaler([]int{1, 2, 3})) + + err = yaml.Unmarshal([]byte("1"), &su) + c.Assert(err, IsNil) + c.Assert(su, DeepEquals, sliceUnmarshaler([]int{1})) +} + +// From http://yaml.org/type/merge.html +var mergeTests = ` +anchors: + list: + - &CENTER { "x": 1, "y": 2 } + - &LEFT { "x": 0, "y": 2 } + - &BIG { "r": 10 } + - &SMALL { "r": 1 } + +# All the following maps are equal: + +plain: + # Explicit keys + "x": 1 + "y": 2 + "r": 10 + label: center/big + +mergeOne: + # Merge one map + << : *CENTER + "r": 10 + label: center/big + +mergeMultiple: + # Merge multiple maps + << : [ *CENTER, *BIG ] + label: center/big + +override: + # Override + << : [ *BIG, *LEFT, *SMALL ] + "x": 1 + label: center/big + +shortTag: + # Explicit short merge tag + !!merge "<<" : [ *CENTER, *BIG ] + label: center/big + +longTag: + # Explicit merge long tag + !<tag:yaml.org,2002:merge> "<<" : [ *CENTER, *BIG ] + label: center/big + +inlineMap: + # Inlined map + << : {"x": 1, "y": 2, "r": 10} + label: center/big + +inlineSequenceMap: + # Inlined map in sequence + << : [ *CENTER, {"r": 10} ] + label: center/big +` + +func (s *S) TestMerge(c *C) { + var want = map[interface{}]interface{}{ + "x": 1, + "y": 2, + "r": 10, + "label": "center/big", + } + + var m map[interface{}]interface{} + err := yaml.Unmarshal([]byte(mergeTests), &m) + c.Assert(err, IsNil) + for name, test := range m { + if name == "anchors" { + continue + } + c.Assert(test, DeepEquals, want, Commentf("test %q failed", name)) + } +} + +func (s *S) TestMergeStruct(c *C) { + type Data struct { + X, Y, R int + Label string + } + want := Data{1, 2, 10, "center/big"} + + var m map[string]Data + err := yaml.Unmarshal([]byte(mergeTests), &m) + c.Assert(err, IsNil) + for name, test := range m { + if name == "anchors" { + continue + } + c.Assert(test, Equals, want, Commentf("test %q failed", name)) + } +} + +var unmarshalNullTests = []func() interface{}{ + func() interface{} { var v interface{}; v = "v"; return &v }, + func() interface{} { var s = "s"; return &s }, + func() interface{} { var s = "s"; sptr := &s; return &sptr }, + func() interface{} { var i = 1; return &i }, + func() interface{} { var i = 1; iptr := &i; return &iptr }, + func() interface{} { m := map[string]int{"s": 1}; return &m }, + func() interface{} { m := map[string]int{"s": 1}; return m }, +} + +func (s *S) TestUnmarshalNull(c *C) { + for _, test := range unmarshalNullTests { + item := test() + zero := reflect.Zero(reflect.TypeOf(item).Elem()).Interface() + err := yaml.Unmarshal([]byte("null"), item) + c.Assert(err, IsNil) + if reflect.TypeOf(item).Kind() == reflect.Map { + c.Assert(reflect.ValueOf(item).Interface(), DeepEquals, reflect.MakeMap(reflect.TypeOf(item)).Interface()) + } else { + c.Assert(reflect.ValueOf(item).Elem().Interface(), DeepEquals, zero) + } + } +} + +func (s *S) TestUnmarshalSliceOnPreset(c *C) { + // Issue #48. + v := struct{ A []int }{[]int{1}} + yaml.Unmarshal([]byte("a: [2]"), &v) + c.Assert(v.A, DeepEquals, []int{2}) +} + +func (s *S) TestUnmarshalStrict(c *C) { + v := struct{ A, B int }{} + + err := yaml.UnmarshalStrict([]byte("a: 1\nb: 2"), &v) + c.Check(err, IsNil) + err = yaml.Unmarshal([]byte("a: 1\nb: 2\nc: 3"), &v) + c.Check(err, IsNil) + err = yaml.UnmarshalStrict([]byte("a: 1\nb: 2\nc: 3"), &v) + c.Check(err, ErrorMatches, "yaml: unmarshal errors:\n line 1: field c not found in struct struct { A int; B int }") +} + +//var data []byte +//func init() { +// var err error +// data, err = ioutil.ReadFile("/tmp/file.yaml") +// if err != nil { +// panic(err) +// } +//} +// +//func (s *S) BenchmarkUnmarshal(c *C) { +// var err error +// for i := 0; i < c.N; i++ { +// var v map[string]interface{} +// err = yaml.Unmarshal(data, &v) +// } +// if err != nil { +// panic(err) +// } +//} +// +//func (s *S) BenchmarkMarshal(c *C) { +// var v map[string]interface{} +// yaml.Unmarshal(data, &v) +// c.ResetTimer() +// for i := 0; i < c.N; i++ { +// yaml.Marshal(&v) +// } +//} diff --git a/generator/vendor/gopkg.in/yaml.v2/emitterc.go b/generator/vendor/gopkg.in/yaml.v2/emitterc.go new file mode 100644 index 00000000..41de8b85 --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/emitterc.go @@ -0,0 +1,1684 @@ +package yaml + +import ( + "bytes" +) + +// Flush the buffer if needed. +func flush(emitter *yaml_emitter_t) bool { + if emitter.buffer_pos+5 >= len(emitter.buffer) { + return yaml_emitter_flush(emitter) + } + return true +} + +// Put a character to the output buffer. +func put(emitter *yaml_emitter_t, value byte) bool { + if emitter.buffer_pos+5 >= len(emitter.buffer) && !yaml_emitter_flush(emitter) { + return false + } + emitter.buffer[emitter.buffer_pos] = value + emitter.buffer_pos++ + emitter.column++ + return true +} + +// Put a line break to the output buffer. +func put_break(emitter *yaml_emitter_t) bool { + if emitter.buffer_pos+5 >= len(emitter.buffer) && !yaml_emitter_flush(emitter) { + return false + } + switch emitter.line_break { + case yaml_CR_BREAK: + emitter.buffer[emitter.buffer_pos] = '\r' + emitter.buffer_pos += 1 + case yaml_LN_BREAK: + emitter.buffer[emitter.buffer_pos] = '\n' + emitter.buffer_pos += 1 + case yaml_CRLN_BREAK: + emitter.buffer[emitter.buffer_pos+0] = '\r' + emitter.buffer[emitter.buffer_pos+1] = '\n' + emitter.buffer_pos += 2 + default: + panic("unknown line break setting") + } + emitter.column = 0 + emitter.line++ + return true +} + +// Copy a character from a string into buffer. +func write(emitter *yaml_emitter_t, s []byte, i *int) bool { + if emitter.buffer_pos+5 >= len(emitter.buffer) && !yaml_emitter_flush(emitter) { + return false + } + p := emitter.buffer_pos + w := width(s[*i]) + switch w { + case 4: + emitter.buffer[p+3] = s[*i+3] + fallthrough + case 3: + emitter.buffer[p+2] = s[*i+2] + fallthrough + case 2: + emitter.buffer[p+1] = s[*i+1] + fallthrough + case 1: + emitter.buffer[p+0] = s[*i+0] + default: + panic("unknown character width") + } + emitter.column++ + emitter.buffer_pos += w + *i += w + return true +} + +// Write a whole string into buffer. +func write_all(emitter *yaml_emitter_t, s []byte) bool { + for i := 0; i < len(s); { + if !write(emitter, s, &i) { + return false + } + } + return true +} + +// Copy a line break character from a string into buffer. +func write_break(emitter *yaml_emitter_t, s []byte, i *int) bool { + if s[*i] == '\n' { + if !put_break(emitter) { + return false + } + *i++ + } else { + if !write(emitter, s, i) { + return false + } + emitter.column = 0 + emitter.line++ + } + return true +} + +// Set an emitter error and return false. +func yaml_emitter_set_emitter_error(emitter *yaml_emitter_t, problem string) bool { + emitter.error = yaml_EMITTER_ERROR + emitter.problem = problem + return false +} + +// Emit an event. +func yaml_emitter_emit(emitter *yaml_emitter_t, event *yaml_event_t) bool { + emitter.events = append(emitter.events, *event) + for !yaml_emitter_need_more_events(emitter) { + event := &emitter.events[emitter.events_head] + if !yaml_emitter_analyze_event(emitter, event) { + return false + } + if !yaml_emitter_state_machine(emitter, event) { + return false + } + yaml_event_delete(event) + emitter.events_head++ + } + return true +} + +// Check if we need to accumulate more events before emitting. +// +// We accumulate extra +// - 1 event for DOCUMENT-START +// - 2 events for SEQUENCE-START +// - 3 events for MAPPING-START +// +func yaml_emitter_need_more_events(emitter *yaml_emitter_t) bool { + if emitter.events_head == len(emitter.events) { + return true + } + var accumulate int + switch emitter.events[emitter.events_head].typ { + case yaml_DOCUMENT_START_EVENT: + accumulate = 1 + break + case yaml_SEQUENCE_START_EVENT: + accumulate = 2 + break + case yaml_MAPPING_START_EVENT: + accumulate = 3 + break + default: + return false + } + if len(emitter.events)-emitter.events_head > accumulate { + return false + } + var level int + for i := emitter.events_head; i < len(emitter.events); i++ { + switch emitter.events[i].typ { + case yaml_STREAM_START_EVENT, yaml_DOCUMENT_START_EVENT, yaml_SEQUENCE_START_EVENT, yaml_MAPPING_START_EVENT: + level++ + case yaml_STREAM_END_EVENT, yaml_DOCUMENT_END_EVENT, yaml_SEQUENCE_END_EVENT, yaml_MAPPING_END_EVENT: + level-- + } + if level == 0 { + return false + } + } + return true +} + +// Append a directive to the directives stack. +func yaml_emitter_append_tag_directive(emitter *yaml_emitter_t, value *yaml_tag_directive_t, allow_duplicates bool) bool { + for i := 0; i < len(emitter.tag_directives); i++ { + if bytes.Equal(value.handle, emitter.tag_directives[i].handle) { + if allow_duplicates { + return true + } + return yaml_emitter_set_emitter_error(emitter, "duplicate %TAG directive") + } + } + + // [Go] Do we actually need to copy this given garbage collection + // and the lack of deallocating destructors? + tag_copy := yaml_tag_directive_t{ + handle: make([]byte, len(value.handle)), + prefix: make([]byte, len(value.prefix)), + } + copy(tag_copy.handle, value.handle) + copy(tag_copy.prefix, value.prefix) + emitter.tag_directives = append(emitter.tag_directives, tag_copy) + return true +} + +// Increase the indentation level. +func yaml_emitter_increase_indent(emitter *yaml_emitter_t, flow, indentless bool) bool { + emitter.indents = append(emitter.indents, emitter.indent) + if emitter.indent < 0 { + if flow { + emitter.indent = emitter.best_indent + } else { + emitter.indent = 0 + } + } else if !indentless { + emitter.indent += emitter.best_indent + } + return true +} + +// State dispatcher. +func yaml_emitter_state_machine(emitter *yaml_emitter_t, event *yaml_event_t) bool { + switch emitter.state { + default: + case yaml_EMIT_STREAM_START_STATE: + return yaml_emitter_emit_stream_start(emitter, event) + + case yaml_EMIT_FIRST_DOCUMENT_START_STATE: + return yaml_emitter_emit_document_start(emitter, event, true) + + case yaml_EMIT_DOCUMENT_START_STATE: + return yaml_emitter_emit_document_start(emitter, event, false) + + case yaml_EMIT_DOCUMENT_CONTENT_STATE: + return yaml_emitter_emit_document_content(emitter, event) + + case yaml_EMIT_DOCUMENT_END_STATE: + return yaml_emitter_emit_document_end(emitter, event) + + case yaml_EMIT_FLOW_SEQUENCE_FIRST_ITEM_STATE: + return yaml_emitter_emit_flow_sequence_item(emitter, event, true) + + case yaml_EMIT_FLOW_SEQUENCE_ITEM_STATE: + return yaml_emitter_emit_flow_sequence_item(emitter, event, false) + + case yaml_EMIT_FLOW_MAPPING_FIRST_KEY_STATE: + return yaml_emitter_emit_flow_mapping_key(emitter, event, true) + + case yaml_EMIT_FLOW_MAPPING_KEY_STATE: + return yaml_emitter_emit_flow_mapping_key(emitter, event, false) + + case yaml_EMIT_FLOW_MAPPING_SIMPLE_VALUE_STATE: + return yaml_emitter_emit_flow_mapping_value(emitter, event, true) + + case yaml_EMIT_FLOW_MAPPING_VALUE_STATE: + return yaml_emitter_emit_flow_mapping_value(emitter, event, false) + + case yaml_EMIT_BLOCK_SEQUENCE_FIRST_ITEM_STATE: + return yaml_emitter_emit_block_sequence_item(emitter, event, true) + + case yaml_EMIT_BLOCK_SEQUENCE_ITEM_STATE: + return yaml_emitter_emit_block_sequence_item(emitter, event, false) + + case yaml_EMIT_BLOCK_MAPPING_FIRST_KEY_STATE: + return yaml_emitter_emit_block_mapping_key(emitter, event, true) + + case yaml_EMIT_BLOCK_MAPPING_KEY_STATE: + return yaml_emitter_emit_block_mapping_key(emitter, event, false) + + case yaml_EMIT_BLOCK_MAPPING_SIMPLE_VALUE_STATE: + return yaml_emitter_emit_block_mapping_value(emitter, event, true) + + case yaml_EMIT_BLOCK_MAPPING_VALUE_STATE: + return yaml_emitter_emit_block_mapping_value(emitter, event, false) + + case yaml_EMIT_END_STATE: + return yaml_emitter_set_emitter_error(emitter, "expected nothing after STREAM-END") + } + panic("invalid emitter state") +} + +// Expect STREAM-START. +func yaml_emitter_emit_stream_start(emitter *yaml_emitter_t, event *yaml_event_t) bool { + if event.typ != yaml_STREAM_START_EVENT { + return yaml_emitter_set_emitter_error(emitter, "expected STREAM-START") + } + if emitter.encoding == yaml_ANY_ENCODING { + emitter.encoding = event.encoding + if emitter.encoding == yaml_ANY_ENCODING { + emitter.encoding = yaml_UTF8_ENCODING + } + } + if emitter.best_indent < 2 || emitter.best_indent > 9 { + emitter.best_indent = 2 + } + if emitter.best_width >= 0 && emitter.best_width <= emitter.best_indent*2 { + emitter.best_width = 80 + } + if emitter.best_width < 0 { + emitter.best_width = 1<<31 - 1 + } + if emitter.line_break == yaml_ANY_BREAK { + emitter.line_break = yaml_LN_BREAK + } + + emitter.indent = -1 + emitter.line = 0 + emitter.column = 0 + emitter.whitespace = true + emitter.indention = true + + if emitter.encoding != yaml_UTF8_ENCODING { + if !yaml_emitter_write_bom(emitter) { + return false + } + } + emitter.state = yaml_EMIT_FIRST_DOCUMENT_START_STATE + return true +} + +// Expect DOCUMENT-START or STREAM-END. +func yaml_emitter_emit_document_start(emitter *yaml_emitter_t, event *yaml_event_t, first bool) bool { + + if event.typ == yaml_DOCUMENT_START_EVENT { + + if event.version_directive != nil { + if !yaml_emitter_analyze_version_directive(emitter, event.version_directive) { + return false + } + } + + for i := 0; i < len(event.tag_directives); i++ { + tag_directive := &event.tag_directives[i] + if !yaml_emitter_analyze_tag_directive(emitter, tag_directive) { + return false + } + if !yaml_emitter_append_tag_directive(emitter, tag_directive, false) { + return false + } + } + + for i := 0; i < len(default_tag_directives); i++ { + tag_directive := &default_tag_directives[i] + if !yaml_emitter_append_tag_directive(emitter, tag_directive, true) { + return false + } + } + + implicit := event.implicit + if !first || emitter.canonical { + implicit = false + } + + if emitter.open_ended && (event.version_directive != nil || len(event.tag_directives) > 0) { + if !yaml_emitter_write_indicator(emitter, []byte("..."), true, false, false) { + return false + } + if !yaml_emitter_write_indent(emitter) { + return false + } + } + + if event.version_directive != nil { + implicit = false + if !yaml_emitter_write_indicator(emitter, []byte("%YAML"), true, false, false) { + return false + } + if !yaml_emitter_write_indicator(emitter, []byte("1.1"), true, false, false) { + return false + } + if !yaml_emitter_write_indent(emitter) { + return false + } + } + + if len(event.tag_directives) > 0 { + implicit = false + for i := 0; i < len(event.tag_directives); i++ { + tag_directive := &event.tag_directives[i] + if !yaml_emitter_write_indicator(emitter, []byte("%TAG"), true, false, false) { + return false + } + if !yaml_emitter_write_tag_handle(emitter, tag_directive.handle) { + return false + } + if !yaml_emitter_write_tag_content(emitter, tag_directive.prefix, true) { + return false + } + if !yaml_emitter_write_indent(emitter) { + return false + } + } + } + + if yaml_emitter_check_empty_document(emitter) { + implicit = false + } + if !implicit { + if !yaml_emitter_write_indent(emitter) { + return false + } + if !yaml_emitter_write_indicator(emitter, []byte("---"), true, false, false) { + return false + } + if emitter.canonical { + if !yaml_emitter_write_indent(emitter) { + return false + } + } + } + + emitter.state = yaml_EMIT_DOCUMENT_CONTENT_STATE + return true + } + + if event.typ == yaml_STREAM_END_EVENT { + if emitter.open_ended { + if !yaml_emitter_write_indicator(emitter, []byte("..."), true, false, false) { + return false + } + if !yaml_emitter_write_indent(emitter) { + return false + } + } + if !yaml_emitter_flush(emitter) { + return false + } + emitter.state = yaml_EMIT_END_STATE + return true + } + + return yaml_emitter_set_emitter_error(emitter, "expected DOCUMENT-START or STREAM-END") +} + +// Expect the root node. +func yaml_emitter_emit_document_content(emitter *yaml_emitter_t, event *yaml_event_t) bool { + emitter.states = append(emitter.states, yaml_EMIT_DOCUMENT_END_STATE) + return yaml_emitter_emit_node(emitter, event, true, false, false, false) +} + +// Expect DOCUMENT-END. +func yaml_emitter_emit_document_end(emitter *yaml_emitter_t, event *yaml_event_t) bool { + if event.typ != yaml_DOCUMENT_END_EVENT { + return yaml_emitter_set_emitter_error(emitter, "expected DOCUMENT-END") + } + if !yaml_emitter_write_indent(emitter) { + return false + } + if !event.implicit { + // [Go] Allocate the slice elsewhere. + if !yaml_emitter_write_indicator(emitter, []byte("..."), true, false, false) { + return false + } + if !yaml_emitter_write_indent(emitter) { + return false + } + } + if !yaml_emitter_flush(emitter) { + return false + } + emitter.state = yaml_EMIT_DOCUMENT_START_STATE + emitter.tag_directives = emitter.tag_directives[:0] + return true +} + +// Expect a flow item node. +func yaml_emitter_emit_flow_sequence_item(emitter *yaml_emitter_t, event *yaml_event_t, first bool) bool { + if first { + if !yaml_emitter_write_indicator(emitter, []byte{'['}, true, true, false) { + return false + } + if !yaml_emitter_increase_indent(emitter, true, false) { + return false + } + emitter.flow_level++ + } + + if event.typ == yaml_SEQUENCE_END_EVENT { + emitter.flow_level-- + emitter.indent = emitter.indents[len(emitter.indents)-1] + emitter.indents = emitter.indents[:len(emitter.indents)-1] + if emitter.canonical && !first { + if !yaml_emitter_write_indicator(emitter, []byte{','}, false, false, false) { + return false + } + if !yaml_emitter_write_indent(emitter) { + return false + } + } + if !yaml_emitter_write_indicator(emitter, []byte{']'}, false, false, false) { + return false + } + emitter.state = emitter.states[len(emitter.states)-1] + emitter.states = emitter.states[:len(emitter.states)-1] + + return true + } + + if !first { + if !yaml_emitter_write_indicator(emitter, []byte{','}, false, false, false) { + return false + } + } + + if emitter.canonical || emitter.column > emitter.best_width { + if !yaml_emitter_write_indent(emitter) { + return false + } + } + emitter.states = append(emitter.states, yaml_EMIT_FLOW_SEQUENCE_ITEM_STATE) + return yaml_emitter_emit_node(emitter, event, false, true, false, false) +} + +// Expect a flow key node. +func yaml_emitter_emit_flow_mapping_key(emitter *yaml_emitter_t, event *yaml_event_t, first bool) bool { + if first { + if !yaml_emitter_write_indicator(emitter, []byte{'{'}, true, true, false) { + return false + } + if !yaml_emitter_increase_indent(emitter, true, false) { + return false + } + emitter.flow_level++ + } + + if event.typ == yaml_MAPPING_END_EVENT { + emitter.flow_level-- + emitter.indent = emitter.indents[len(emitter.indents)-1] + emitter.indents = emitter.indents[:len(emitter.indents)-1] + if emitter.canonical && !first { + if !yaml_emitter_write_indicator(emitter, []byte{','}, false, false, false) { + return false + } + if !yaml_emitter_write_indent(emitter) { + return false + } + } + if !yaml_emitter_write_indicator(emitter, []byte{'}'}, false, false, false) { + return false + } + emitter.state = emitter.states[len(emitter.states)-1] + emitter.states = emitter.states[:len(emitter.states)-1] + return true + } + + if !first { + if !yaml_emitter_write_indicator(emitter, []byte{','}, false, false, false) { + return false + } + } + if emitter.canonical || emitter.column > emitter.best_width { + if !yaml_emitter_write_indent(emitter) { + return false + } + } + + if !emitter.canonical && yaml_emitter_check_simple_key(emitter) { + emitter.states = append(emitter.states, yaml_EMIT_FLOW_MAPPING_SIMPLE_VALUE_STATE) + return yaml_emitter_emit_node(emitter, event, false, false, true, true) + } + if !yaml_emitter_write_indicator(emitter, []byte{'?'}, true, false, false) { + return false + } + emitter.states = append(emitter.states, yaml_EMIT_FLOW_MAPPING_VALUE_STATE) + return yaml_emitter_emit_node(emitter, event, false, false, true, false) +} + +// Expect a flow value node. +func yaml_emitter_emit_flow_mapping_value(emitter *yaml_emitter_t, event *yaml_event_t, simple bool) bool { + if simple { + if !yaml_emitter_write_indicator(emitter, []byte{':'}, false, false, false) { + return false + } + } else { + if emitter.canonical || emitter.column > emitter.best_width { + if !yaml_emitter_write_indent(emitter) { + return false + } + } + if !yaml_emitter_write_indicator(emitter, []byte{':'}, true, false, false) { + return false + } + } + emitter.states = append(emitter.states, yaml_EMIT_FLOW_MAPPING_KEY_STATE) + return yaml_emitter_emit_node(emitter, event, false, false, true, false) +} + +// Expect a block item node. +func yaml_emitter_emit_block_sequence_item(emitter *yaml_emitter_t, event *yaml_event_t, first bool) bool { + if first { + if !yaml_emitter_increase_indent(emitter, false, emitter.mapping_context && !emitter.indention) { + return false + } + } + if event.typ == yaml_SEQUENCE_END_EVENT { + emitter.indent = emitter.indents[len(emitter.indents)-1] + emitter.indents = emitter.indents[:len(emitter.indents)-1] + emitter.state = emitter.states[len(emitter.states)-1] + emitter.states = emitter.states[:len(emitter.states)-1] + return true + } + if !yaml_emitter_write_indent(emitter) { + return false + } + if !yaml_emitter_write_indicator(emitter, []byte{'-'}, true, false, true) { + return false + } + emitter.states = append(emitter.states, yaml_EMIT_BLOCK_SEQUENCE_ITEM_STATE) + return yaml_emitter_emit_node(emitter, event, false, true, false, false) +} + +// Expect a block key node. +func yaml_emitter_emit_block_mapping_key(emitter *yaml_emitter_t, event *yaml_event_t, first bool) bool { + if first { + if !yaml_emitter_increase_indent(emitter, false, false) { + return false + } + } + if event.typ == yaml_MAPPING_END_EVENT { + emitter.indent = emitter.indents[len(emitter.indents)-1] + emitter.indents = emitter.indents[:len(emitter.indents)-1] + emitter.state = emitter.states[len(emitter.states)-1] + emitter.states = emitter.states[:len(emitter.states)-1] + return true + } + if !yaml_emitter_write_indent(emitter) { + return false + } + if yaml_emitter_check_simple_key(emitter) { + emitter.states = append(emitter.states, yaml_EMIT_BLOCK_MAPPING_SIMPLE_VALUE_STATE) + return yaml_emitter_emit_node(emitter, event, false, false, true, true) + } + if !yaml_emitter_write_indicator(emitter, []byte{'?'}, true, false, true) { + return false + } + emitter.states = append(emitter.states, yaml_EMIT_BLOCK_MAPPING_VALUE_STATE) + return yaml_emitter_emit_node(emitter, event, false, false, true, false) +} + +// Expect a block value node. +func yaml_emitter_emit_block_mapping_value(emitter *yaml_emitter_t, event *yaml_event_t, simple bool) bool { + if simple { + if !yaml_emitter_write_indicator(emitter, []byte{':'}, false, false, false) { + return false + } + } else { + if !yaml_emitter_write_indent(emitter) { + return false + } + if !yaml_emitter_write_indicator(emitter, []byte{':'}, true, false, true) { + return false + } + } + emitter.states = append(emitter.states, yaml_EMIT_BLOCK_MAPPING_KEY_STATE) + return yaml_emitter_emit_node(emitter, event, false, false, true, false) +} + +// Expect a node. +func yaml_emitter_emit_node(emitter *yaml_emitter_t, event *yaml_event_t, + root bool, sequence bool, mapping bool, simple_key bool) bool { + + emitter.root_context = root + emitter.sequence_context = sequence + emitter.mapping_context = mapping + emitter.simple_key_context = simple_key + + switch event.typ { + case yaml_ALIAS_EVENT: + return yaml_emitter_emit_alias(emitter, event) + case yaml_SCALAR_EVENT: + return yaml_emitter_emit_scalar(emitter, event) + case yaml_SEQUENCE_START_EVENT: + return yaml_emitter_emit_sequence_start(emitter, event) + case yaml_MAPPING_START_EVENT: + return yaml_emitter_emit_mapping_start(emitter, event) + default: + return yaml_emitter_set_emitter_error(emitter, + "expected SCALAR, SEQUENCE-START, MAPPING-START, or ALIAS") + } +} + +// Expect ALIAS. +func yaml_emitter_emit_alias(emitter *yaml_emitter_t, event *yaml_event_t) bool { + if !yaml_emitter_process_anchor(emitter) { + return false + } + emitter.state = emitter.states[len(emitter.states)-1] + emitter.states = emitter.states[:len(emitter.states)-1] + return true +} + +// Expect SCALAR. +func yaml_emitter_emit_scalar(emitter *yaml_emitter_t, event *yaml_event_t) bool { + if !yaml_emitter_select_scalar_style(emitter, event) { + return false + } + if !yaml_emitter_process_anchor(emitter) { + return false + } + if !yaml_emitter_process_tag(emitter) { + return false + } + if !yaml_emitter_increase_indent(emitter, true, false) { + return false + } + if !yaml_emitter_process_scalar(emitter) { + return false + } + emitter.indent = emitter.indents[len(emitter.indents)-1] + emitter.indents = emitter.indents[:len(emitter.indents)-1] + emitter.state = emitter.states[len(emitter.states)-1] + emitter.states = emitter.states[:len(emitter.states)-1] + return true +} + +// Expect SEQUENCE-START. +func yaml_emitter_emit_sequence_start(emitter *yaml_emitter_t, event *yaml_event_t) bool { + if !yaml_emitter_process_anchor(emitter) { + return false + } + if !yaml_emitter_process_tag(emitter) { + return false + } + if emitter.flow_level > 0 || emitter.canonical || event.sequence_style() == yaml_FLOW_SEQUENCE_STYLE || + yaml_emitter_check_empty_sequence(emitter) { + emitter.state = yaml_EMIT_FLOW_SEQUENCE_FIRST_ITEM_STATE + } else { + emitter.state = yaml_EMIT_BLOCK_SEQUENCE_FIRST_ITEM_STATE + } + return true +} + +// Expect MAPPING-START. +func yaml_emitter_emit_mapping_start(emitter *yaml_emitter_t, event *yaml_event_t) bool { + if !yaml_emitter_process_anchor(emitter) { + return false + } + if !yaml_emitter_process_tag(emitter) { + return false + } + if emitter.flow_level > 0 || emitter.canonical || event.mapping_style() == yaml_FLOW_MAPPING_STYLE || + yaml_emitter_check_empty_mapping(emitter) { + emitter.state = yaml_EMIT_FLOW_MAPPING_FIRST_KEY_STATE + } else { + emitter.state = yaml_EMIT_BLOCK_MAPPING_FIRST_KEY_STATE + } + return true +} + +// Check if the document content is an empty scalar. +func yaml_emitter_check_empty_document(emitter *yaml_emitter_t) bool { + return false // [Go] Huh? +} + +// Check if the next events represent an empty sequence. +func yaml_emitter_check_empty_sequence(emitter *yaml_emitter_t) bool { + if len(emitter.events)-emitter.events_head < 2 { + return false + } + return emitter.events[emitter.events_head].typ == yaml_SEQUENCE_START_EVENT && + emitter.events[emitter.events_head+1].typ == yaml_SEQUENCE_END_EVENT +} + +// Check if the next events represent an empty mapping. +func yaml_emitter_check_empty_mapping(emitter *yaml_emitter_t) bool { + if len(emitter.events)-emitter.events_head < 2 { + return false + } + return emitter.events[emitter.events_head].typ == yaml_MAPPING_START_EVENT && + emitter.events[emitter.events_head+1].typ == yaml_MAPPING_END_EVENT +} + +// Check if the next node can be expressed as a simple key. +func yaml_emitter_check_simple_key(emitter *yaml_emitter_t) bool { + length := 0 + switch emitter.events[emitter.events_head].typ { + case yaml_ALIAS_EVENT: + length += len(emitter.anchor_data.anchor) + case yaml_SCALAR_EVENT: + if emitter.scalar_data.multiline { + return false + } + length += len(emitter.anchor_data.anchor) + + len(emitter.tag_data.handle) + + len(emitter.tag_data.suffix) + + len(emitter.scalar_data.value) + case yaml_SEQUENCE_START_EVENT: + if !yaml_emitter_check_empty_sequence(emitter) { + return false + } + length += len(emitter.anchor_data.anchor) + + len(emitter.tag_data.handle) + + len(emitter.tag_data.suffix) + case yaml_MAPPING_START_EVENT: + if !yaml_emitter_check_empty_mapping(emitter) { + return false + } + length += len(emitter.anchor_data.anchor) + + len(emitter.tag_data.handle) + + len(emitter.tag_data.suffix) + default: + return false + } + return length <= 128 +} + +// Determine an acceptable scalar style. +func yaml_emitter_select_scalar_style(emitter *yaml_emitter_t, event *yaml_event_t) bool { + + no_tag := len(emitter.tag_data.handle) == 0 && len(emitter.tag_data.suffix) == 0 + if no_tag && !event.implicit && !event.quoted_implicit { + return yaml_emitter_set_emitter_error(emitter, "neither tag nor implicit flags are specified") + } + + style := event.scalar_style() + if style == yaml_ANY_SCALAR_STYLE { + style = yaml_PLAIN_SCALAR_STYLE + } + if emitter.canonical { + style = yaml_DOUBLE_QUOTED_SCALAR_STYLE + } + if emitter.simple_key_context && emitter.scalar_data.multiline { + style = yaml_DOUBLE_QUOTED_SCALAR_STYLE + } + + if style == yaml_PLAIN_SCALAR_STYLE { + if emitter.flow_level > 0 && !emitter.scalar_data.flow_plain_allowed || + emitter.flow_level == 0 && !emitter.scalar_data.block_plain_allowed { + style = yaml_SINGLE_QUOTED_SCALAR_STYLE + } + if len(emitter.scalar_data.value) == 0 && (emitter.flow_level > 0 || emitter.simple_key_context) { + style = yaml_SINGLE_QUOTED_SCALAR_STYLE + } + if no_tag && !event.implicit { + style = yaml_SINGLE_QUOTED_SCALAR_STYLE + } + } + if style == yaml_SINGLE_QUOTED_SCALAR_STYLE { + if !emitter.scalar_data.single_quoted_allowed { + style = yaml_DOUBLE_QUOTED_SCALAR_STYLE + } + } + if style == yaml_LITERAL_SCALAR_STYLE || style == yaml_FOLDED_SCALAR_STYLE { + if !emitter.scalar_data.block_allowed || emitter.flow_level > 0 || emitter.simple_key_context { + style = yaml_DOUBLE_QUOTED_SCALAR_STYLE + } + } + + if no_tag && !event.quoted_implicit && style != yaml_PLAIN_SCALAR_STYLE { + emitter.tag_data.handle = []byte{'!'} + } + emitter.scalar_data.style = style + return true +} + +// Write an achor. +func yaml_emitter_process_anchor(emitter *yaml_emitter_t) bool { + if emitter.anchor_data.anchor == nil { + return true + } + c := []byte{'&'} + if emitter.anchor_data.alias { + c[0] = '*' + } + if !yaml_emitter_write_indicator(emitter, c, true, false, false) { + return false + } + return yaml_emitter_write_anchor(emitter, emitter.anchor_data.anchor) +} + +// Write a tag. +func yaml_emitter_process_tag(emitter *yaml_emitter_t) bool { + if len(emitter.tag_data.handle) == 0 && len(emitter.tag_data.suffix) == 0 { + return true + } + if len(emitter.tag_data.handle) > 0 { + if !yaml_emitter_write_tag_handle(emitter, emitter.tag_data.handle) { + return false + } + if len(emitter.tag_data.suffix) > 0 { + if !yaml_emitter_write_tag_content(emitter, emitter.tag_data.suffix, false) { + return false + } + } + } else { + // [Go] Allocate these slices elsewhere. + if !yaml_emitter_write_indicator(emitter, []byte("!<"), true, false, false) { + return false + } + if !yaml_emitter_write_tag_content(emitter, emitter.tag_data.suffix, false) { + return false + } + if !yaml_emitter_write_indicator(emitter, []byte{'>'}, false, false, false) { + return false + } + } + return true +} + +// Write a scalar. +func yaml_emitter_process_scalar(emitter *yaml_emitter_t) bool { + switch emitter.scalar_data.style { + case yaml_PLAIN_SCALAR_STYLE: + return yaml_emitter_write_plain_scalar(emitter, emitter.scalar_data.value, !emitter.simple_key_context) + + case yaml_SINGLE_QUOTED_SCALAR_STYLE: + return yaml_emitter_write_single_quoted_scalar(emitter, emitter.scalar_data.value, !emitter.simple_key_context) + + case yaml_DOUBLE_QUOTED_SCALAR_STYLE: + return yaml_emitter_write_double_quoted_scalar(emitter, emitter.scalar_data.value, !emitter.simple_key_context) + + case yaml_LITERAL_SCALAR_STYLE: + return yaml_emitter_write_literal_scalar(emitter, emitter.scalar_data.value) + + case yaml_FOLDED_SCALAR_STYLE: + return yaml_emitter_write_folded_scalar(emitter, emitter.scalar_data.value) + } + panic("unknown scalar style") +} + +// Check if a %YAML directive is valid. +func yaml_emitter_analyze_version_directive(emitter *yaml_emitter_t, version_directive *yaml_version_directive_t) bool { + if version_directive.major != 1 || version_directive.minor != 1 { + return yaml_emitter_set_emitter_error(emitter, "incompatible %YAML directive") + } + return true +} + +// Check if a %TAG directive is valid. +func yaml_emitter_analyze_tag_directive(emitter *yaml_emitter_t, tag_directive *yaml_tag_directive_t) bool { + handle := tag_directive.handle + prefix := tag_directive.prefix + if len(handle) == 0 { + return yaml_emitter_set_emitter_error(emitter, "tag handle must not be empty") + } + if handle[0] != '!' { + return yaml_emitter_set_emitter_error(emitter, "tag handle must start with '!'") + } + if handle[len(handle)-1] != '!' { + return yaml_emitter_set_emitter_error(emitter, "tag handle must end with '!'") + } + for i := 1; i < len(handle)-1; i += width(handle[i]) { + if !is_alpha(handle, i) { + return yaml_emitter_set_emitter_error(emitter, "tag handle must contain alphanumerical characters only") + } + } + if len(prefix) == 0 { + return yaml_emitter_set_emitter_error(emitter, "tag prefix must not be empty") + } + return true +} + +// Check if an anchor is valid. +func yaml_emitter_analyze_anchor(emitter *yaml_emitter_t, anchor []byte, alias bool) bool { + if len(anchor) == 0 { + problem := "anchor value must not be empty" + if alias { + problem = "alias value must not be empty" + } + return yaml_emitter_set_emitter_error(emitter, problem) + } + for i := 0; i < len(anchor); i += width(anchor[i]) { + if !is_alpha(anchor, i) { + problem := "anchor value must contain alphanumerical characters only" + if alias { + problem = "alias value must contain alphanumerical characters only" + } + return yaml_emitter_set_emitter_error(emitter, problem) + } + } + emitter.anchor_data.anchor = anchor + emitter.anchor_data.alias = alias + return true +} + +// Check if a tag is valid. +func yaml_emitter_analyze_tag(emitter *yaml_emitter_t, tag []byte) bool { + if len(tag) == 0 { + return yaml_emitter_set_emitter_error(emitter, "tag value must not be empty") + } + for i := 0; i < len(emitter.tag_directives); i++ { + tag_directive := &emitter.tag_directives[i] + if bytes.HasPrefix(tag, tag_directive.prefix) { + emitter.tag_data.handle = tag_directive.handle + emitter.tag_data.suffix = tag[len(tag_directive.prefix):] + return true + } + } + emitter.tag_data.suffix = tag + return true +} + +// Check if a scalar is valid. +func yaml_emitter_analyze_scalar(emitter *yaml_emitter_t, value []byte) bool { + var ( + block_indicators = false + flow_indicators = false + line_breaks = false + special_characters = false + + leading_space = false + leading_break = false + trailing_space = false + trailing_break = false + break_space = false + space_break = false + + preceded_by_whitespace = false + followed_by_whitespace = false + previous_space = false + previous_break = false + ) + + emitter.scalar_data.value = value + + if len(value) == 0 { + emitter.scalar_data.multiline = false + emitter.scalar_data.flow_plain_allowed = false + emitter.scalar_data.block_plain_allowed = true + emitter.scalar_data.single_quoted_allowed = true + emitter.scalar_data.block_allowed = false + return true + } + + if len(value) >= 3 && ((value[0] == '-' && value[1] == '-' && value[2] == '-') || (value[0] == '.' && value[1] == '.' && value[2] == '.')) { + block_indicators = true + flow_indicators = true + } + + preceded_by_whitespace = true + for i, w := 0, 0; i < len(value); i += w { + w = width(value[i]) + followed_by_whitespace = i+w >= len(value) || is_blank(value, i+w) + + if i == 0 { + switch value[i] { + case '#', ',', '[', ']', '{', '}', '&', '*', '!', '|', '>', '\'', '"', '%', '@', '`': + flow_indicators = true + block_indicators = true + case '?', ':': + flow_indicators = true + if followed_by_whitespace { + block_indicators = true + } + case '-': + if followed_by_whitespace { + flow_indicators = true + block_indicators = true + } + } + } else { + switch value[i] { + case ',', '?', '[', ']', '{', '}': + flow_indicators = true + case ':': + flow_indicators = true + if followed_by_whitespace { + block_indicators = true + } + case '#': + if preceded_by_whitespace { + flow_indicators = true + block_indicators = true + } + } + } + + if !is_printable(value, i) || !is_ascii(value, i) && !emitter.unicode { + special_characters = true + } + if is_space(value, i) { + if i == 0 { + leading_space = true + } + if i+width(value[i]) == len(value) { + trailing_space = true + } + if previous_break { + break_space = true + } + previous_space = true + previous_break = false + } else if is_break(value, i) { + line_breaks = true + if i == 0 { + leading_break = true + } + if i+width(value[i]) == len(value) { + trailing_break = true + } + if previous_space { + space_break = true + } + previous_space = false + previous_break = true + } else { + previous_space = false + previous_break = false + } + + // [Go]: Why 'z'? Couldn't be the end of the string as that's the loop condition. + preceded_by_whitespace = is_blankz(value, i) + } + + emitter.scalar_data.multiline = line_breaks + emitter.scalar_data.flow_plain_allowed = true + emitter.scalar_data.block_plain_allowed = true + emitter.scalar_data.single_quoted_allowed = true + emitter.scalar_data.block_allowed = true + + if leading_space || leading_break || trailing_space || trailing_break { + emitter.scalar_data.flow_plain_allowed = false + emitter.scalar_data.block_plain_allowed = false + } + if trailing_space { + emitter.scalar_data.block_allowed = false + } + if break_space { + emitter.scalar_data.flow_plain_allowed = false + emitter.scalar_data.block_plain_allowed = false + emitter.scalar_data.single_quoted_allowed = false + } + if space_break || special_characters { + emitter.scalar_data.flow_plain_allowed = false + emitter.scalar_data.block_plain_allowed = false + emitter.scalar_data.single_quoted_allowed = false + emitter.scalar_data.block_allowed = false + } + if line_breaks { + emitter.scalar_data.flow_plain_allowed = false + emitter.scalar_data.block_plain_allowed = false + } + if flow_indicators { + emitter.scalar_data.flow_plain_allowed = false + } + if block_indicators { + emitter.scalar_data.block_plain_allowed = false + } + return true +} + +// Check if the event data is valid. +func yaml_emitter_analyze_event(emitter *yaml_emitter_t, event *yaml_event_t) bool { + + emitter.anchor_data.anchor = nil + emitter.tag_data.handle = nil + emitter.tag_data.suffix = nil + emitter.scalar_data.value = nil + + switch event.typ { + case yaml_ALIAS_EVENT: + if !yaml_emitter_analyze_anchor(emitter, event.anchor, true) { + return false + } + + case yaml_SCALAR_EVENT: + if len(event.anchor) > 0 { + if !yaml_emitter_analyze_anchor(emitter, event.anchor, false) { + return false + } + } + if len(event.tag) > 0 && (emitter.canonical || (!event.implicit && !event.quoted_implicit)) { + if !yaml_emitter_analyze_tag(emitter, event.tag) { + return false + } + } + if !yaml_emitter_analyze_scalar(emitter, event.value) { + return false + } + + case yaml_SEQUENCE_START_EVENT: + if len(event.anchor) > 0 { + if !yaml_emitter_analyze_anchor(emitter, event.anchor, false) { + return false + } + } + if len(event.tag) > 0 && (emitter.canonical || !event.implicit) { + if !yaml_emitter_analyze_tag(emitter, event.tag) { + return false + } + } + + case yaml_MAPPING_START_EVENT: + if len(event.anchor) > 0 { + if !yaml_emitter_analyze_anchor(emitter, event.anchor, false) { + return false + } + } + if len(event.tag) > 0 && (emitter.canonical || !event.implicit) { + if !yaml_emitter_analyze_tag(emitter, event.tag) { + return false + } + } + } + return true +} + +// Write the BOM character. +func yaml_emitter_write_bom(emitter *yaml_emitter_t) bool { + if !flush(emitter) { + return false + } + pos := emitter.buffer_pos + emitter.buffer[pos+0] = '\xEF' + emitter.buffer[pos+1] = '\xBB' + emitter.buffer[pos+2] = '\xBF' + emitter.buffer_pos += 3 + return true +} + +func yaml_emitter_write_indent(emitter *yaml_emitter_t) bool { + indent := emitter.indent + if indent < 0 { + indent = 0 + } + if !emitter.indention || emitter.column > indent || (emitter.column == indent && !emitter.whitespace) { + if !put_break(emitter) { + return false + } + } + for emitter.column < indent { + if !put(emitter, ' ') { + return false + } + } + emitter.whitespace = true + emitter.indention = true + return true +} + +func yaml_emitter_write_indicator(emitter *yaml_emitter_t, indicator []byte, need_whitespace, is_whitespace, is_indention bool) bool { + if need_whitespace && !emitter.whitespace { + if !put(emitter, ' ') { + return false + } + } + if !write_all(emitter, indicator) { + return false + } + emitter.whitespace = is_whitespace + emitter.indention = (emitter.indention && is_indention) + emitter.open_ended = false + return true +} + +func yaml_emitter_write_anchor(emitter *yaml_emitter_t, value []byte) bool { + if !write_all(emitter, value) { + return false + } + emitter.whitespace = false + emitter.indention = false + return true +} + +func yaml_emitter_write_tag_handle(emitter *yaml_emitter_t, value []byte) bool { + if !emitter.whitespace { + if !put(emitter, ' ') { + return false + } + } + if !write_all(emitter, value) { + return false + } + emitter.whitespace = false + emitter.indention = false + return true +} + +func yaml_emitter_write_tag_content(emitter *yaml_emitter_t, value []byte, need_whitespace bool) bool { + if need_whitespace && !emitter.whitespace { + if !put(emitter, ' ') { + return false + } + } + for i := 0; i < len(value); { + var must_write bool + switch value[i] { + case ';', '/', '?', ':', '@', '&', '=', '+', '$', ',', '_', '.', '~', '*', '\'', '(', ')', '[', ']': + must_write = true + default: + must_write = is_alpha(value, i) + } + if must_write { + if !write(emitter, value, &i) { + return false + } + } else { + w := width(value[i]) + for k := 0; k < w; k++ { + octet := value[i] + i++ + if !put(emitter, '%') { + return false + } + + c := octet >> 4 + if c < 10 { + c += '0' + } else { + c += 'A' - 10 + } + if !put(emitter, c) { + return false + } + + c = octet & 0x0f + if c < 10 { + c += '0' + } else { + c += 'A' - 10 + } + if !put(emitter, c) { + return false + } + } + } + } + emitter.whitespace = false + emitter.indention = false + return true +} + +func yaml_emitter_write_plain_scalar(emitter *yaml_emitter_t, value []byte, allow_breaks bool) bool { + if !emitter.whitespace { + if !put(emitter, ' ') { + return false + } + } + + spaces := false + breaks := false + for i := 0; i < len(value); { + if is_space(value, i) { + if allow_breaks && !spaces && emitter.column > emitter.best_width && !is_space(value, i+1) { + if !yaml_emitter_write_indent(emitter) { + return false + } + i += width(value[i]) + } else { + if !write(emitter, value, &i) { + return false + } + } + spaces = true + } else if is_break(value, i) { + if !breaks && value[i] == '\n' { + if !put_break(emitter) { + return false + } + } + if !write_break(emitter, value, &i) { + return false + } + emitter.indention = true + breaks = true + } else { + if breaks { + if !yaml_emitter_write_indent(emitter) { + return false + } + } + if !write(emitter, value, &i) { + return false + } + emitter.indention = false + spaces = false + breaks = false + } + } + + emitter.whitespace = false + emitter.indention = false + if emitter.root_context { + emitter.open_ended = true + } + + return true +} + +func yaml_emitter_write_single_quoted_scalar(emitter *yaml_emitter_t, value []byte, allow_breaks bool) bool { + + if !yaml_emitter_write_indicator(emitter, []byte{'\''}, true, false, false) { + return false + } + + spaces := false + breaks := false + for i := 0; i < len(value); { + if is_space(value, i) { + if allow_breaks && !spaces && emitter.column > emitter.best_width && i > 0 && i < len(value)-1 && !is_space(value, i+1) { + if !yaml_emitter_write_indent(emitter) { + return false + } + i += width(value[i]) + } else { + if !write(emitter, value, &i) { + return false + } + } + spaces = true + } else if is_break(value, i) { + if !breaks && value[i] == '\n' { + if !put_break(emitter) { + return false + } + } + if !write_break(emitter, value, &i) { + return false + } + emitter.indention = true + breaks = true + } else { + if breaks { + if !yaml_emitter_write_indent(emitter) { + return false + } + } + if value[i] == '\'' { + if !put(emitter, '\'') { + return false + } + } + if !write(emitter, value, &i) { + return false + } + emitter.indention = false + spaces = false + breaks = false + } + } + if !yaml_emitter_write_indicator(emitter, []byte{'\''}, false, false, false) { + return false + } + emitter.whitespace = false + emitter.indention = false + return true +} + +func yaml_emitter_write_double_quoted_scalar(emitter *yaml_emitter_t, value []byte, allow_breaks bool) bool { + spaces := false + if !yaml_emitter_write_indicator(emitter, []byte{'"'}, true, false, false) { + return false + } + + for i := 0; i < len(value); { + if !is_printable(value, i) || (!emitter.unicode && !is_ascii(value, i)) || + is_bom(value, i) || is_break(value, i) || + value[i] == '"' || value[i] == '\\' { + + octet := value[i] + + var w int + var v rune + switch { + case octet&0x80 == 0x00: + w, v = 1, rune(octet&0x7F) + case octet&0xE0 == 0xC0: + w, v = 2, rune(octet&0x1F) + case octet&0xF0 == 0xE0: + w, v = 3, rune(octet&0x0F) + case octet&0xF8 == 0xF0: + w, v = 4, rune(octet&0x07) + } + for k := 1; k < w; k++ { + octet = value[i+k] + v = (v << 6) + (rune(octet) & 0x3F) + } + i += w + + if !put(emitter, '\\') { + return false + } + + var ok bool + switch v { + case 0x00: + ok = put(emitter, '0') + case 0x07: + ok = put(emitter, 'a') + case 0x08: + ok = put(emitter, 'b') + case 0x09: + ok = put(emitter, 't') + case 0x0A: + ok = put(emitter, 'n') + case 0x0b: + ok = put(emitter, 'v') + case 0x0c: + ok = put(emitter, 'f') + case 0x0d: + ok = put(emitter, 'r') + case 0x1b: + ok = put(emitter, 'e') + case 0x22: + ok = put(emitter, '"') + case 0x5c: + ok = put(emitter, '\\') + case 0x85: + ok = put(emitter, 'N') + case 0xA0: + ok = put(emitter, '_') + case 0x2028: + ok = put(emitter, 'L') + case 0x2029: + ok = put(emitter, 'P') + default: + if v <= 0xFF { + ok = put(emitter, 'x') + w = 2 + } else if v <= 0xFFFF { + ok = put(emitter, 'u') + w = 4 + } else { + ok = put(emitter, 'U') + w = 8 + } + for k := (w - 1) * 4; ok && k >= 0; k -= 4 { + digit := byte((v >> uint(k)) & 0x0F) + if digit < 10 { + ok = put(emitter, digit+'0') + } else { + ok = put(emitter, digit+'A'-10) + } + } + } + if !ok { + return false + } + spaces = false + } else if is_space(value, i) { + if allow_breaks && !spaces && emitter.column > emitter.best_width && i > 0 && i < len(value)-1 { + if !yaml_emitter_write_indent(emitter) { + return false + } + if is_space(value, i+1) { + if !put(emitter, '\\') { + return false + } + } + i += width(value[i]) + } else if !write(emitter, value, &i) { + return false + } + spaces = true + } else { + if !write(emitter, value, &i) { + return false + } + spaces = false + } + } + if !yaml_emitter_write_indicator(emitter, []byte{'"'}, false, false, false) { + return false + } + emitter.whitespace = false + emitter.indention = false + return true +} + +func yaml_emitter_write_block_scalar_hints(emitter *yaml_emitter_t, value []byte) bool { + if is_space(value, 0) || is_break(value, 0) { + indent_hint := []byte{'0' + byte(emitter.best_indent)} + if !yaml_emitter_write_indicator(emitter, indent_hint, false, false, false) { + return false + } + } + + emitter.open_ended = false + + var chomp_hint [1]byte + if len(value) == 0 { + chomp_hint[0] = '-' + } else { + i := len(value) - 1 + for value[i]&0xC0 == 0x80 { + i-- + } + if !is_break(value, i) { + chomp_hint[0] = '-' + } else if i == 0 { + chomp_hint[0] = '+' + emitter.open_ended = true + } else { + i-- + for value[i]&0xC0 == 0x80 { + i-- + } + if is_break(value, i) { + chomp_hint[0] = '+' + emitter.open_ended = true + } + } + } + if chomp_hint[0] != 0 { + if !yaml_emitter_write_indicator(emitter, chomp_hint[:], false, false, false) { + return false + } + } + return true +} + +func yaml_emitter_write_literal_scalar(emitter *yaml_emitter_t, value []byte) bool { + if !yaml_emitter_write_indicator(emitter, []byte{'|'}, true, false, false) { + return false + } + if !yaml_emitter_write_block_scalar_hints(emitter, value) { + return false + } + if !put_break(emitter) { + return false + } + emitter.indention = true + emitter.whitespace = true + breaks := true + for i := 0; i < len(value); { + if is_break(value, i) { + if !write_break(emitter, value, &i) { + return false + } + emitter.indention = true + breaks = true + } else { + if breaks { + if !yaml_emitter_write_indent(emitter) { + return false + } + } + if !write(emitter, value, &i) { + return false + } + emitter.indention = false + breaks = false + } + } + + return true +} + +func yaml_emitter_write_folded_scalar(emitter *yaml_emitter_t, value []byte) bool { + if !yaml_emitter_write_indicator(emitter, []byte{'>'}, true, false, false) { + return false + } + if !yaml_emitter_write_block_scalar_hints(emitter, value) { + return false + } + + if !put_break(emitter) { + return false + } + emitter.indention = true + emitter.whitespace = true + + breaks := true + leading_spaces := true + for i := 0; i < len(value); { + if is_break(value, i) { + if !breaks && !leading_spaces && value[i] == '\n' { + k := 0 + for is_break(value, k) { + k += width(value[k]) + } + if !is_blankz(value, k) { + if !put_break(emitter) { + return false + } + } + } + if !write_break(emitter, value, &i) { + return false + } + emitter.indention = true + breaks = true + } else { + if breaks { + if !yaml_emitter_write_indent(emitter) { + return false + } + leading_spaces = is_blank(value, i) + } + if !breaks && is_space(value, i) && !is_space(value, i+1) && emitter.column > emitter.best_width { + if !yaml_emitter_write_indent(emitter) { + return false + } + i += width(value[i]) + } else { + if !write(emitter, value, &i) { + return false + } + } + emitter.indention = false + breaks = false + } + } + return true +} diff --git a/generator/vendor/gopkg.in/yaml.v2/encode.go b/generator/vendor/gopkg.in/yaml.v2/encode.go new file mode 100644 index 00000000..84f84995 --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/encode.go @@ -0,0 +1,306 @@ +package yaml + +import ( + "encoding" + "fmt" + "reflect" + "regexp" + "sort" + "strconv" + "strings" + "time" +) + +type encoder struct { + emitter yaml_emitter_t + event yaml_event_t + out []byte + flow bool +} + +func newEncoder() (e *encoder) { + e = &encoder{} + e.must(yaml_emitter_initialize(&e.emitter)) + yaml_emitter_set_output_string(&e.emitter, &e.out) + yaml_emitter_set_unicode(&e.emitter, true) + e.must(yaml_stream_start_event_initialize(&e.event, yaml_UTF8_ENCODING)) + e.emit() + e.must(yaml_document_start_event_initialize(&e.event, nil, nil, true)) + e.emit() + return e +} + +func (e *encoder) finish() { + e.must(yaml_document_end_event_initialize(&e.event, true)) + e.emit() + e.emitter.open_ended = false + e.must(yaml_stream_end_event_initialize(&e.event)) + e.emit() +} + +func (e *encoder) destroy() { + yaml_emitter_delete(&e.emitter) +} + +func (e *encoder) emit() { + // This will internally delete the e.event value. + if !yaml_emitter_emit(&e.emitter, &e.event) && e.event.typ != yaml_DOCUMENT_END_EVENT && e.event.typ != yaml_STREAM_END_EVENT { + e.must(false) + } +} + +func (e *encoder) must(ok bool) { + if !ok { + msg := e.emitter.problem + if msg == "" { + msg = "unknown problem generating YAML content" + } + failf("%s", msg) + } +} + +func (e *encoder) marshal(tag string, in reflect.Value) { + if !in.IsValid() { + e.nilv() + return + } + iface := in.Interface() + if m, ok := iface.(Marshaler); ok { + v, err := m.MarshalYAML() + if err != nil { + fail(err) + } + if v == nil { + e.nilv() + return + } + in = reflect.ValueOf(v) + } else if m, ok := iface.(encoding.TextMarshaler); ok { + text, err := m.MarshalText() + if err != nil { + fail(err) + } + in = reflect.ValueOf(string(text)) + } + switch in.Kind() { + case reflect.Interface: + if in.IsNil() { + e.nilv() + } else { + e.marshal(tag, in.Elem()) + } + case reflect.Map: + e.mapv(tag, in) + case reflect.Ptr: + if in.IsNil() { + e.nilv() + } else { + e.marshal(tag, in.Elem()) + } + case reflect.Struct: + e.structv(tag, in) + case reflect.Slice: + if in.Type().Elem() == mapItemType { + e.itemsv(tag, in) + } else { + e.slicev(tag, in) + } + case reflect.String: + e.stringv(tag, in) + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + if in.Type() == durationType { + e.stringv(tag, reflect.ValueOf(iface.(time.Duration).String())) + } else { + e.intv(tag, in) + } + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: + e.uintv(tag, in) + case reflect.Float32, reflect.Float64: + e.floatv(tag, in) + case reflect.Bool: + e.boolv(tag, in) + default: + panic("cannot marshal type: " + in.Type().String()) + } +} + +func (e *encoder) mapv(tag string, in reflect.Value) { + e.mappingv(tag, func() { + keys := keyList(in.MapKeys()) + sort.Sort(keys) + for _, k := range keys { + e.marshal("", k) + e.marshal("", in.MapIndex(k)) + } + }) +} + +func (e *encoder) itemsv(tag string, in reflect.Value) { + e.mappingv(tag, func() { + slice := in.Convert(reflect.TypeOf([]MapItem{})).Interface().([]MapItem) + for _, item := range slice { + e.marshal("", reflect.ValueOf(item.Key)) + e.marshal("", reflect.ValueOf(item.Value)) + } + }) +} + +func (e *encoder) structv(tag string, in reflect.Value) { + sinfo, err := getStructInfo(in.Type()) + if err != nil { + panic(err) + } + e.mappingv(tag, func() { + for _, info := range sinfo.FieldsList { + var value reflect.Value + if info.Inline == nil { + value = in.Field(info.Num) + } else { + value = in.FieldByIndex(info.Inline) + } + if info.OmitEmpty && isZero(value) { + continue + } + e.marshal("", reflect.ValueOf(info.Key)) + e.flow = info.Flow + e.marshal("", value) + } + if sinfo.InlineMap >= 0 { + m := in.Field(sinfo.InlineMap) + if m.Len() > 0 { + e.flow = false + keys := keyList(m.MapKeys()) + sort.Sort(keys) + for _, k := range keys { + if _, found := sinfo.FieldsMap[k.String()]; found { + panic(fmt.Sprintf("Can't have key %q in inlined map; conflicts with struct field", k.String())) + } + e.marshal("", k) + e.flow = false + e.marshal("", m.MapIndex(k)) + } + } + } + }) +} + +func (e *encoder) mappingv(tag string, f func()) { + implicit := tag == "" + style := yaml_BLOCK_MAPPING_STYLE + if e.flow { + e.flow = false + style = yaml_FLOW_MAPPING_STYLE + } + e.must(yaml_mapping_start_event_initialize(&e.event, nil, []byte(tag), implicit, style)) + e.emit() + f() + e.must(yaml_mapping_end_event_initialize(&e.event)) + e.emit() +} + +func (e *encoder) slicev(tag string, in reflect.Value) { + implicit := tag == "" + style := yaml_BLOCK_SEQUENCE_STYLE + if e.flow { + e.flow = false + style = yaml_FLOW_SEQUENCE_STYLE + } + e.must(yaml_sequence_start_event_initialize(&e.event, nil, []byte(tag), implicit, style)) + e.emit() + n := in.Len() + for i := 0; i < n; i++ { + e.marshal("", in.Index(i)) + } + e.must(yaml_sequence_end_event_initialize(&e.event)) + e.emit() +} + +// isBase60 returns whether s is in base 60 notation as defined in YAML 1.1. +// +// The base 60 float notation in YAML 1.1 is a terrible idea and is unsupported +// in YAML 1.2 and by this package, but these should be marshalled quoted for +// the time being for compatibility with other parsers. +func isBase60Float(s string) (result bool) { + // Fast path. + if s == "" { + return false + } + c := s[0] + if !(c == '+' || c == '-' || c >= '0' && c <= '9') || strings.IndexByte(s, ':') < 0 { + return false + } + // Do the full match. + return base60float.MatchString(s) +} + +// From http://yaml.org/type/float.html, except the regular expression there +// is bogus. In practice parsers do not enforce the "\.[0-9_]*" suffix. +var base60float = regexp.MustCompile(`^[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+(?:\.[0-9_]*)?$`) + +func (e *encoder) stringv(tag string, in reflect.Value) { + var style yaml_scalar_style_t + s := in.String() + rtag, rs := resolve("", s) + if rtag == yaml_BINARY_TAG { + if tag == "" || tag == yaml_STR_TAG { + tag = rtag + s = rs.(string) + } else if tag == yaml_BINARY_TAG { + failf("explicitly tagged !!binary data must be base64-encoded") + } else { + failf("cannot marshal invalid UTF-8 data as %s", shortTag(tag)) + } + } + if tag == "" && (rtag != yaml_STR_TAG || isBase60Float(s)) { + style = yaml_DOUBLE_QUOTED_SCALAR_STYLE + } else if strings.Contains(s, "\n") { + style = yaml_LITERAL_SCALAR_STYLE + } else { + style = yaml_PLAIN_SCALAR_STYLE + } + e.emitScalar(s, "", tag, style) +} + +func (e *encoder) boolv(tag string, in reflect.Value) { + var s string + if in.Bool() { + s = "true" + } else { + s = "false" + } + e.emitScalar(s, "", tag, yaml_PLAIN_SCALAR_STYLE) +} + +func (e *encoder) intv(tag string, in reflect.Value) { + s := strconv.FormatInt(in.Int(), 10) + e.emitScalar(s, "", tag, yaml_PLAIN_SCALAR_STYLE) +} + +func (e *encoder) uintv(tag string, in reflect.Value) { + s := strconv.FormatUint(in.Uint(), 10) + e.emitScalar(s, "", tag, yaml_PLAIN_SCALAR_STYLE) +} + +func (e *encoder) floatv(tag string, in reflect.Value) { + // FIXME: Handle 64 bits here. + s := strconv.FormatFloat(float64(in.Float()), 'g', -1, 32) + switch s { + case "+Inf": + s = ".inf" + case "-Inf": + s = "-.inf" + case "NaN": + s = ".nan" + } + e.emitScalar(s, "", tag, yaml_PLAIN_SCALAR_STYLE) +} + +func (e *encoder) nilv() { + e.emitScalar("null", "", "", yaml_PLAIN_SCALAR_STYLE) +} + +func (e *encoder) emitScalar(value, anchor, tag string, style yaml_scalar_style_t) { + implicit := tag == "" + e.must(yaml_scalar_event_initialize(&e.event, []byte(anchor), []byte(tag), []byte(value), implicit, implicit, style)) + e.emit() +} diff --git a/generator/vendor/gopkg.in/yaml.v2/encode_test.go b/generator/vendor/gopkg.in/yaml.v2/encode_test.go new file mode 100644 index 00000000..84099bd3 --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/encode_test.go @@ -0,0 +1,501 @@ +package yaml_test + +import ( + "fmt" + "math" + "strconv" + "strings" + "time" + + . "gopkg.in/check.v1" + "gopkg.in/yaml.v2" + "net" + "os" +) + +var marshalIntTest = 123 + +var marshalTests = []struct { + value interface{} + data string +}{ + { + nil, + "null\n", + }, { + &struct{}{}, + "{}\n", + }, { + map[string]string{"v": "hi"}, + "v: hi\n", + }, { + map[string]interface{}{"v": "hi"}, + "v: hi\n", + }, { + map[string]string{"v": "true"}, + "v: \"true\"\n", + }, { + map[string]string{"v": "false"}, + "v: \"false\"\n", + }, { + map[string]interface{}{"v": true}, + "v: true\n", + }, { + map[string]interface{}{"v": false}, + "v: false\n", + }, { + map[string]interface{}{"v": 10}, + "v: 10\n", + }, { + map[string]interface{}{"v": -10}, + "v: -10\n", + }, { + map[string]uint{"v": 42}, + "v: 42\n", + }, { + map[string]interface{}{"v": int64(4294967296)}, + "v: 4294967296\n", + }, { + map[string]int64{"v": int64(4294967296)}, + "v: 4294967296\n", + }, { + map[string]uint64{"v": 4294967296}, + "v: 4294967296\n", + }, { + map[string]interface{}{"v": "10"}, + "v: \"10\"\n", + }, { + map[string]interface{}{"v": 0.1}, + "v: 0.1\n", + }, { + map[string]interface{}{"v": float64(0.1)}, + "v: 0.1\n", + }, { + map[string]interface{}{"v": -0.1}, + "v: -0.1\n", + }, { + map[string]interface{}{"v": math.Inf(+1)}, + "v: .inf\n", + }, { + map[string]interface{}{"v": math.Inf(-1)}, + "v: -.inf\n", + }, { + map[string]interface{}{"v": math.NaN()}, + "v: .nan\n", + }, { + map[string]interface{}{"v": nil}, + "v: null\n", + }, { + map[string]interface{}{"v": ""}, + "v: \"\"\n", + }, { + map[string][]string{"v": []string{"A", "B"}}, + "v:\n- A\n- B\n", + }, { + map[string][]string{"v": []string{"A", "B\nC"}}, + "v:\n- A\n- |-\n B\n C\n", + }, { + map[string][]interface{}{"v": []interface{}{"A", 1, map[string][]int{"B": []int{2, 3}}}}, + "v:\n- A\n- 1\n- B:\n - 2\n - 3\n", + }, { + map[string]interface{}{"a": map[interface{}]interface{}{"b": "c"}}, + "a:\n b: c\n", + }, { + map[string]interface{}{"a": "-"}, + "a: '-'\n", + }, + + // Simple values. + { + &marshalIntTest, + "123\n", + }, + + // Structures + { + &struct{ Hello string }{"world"}, + "hello: world\n", + }, { + &struct { + A struct { + B string + } + }{struct{ B string }{"c"}}, + "a:\n b: c\n", + }, { + &struct { + A *struct { + B string + } + }{&struct{ B string }{"c"}}, + "a:\n b: c\n", + }, { + &struct { + A *struct { + B string + } + }{}, + "a: null\n", + }, { + &struct{ A int }{1}, + "a: 1\n", + }, { + &struct{ A []int }{[]int{1, 2}}, + "a:\n- 1\n- 2\n", + }, { + &struct { + B int "a" + }{1}, + "a: 1\n", + }, { + &struct{ A bool }{true}, + "a: true\n", + }, + + // Conditional flag + { + &struct { + A int "a,omitempty" + B int "b,omitempty" + }{1, 0}, + "a: 1\n", + }, { + &struct { + A int "a,omitempty" + B int "b,omitempty" + }{0, 0}, + "{}\n", + }, { + &struct { + A *struct{ X, y int } "a,omitempty,flow" + }{&struct{ X, y int }{1, 2}}, + "a: {x: 1}\n", + }, { + &struct { + A *struct{ X, y int } "a,omitempty,flow" + }{nil}, + "{}\n", + }, { + &struct { + A *struct{ X, y int } "a,omitempty,flow" + }{&struct{ X, y int }{}}, + "a: {x: 0}\n", + }, { + &struct { + A struct{ X, y int } "a,omitempty,flow" + }{struct{ X, y int }{1, 2}}, + "a: {x: 1}\n", + }, { + &struct { + A struct{ X, y int } "a,omitempty,flow" + }{struct{ X, y int }{0, 1}}, + "{}\n", + }, { + &struct { + A float64 "a,omitempty" + B float64 "b,omitempty" + }{1, 0}, + "a: 1\n", + }, + + // Flow flag + { + &struct { + A []int "a,flow" + }{[]int{1, 2}}, + "a: [1, 2]\n", + }, { + &struct { + A map[string]string "a,flow" + }{map[string]string{"b": "c", "d": "e"}}, + "a: {b: c, d: e}\n", + }, { + &struct { + A struct { + B, D string + } "a,flow" + }{struct{ B, D string }{"c", "e"}}, + "a: {b: c, d: e}\n", + }, + + // Unexported field + { + &struct { + u int + A int + }{0, 1}, + "a: 1\n", + }, + + // Ignored field + { + &struct { + A int + B int "-" + }{1, 2}, + "a: 1\n", + }, + + // Struct inlining + { + &struct { + A int + C inlineB `yaml:",inline"` + }{1, inlineB{2, inlineC{3}}}, + "a: 1\nb: 2\nc: 3\n", + }, + + // Map inlining + { + &struct { + A int + C map[string]int `yaml:",inline"` + }{1, map[string]int{"b": 2, "c": 3}}, + "a: 1\nb: 2\nc: 3\n", + }, + + // Duration + { + map[string]time.Duration{"a": 3 * time.Second}, + "a: 3s\n", + }, + + // Issue #24: bug in map merging logic. + { + map[string]string{"a": "<foo>"}, + "a: <foo>\n", + }, + + // Issue #34: marshal unsupported base 60 floats quoted for compatibility + // with old YAML 1.1 parsers. + { + map[string]string{"a": "1:1"}, + "a: \"1:1\"\n", + }, + + // Binary data. + { + map[string]string{"a": "\x00"}, + "a: \"\\0\"\n", + }, { + map[string]string{"a": "\x80\x81\x82"}, + "a: !!binary gIGC\n", + }, { + map[string]string{"a": strings.Repeat("\x90", 54)}, + "a: !!binary |\n " + strings.Repeat("kJCQ", 17) + "kJ\n CQ\n", + }, + + // Ordered maps. + { + &yaml.MapSlice{{"b", 2}, {"a", 1}, {"d", 4}, {"c", 3}, {"sub", yaml.MapSlice{{"e", 5}}}}, + "b: 2\na: 1\nd: 4\nc: 3\nsub:\n e: 5\n", + }, + + // Encode unicode as utf-8 rather than in escaped form. + { + map[string]string{"a": "你好"}, + "a: 你好\n", + }, + + // Support encoding.TextMarshaler. + { + map[string]net.IP{"a": net.IPv4(1, 2, 3, 4)}, + "a: 1.2.3.4\n", + }, + { + map[string]time.Time{"a": time.Unix(1424801979, 0)}, + "a: 2015-02-24T18:19:39Z\n", + }, + + // Ensure strings containing ": " are quoted (reported as PR #43, but not reproducible). + { + map[string]string{"a": "b: c"}, + "a: 'b: c'\n", + }, + + // Containing hash mark ('#') in string should be quoted + { + map[string]string{"a": "Hello #comment"}, + "a: 'Hello #comment'\n", + }, + { + map[string]string{"a": "你好 #comment"}, + "a: '你好 #comment'\n", + }, +} + +func (s *S) TestMarshal(c *C) { + defer os.Setenv("TZ", os.Getenv("TZ")) + os.Setenv("TZ", "UTC") + for _, item := range marshalTests { + data, err := yaml.Marshal(item.value) + c.Assert(err, IsNil) + c.Assert(string(data), Equals, item.data) + } +} + +var marshalErrorTests = []struct { + value interface{} + error string + panic string +}{{ + value: &struct { + B int + inlineB ",inline" + }{1, inlineB{2, inlineC{3}}}, + panic: `Duplicated key 'b' in struct struct \{ B int; .*`, +}, { + value: &struct { + A int + B map[string]int ",inline" + }{1, map[string]int{"a": 2}}, + panic: `Can't have key "a" in inlined map; conflicts with struct field`, +}} + +func (s *S) TestMarshalErrors(c *C) { + for _, item := range marshalErrorTests { + if item.panic != "" { + c.Assert(func() { yaml.Marshal(item.value) }, PanicMatches, item.panic) + } else { + _, err := yaml.Marshal(item.value) + c.Assert(err, ErrorMatches, item.error) + } + } +} + +func (s *S) TestMarshalTypeCache(c *C) { + var data []byte + var err error + func() { + type T struct{ A int } + data, err = yaml.Marshal(&T{}) + c.Assert(err, IsNil) + }() + func() { + type T struct{ B int } + data, err = yaml.Marshal(&T{}) + c.Assert(err, IsNil) + }() + c.Assert(string(data), Equals, "b: 0\n") +} + +var marshalerTests = []struct { + data string + value interface{} +}{ + {"_:\n hi: there\n", map[interface{}]interface{}{"hi": "there"}}, + {"_:\n- 1\n- A\n", []interface{}{1, "A"}}, + {"_: 10\n", 10}, + {"_: null\n", nil}, + {"_: BAR!\n", "BAR!"}, +} + +type marshalerType struct { + value interface{} +} + +func (o marshalerType) MarshalText() ([]byte, error) { + panic("MarshalText called on type with MarshalYAML") +} + +func (o marshalerType) MarshalYAML() (interface{}, error) { + return o.value, nil +} + +type marshalerValue struct { + Field marshalerType "_" +} + +func (s *S) TestMarshaler(c *C) { + for _, item := range marshalerTests { + obj := &marshalerValue{} + obj.Field.value = item.value + data, err := yaml.Marshal(obj) + c.Assert(err, IsNil) + c.Assert(string(data), Equals, string(item.data)) + } +} + +func (s *S) TestMarshalerWholeDocument(c *C) { + obj := &marshalerType{} + obj.value = map[string]string{"hello": "world!"} + data, err := yaml.Marshal(obj) + c.Assert(err, IsNil) + c.Assert(string(data), Equals, "hello: world!\n") +} + +type failingMarshaler struct{} + +func (ft *failingMarshaler) MarshalYAML() (interface{}, error) { + return nil, failingErr +} + +func (s *S) TestMarshalerError(c *C) { + _, err := yaml.Marshal(&failingMarshaler{}) + c.Assert(err, Equals, failingErr) +} + +func (s *S) TestSortedOutput(c *C) { + order := []interface{}{ + false, + true, + 1, + uint(1), + 1.0, + 1.1, + 1.2, + 2, + uint(2), + 2.0, + 2.1, + "", + ".1", + ".2", + ".a", + "1", + "2", + "a!10", + "a/2", + "a/10", + "a~10", + "ab/1", + "b/1", + "b/01", + "b/2", + "b/02", + "b/3", + "b/03", + "b1", + "b01", + "b3", + "c2.10", + "c10.2", + "d1", + "d12", + "d12a", + } + m := make(map[interface{}]int) + for _, k := range order { + m[k] = 1 + } + data, err := yaml.Marshal(m) + c.Assert(err, IsNil) + out := "\n" + string(data) + last := 0 + for i, k := range order { + repr := fmt.Sprint(k) + if s, ok := k.(string); ok { + if _, err = strconv.ParseFloat(repr, 32); s == "" || err == nil { + repr = `"` + repr + `"` + } + } + index := strings.Index(out, "\n"+repr+":") + if index == -1 { + c.Fatalf("%#v is not in the output: %#v", k, out) + } + if index < last { + c.Fatalf("%#v was generated before %#v: %q", k, order[i-1], out) + } + last = index + } +} diff --git a/generator/vendor/gopkg.in/yaml.v2/example_embedded_test.go b/generator/vendor/gopkg.in/yaml.v2/example_embedded_test.go new file mode 100644 index 00000000..c8b241d5 --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/example_embedded_test.go @@ -0,0 +1,41 @@ +package yaml_test + +import ( + "fmt" + "log" + + "gopkg.in/yaml.v2" +) + +// An example showing how to unmarshal embedded +// structs from YAML. + +type StructA struct { + A string `yaml:"a"` +} + +type StructB struct { + // Embedded structs are not treated as embedded in YAML by default. To do that, + // add the ",inline" annotation below + StructA `yaml:",inline"` + B string `yaml:"b"` +} + +var data = ` +a: a string from struct A +b: a string from struct B +` + +func ExampleUnmarshal_embedded() { + var b StructB + + err := yaml.Unmarshal([]byte(data), &b) + if err != nil { + log.Fatal("cannot unmarshal data: %v", err) + } + fmt.Println(b.A) + fmt.Println(b.B) + // Output: + // a string from struct A + // a string from struct B +} diff --git a/generator/vendor/gopkg.in/yaml.v2/parserc.go b/generator/vendor/gopkg.in/yaml.v2/parserc.go new file mode 100644 index 00000000..81d05dfe --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/parserc.go @@ -0,0 +1,1095 @@ +package yaml + +import ( + "bytes" +) + +// The parser implements the following grammar: +// +// stream ::= STREAM-START implicit_document? explicit_document* STREAM-END +// implicit_document ::= block_node DOCUMENT-END* +// explicit_document ::= DIRECTIVE* DOCUMENT-START block_node? DOCUMENT-END* +// block_node_or_indentless_sequence ::= +// ALIAS +// | properties (block_content | indentless_block_sequence)? +// | block_content +// | indentless_block_sequence +// block_node ::= ALIAS +// | properties block_content? +// | block_content +// flow_node ::= ALIAS +// | properties flow_content? +// | flow_content +// properties ::= TAG ANCHOR? | ANCHOR TAG? +// block_content ::= block_collection | flow_collection | SCALAR +// flow_content ::= flow_collection | SCALAR +// block_collection ::= block_sequence | block_mapping +// flow_collection ::= flow_sequence | flow_mapping +// block_sequence ::= BLOCK-SEQUENCE-START (BLOCK-ENTRY block_node?)* BLOCK-END +// indentless_sequence ::= (BLOCK-ENTRY block_node?)+ +// block_mapping ::= BLOCK-MAPPING_START +// ((KEY block_node_or_indentless_sequence?)? +// (VALUE block_node_or_indentless_sequence?)?)* +// BLOCK-END +// flow_sequence ::= FLOW-SEQUENCE-START +// (flow_sequence_entry FLOW-ENTRY)* +// flow_sequence_entry? +// FLOW-SEQUENCE-END +// flow_sequence_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? +// flow_mapping ::= FLOW-MAPPING-START +// (flow_mapping_entry FLOW-ENTRY)* +// flow_mapping_entry? +// FLOW-MAPPING-END +// flow_mapping_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? + +// Peek the next token in the token queue. +func peek_token(parser *yaml_parser_t) *yaml_token_t { + if parser.token_available || yaml_parser_fetch_more_tokens(parser) { + return &parser.tokens[parser.tokens_head] + } + return nil +} + +// Remove the next token from the queue (must be called after peek_token). +func skip_token(parser *yaml_parser_t) { + parser.token_available = false + parser.tokens_parsed++ + parser.stream_end_produced = parser.tokens[parser.tokens_head].typ == yaml_STREAM_END_TOKEN + parser.tokens_head++ +} + +// Get the next event. +func yaml_parser_parse(parser *yaml_parser_t, event *yaml_event_t) bool { + // Erase the event object. + *event = yaml_event_t{} + + // No events after the end of the stream or error. + if parser.stream_end_produced || parser.error != yaml_NO_ERROR || parser.state == yaml_PARSE_END_STATE { + return true + } + + // Generate the next event. + return yaml_parser_state_machine(parser, event) +} + +// Set parser error. +func yaml_parser_set_parser_error(parser *yaml_parser_t, problem string, problem_mark yaml_mark_t) bool { + parser.error = yaml_PARSER_ERROR + parser.problem = problem + parser.problem_mark = problem_mark + return false +} + +func yaml_parser_set_parser_error_context(parser *yaml_parser_t, context string, context_mark yaml_mark_t, problem string, problem_mark yaml_mark_t) bool { + parser.error = yaml_PARSER_ERROR + parser.context = context + parser.context_mark = context_mark + parser.problem = problem + parser.problem_mark = problem_mark + return false +} + +// State dispatcher. +func yaml_parser_state_machine(parser *yaml_parser_t, event *yaml_event_t) bool { + //trace("yaml_parser_state_machine", "state:", parser.state.String()) + + switch parser.state { + case yaml_PARSE_STREAM_START_STATE: + return yaml_parser_parse_stream_start(parser, event) + + case yaml_PARSE_IMPLICIT_DOCUMENT_START_STATE: + return yaml_parser_parse_document_start(parser, event, true) + + case yaml_PARSE_DOCUMENT_START_STATE: + return yaml_parser_parse_document_start(parser, event, false) + + case yaml_PARSE_DOCUMENT_CONTENT_STATE: + return yaml_parser_parse_document_content(parser, event) + + case yaml_PARSE_DOCUMENT_END_STATE: + return yaml_parser_parse_document_end(parser, event) + + case yaml_PARSE_BLOCK_NODE_STATE: + return yaml_parser_parse_node(parser, event, true, false) + + case yaml_PARSE_BLOCK_NODE_OR_INDENTLESS_SEQUENCE_STATE: + return yaml_parser_parse_node(parser, event, true, true) + + case yaml_PARSE_FLOW_NODE_STATE: + return yaml_parser_parse_node(parser, event, false, false) + + case yaml_PARSE_BLOCK_SEQUENCE_FIRST_ENTRY_STATE: + return yaml_parser_parse_block_sequence_entry(parser, event, true) + + case yaml_PARSE_BLOCK_SEQUENCE_ENTRY_STATE: + return yaml_parser_parse_block_sequence_entry(parser, event, false) + + case yaml_PARSE_INDENTLESS_SEQUENCE_ENTRY_STATE: + return yaml_parser_parse_indentless_sequence_entry(parser, event) + + case yaml_PARSE_BLOCK_MAPPING_FIRST_KEY_STATE: + return yaml_parser_parse_block_mapping_key(parser, event, true) + + case yaml_PARSE_BLOCK_MAPPING_KEY_STATE: + return yaml_parser_parse_block_mapping_key(parser, event, false) + + case yaml_PARSE_BLOCK_MAPPING_VALUE_STATE: + return yaml_parser_parse_block_mapping_value(parser, event) + + case yaml_PARSE_FLOW_SEQUENCE_FIRST_ENTRY_STATE: + return yaml_parser_parse_flow_sequence_entry(parser, event, true) + + case yaml_PARSE_FLOW_SEQUENCE_ENTRY_STATE: + return yaml_parser_parse_flow_sequence_entry(parser, event, false) + + case yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_KEY_STATE: + return yaml_parser_parse_flow_sequence_entry_mapping_key(parser, event) + + case yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_VALUE_STATE: + return yaml_parser_parse_flow_sequence_entry_mapping_value(parser, event) + + case yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_END_STATE: + return yaml_parser_parse_flow_sequence_entry_mapping_end(parser, event) + + case yaml_PARSE_FLOW_MAPPING_FIRST_KEY_STATE: + return yaml_parser_parse_flow_mapping_key(parser, event, true) + + case yaml_PARSE_FLOW_MAPPING_KEY_STATE: + return yaml_parser_parse_flow_mapping_key(parser, event, false) + + case yaml_PARSE_FLOW_MAPPING_VALUE_STATE: + return yaml_parser_parse_flow_mapping_value(parser, event, false) + + case yaml_PARSE_FLOW_MAPPING_EMPTY_VALUE_STATE: + return yaml_parser_parse_flow_mapping_value(parser, event, true) + + default: + panic("invalid parser state") + } +} + +// Parse the production: +// stream ::= STREAM-START implicit_document? explicit_document* STREAM-END +// ************ +func yaml_parser_parse_stream_start(parser *yaml_parser_t, event *yaml_event_t) bool { + token := peek_token(parser) + if token == nil { + return false + } + if token.typ != yaml_STREAM_START_TOKEN { + return yaml_parser_set_parser_error(parser, "did not find expected <stream-start>", token.start_mark) + } + parser.state = yaml_PARSE_IMPLICIT_DOCUMENT_START_STATE + *event = yaml_event_t{ + typ: yaml_STREAM_START_EVENT, + start_mark: token.start_mark, + end_mark: token.end_mark, + encoding: token.encoding, + } + skip_token(parser) + return true +} + +// Parse the productions: +// implicit_document ::= block_node DOCUMENT-END* +// * +// explicit_document ::= DIRECTIVE* DOCUMENT-START block_node? DOCUMENT-END* +// ************************* +func yaml_parser_parse_document_start(parser *yaml_parser_t, event *yaml_event_t, implicit bool) bool { + + token := peek_token(parser) + if token == nil { + return false + } + + // Parse extra document end indicators. + if !implicit { + for token.typ == yaml_DOCUMENT_END_TOKEN { + skip_token(parser) + token = peek_token(parser) + if token == nil { + return false + } + } + } + + if implicit && token.typ != yaml_VERSION_DIRECTIVE_TOKEN && + token.typ != yaml_TAG_DIRECTIVE_TOKEN && + token.typ != yaml_DOCUMENT_START_TOKEN && + token.typ != yaml_STREAM_END_TOKEN { + // Parse an implicit document. + if !yaml_parser_process_directives(parser, nil, nil) { + return false + } + parser.states = append(parser.states, yaml_PARSE_DOCUMENT_END_STATE) + parser.state = yaml_PARSE_BLOCK_NODE_STATE + + *event = yaml_event_t{ + typ: yaml_DOCUMENT_START_EVENT, + start_mark: token.start_mark, + end_mark: token.end_mark, + } + + } else if token.typ != yaml_STREAM_END_TOKEN { + // Parse an explicit document. + var version_directive *yaml_version_directive_t + var tag_directives []yaml_tag_directive_t + start_mark := token.start_mark + if !yaml_parser_process_directives(parser, &version_directive, &tag_directives) { + return false + } + token = peek_token(parser) + if token == nil { + return false + } + if token.typ != yaml_DOCUMENT_START_TOKEN { + yaml_parser_set_parser_error(parser, + "did not find expected <document start>", token.start_mark) + return false + } + parser.states = append(parser.states, yaml_PARSE_DOCUMENT_END_STATE) + parser.state = yaml_PARSE_DOCUMENT_CONTENT_STATE + end_mark := token.end_mark + + *event = yaml_event_t{ + typ: yaml_DOCUMENT_START_EVENT, + start_mark: start_mark, + end_mark: end_mark, + version_directive: version_directive, + tag_directives: tag_directives, + implicit: false, + } + skip_token(parser) + + } else { + // Parse the stream end. + parser.state = yaml_PARSE_END_STATE + *event = yaml_event_t{ + typ: yaml_STREAM_END_EVENT, + start_mark: token.start_mark, + end_mark: token.end_mark, + } + skip_token(parser) + } + + return true +} + +// Parse the productions: +// explicit_document ::= DIRECTIVE* DOCUMENT-START block_node? DOCUMENT-END* +// *********** +// +func yaml_parser_parse_document_content(parser *yaml_parser_t, event *yaml_event_t) bool { + token := peek_token(parser) + if token == nil { + return false + } + if token.typ == yaml_VERSION_DIRECTIVE_TOKEN || + token.typ == yaml_TAG_DIRECTIVE_TOKEN || + token.typ == yaml_DOCUMENT_START_TOKEN || + token.typ == yaml_DOCUMENT_END_TOKEN || + token.typ == yaml_STREAM_END_TOKEN { + parser.state = parser.states[len(parser.states)-1] + parser.states = parser.states[:len(parser.states)-1] + return yaml_parser_process_empty_scalar(parser, event, + token.start_mark) + } + return yaml_parser_parse_node(parser, event, true, false) +} + +// Parse the productions: +// implicit_document ::= block_node DOCUMENT-END* +// ************* +// explicit_document ::= DIRECTIVE* DOCUMENT-START block_node? DOCUMENT-END* +// +func yaml_parser_parse_document_end(parser *yaml_parser_t, event *yaml_event_t) bool { + token := peek_token(parser) + if token == nil { + return false + } + + start_mark := token.start_mark + end_mark := token.start_mark + + implicit := true + if token.typ == yaml_DOCUMENT_END_TOKEN { + end_mark = token.end_mark + skip_token(parser) + implicit = false + } + + parser.tag_directives = parser.tag_directives[:0] + + parser.state = yaml_PARSE_DOCUMENT_START_STATE + *event = yaml_event_t{ + typ: yaml_DOCUMENT_END_EVENT, + start_mark: start_mark, + end_mark: end_mark, + implicit: implicit, + } + return true +} + +// Parse the productions: +// block_node_or_indentless_sequence ::= +// ALIAS +// ***** +// | properties (block_content | indentless_block_sequence)? +// ********** * +// | block_content | indentless_block_sequence +// * +// block_node ::= ALIAS +// ***** +// | properties block_content? +// ********** * +// | block_content +// * +// flow_node ::= ALIAS +// ***** +// | properties flow_content? +// ********** * +// | flow_content +// * +// properties ::= TAG ANCHOR? | ANCHOR TAG? +// ************************* +// block_content ::= block_collection | flow_collection | SCALAR +// ****** +// flow_content ::= flow_collection | SCALAR +// ****** +func yaml_parser_parse_node(parser *yaml_parser_t, event *yaml_event_t, block, indentless_sequence bool) bool { + //defer trace("yaml_parser_parse_node", "block:", block, "indentless_sequence:", indentless_sequence)() + + token := peek_token(parser) + if token == nil { + return false + } + + if token.typ == yaml_ALIAS_TOKEN { + parser.state = parser.states[len(parser.states)-1] + parser.states = parser.states[:len(parser.states)-1] + *event = yaml_event_t{ + typ: yaml_ALIAS_EVENT, + start_mark: token.start_mark, + end_mark: token.end_mark, + anchor: token.value, + } + skip_token(parser) + return true + } + + start_mark := token.start_mark + end_mark := token.start_mark + + var tag_token bool + var tag_handle, tag_suffix, anchor []byte + var tag_mark yaml_mark_t + if token.typ == yaml_ANCHOR_TOKEN { + anchor = token.value + start_mark = token.start_mark + end_mark = token.end_mark + skip_token(parser) + token = peek_token(parser) + if token == nil { + return false + } + if token.typ == yaml_TAG_TOKEN { + tag_token = true + tag_handle = token.value + tag_suffix = token.suffix + tag_mark = token.start_mark + end_mark = token.end_mark + skip_token(parser) + token = peek_token(parser) + if token == nil { + return false + } + } + } else if token.typ == yaml_TAG_TOKEN { + tag_token = true + tag_handle = token.value + tag_suffix = token.suffix + start_mark = token.start_mark + tag_mark = token.start_mark + end_mark = token.end_mark + skip_token(parser) + token = peek_token(parser) + if token == nil { + return false + } + if token.typ == yaml_ANCHOR_TOKEN { + anchor = token.value + end_mark = token.end_mark + skip_token(parser) + token = peek_token(parser) + if token == nil { + return false + } + } + } + + var tag []byte + if tag_token { + if len(tag_handle) == 0 { + tag = tag_suffix + tag_suffix = nil + } else { + for i := range parser.tag_directives { + if bytes.Equal(parser.tag_directives[i].handle, tag_handle) { + tag = append([]byte(nil), parser.tag_directives[i].prefix...) + tag = append(tag, tag_suffix...) + break + } + } + if len(tag) == 0 { + yaml_parser_set_parser_error_context(parser, + "while parsing a node", start_mark, + "found undefined tag handle", tag_mark) + return false + } + } + } + + implicit := len(tag) == 0 + if indentless_sequence && token.typ == yaml_BLOCK_ENTRY_TOKEN { + end_mark = token.end_mark + parser.state = yaml_PARSE_INDENTLESS_SEQUENCE_ENTRY_STATE + *event = yaml_event_t{ + typ: yaml_SEQUENCE_START_EVENT, + start_mark: start_mark, + end_mark: end_mark, + anchor: anchor, + tag: tag, + implicit: implicit, + style: yaml_style_t(yaml_BLOCK_SEQUENCE_STYLE), + } + return true + } + if token.typ == yaml_SCALAR_TOKEN { + var plain_implicit, quoted_implicit bool + end_mark = token.end_mark + if (len(tag) == 0 && token.style == yaml_PLAIN_SCALAR_STYLE) || (len(tag) == 1 && tag[0] == '!') { + plain_implicit = true + } else if len(tag) == 0 { + quoted_implicit = true + } + parser.state = parser.states[len(parser.states)-1] + parser.states = parser.states[:len(parser.states)-1] + + *event = yaml_event_t{ + typ: yaml_SCALAR_EVENT, + start_mark: start_mark, + end_mark: end_mark, + anchor: anchor, + tag: tag, + value: token.value, + implicit: plain_implicit, + quoted_implicit: quoted_implicit, + style: yaml_style_t(token.style), + } + skip_token(parser) + return true + } + if token.typ == yaml_FLOW_SEQUENCE_START_TOKEN { + // [Go] Some of the events below can be merged as they differ only on style. + end_mark = token.end_mark + parser.state = yaml_PARSE_FLOW_SEQUENCE_FIRST_ENTRY_STATE + *event = yaml_event_t{ + typ: yaml_SEQUENCE_START_EVENT, + start_mark: start_mark, + end_mark: end_mark, + anchor: anchor, + tag: tag, + implicit: implicit, + style: yaml_style_t(yaml_FLOW_SEQUENCE_STYLE), + } + return true + } + if token.typ == yaml_FLOW_MAPPING_START_TOKEN { + end_mark = token.end_mark + parser.state = yaml_PARSE_FLOW_MAPPING_FIRST_KEY_STATE + *event = yaml_event_t{ + typ: yaml_MAPPING_START_EVENT, + start_mark: start_mark, + end_mark: end_mark, + anchor: anchor, + tag: tag, + implicit: implicit, + style: yaml_style_t(yaml_FLOW_MAPPING_STYLE), + } + return true + } + if block && token.typ == yaml_BLOCK_SEQUENCE_START_TOKEN { + end_mark = token.end_mark + parser.state = yaml_PARSE_BLOCK_SEQUENCE_FIRST_ENTRY_STATE + *event = yaml_event_t{ + typ: yaml_SEQUENCE_START_EVENT, + start_mark: start_mark, + end_mark: end_mark, + anchor: anchor, + tag: tag, + implicit: implicit, + style: yaml_style_t(yaml_BLOCK_SEQUENCE_STYLE), + } + return true + } + if block && token.typ == yaml_BLOCK_MAPPING_START_TOKEN { + end_mark = token.end_mark + parser.state = yaml_PARSE_BLOCK_MAPPING_FIRST_KEY_STATE + *event = yaml_event_t{ + typ: yaml_MAPPING_START_EVENT, + start_mark: start_mark, + end_mark: end_mark, + anchor: anchor, + tag: tag, + implicit: implicit, + style: yaml_style_t(yaml_BLOCK_MAPPING_STYLE), + } + return true + } + if len(anchor) > 0 || len(tag) > 0 { + parser.state = parser.states[len(parser.states)-1] + parser.states = parser.states[:len(parser.states)-1] + + *event = yaml_event_t{ + typ: yaml_SCALAR_EVENT, + start_mark: start_mark, + end_mark: end_mark, + anchor: anchor, + tag: tag, + implicit: implicit, + quoted_implicit: false, + style: yaml_style_t(yaml_PLAIN_SCALAR_STYLE), + } + return true + } + + context := "while parsing a flow node" + if block { + context = "while parsing a block node" + } + yaml_parser_set_parser_error_context(parser, context, start_mark, + "did not find expected node content", token.start_mark) + return false +} + +// Parse the productions: +// block_sequence ::= BLOCK-SEQUENCE-START (BLOCK-ENTRY block_node?)* BLOCK-END +// ******************** *********** * ********* +// +func yaml_parser_parse_block_sequence_entry(parser *yaml_parser_t, event *yaml_event_t, first bool) bool { + if first { + token := peek_token(parser) + parser.marks = append(parser.marks, token.start_mark) + skip_token(parser) + } + + token := peek_token(parser) + if token == nil { + return false + } + + if token.typ == yaml_BLOCK_ENTRY_TOKEN { + mark := token.end_mark + skip_token(parser) + token = peek_token(parser) + if token == nil { + return false + } + if token.typ != yaml_BLOCK_ENTRY_TOKEN && token.typ != yaml_BLOCK_END_TOKEN { + parser.states = append(parser.states, yaml_PARSE_BLOCK_SEQUENCE_ENTRY_STATE) + return yaml_parser_parse_node(parser, event, true, false) + } else { + parser.state = yaml_PARSE_BLOCK_SEQUENCE_ENTRY_STATE + return yaml_parser_process_empty_scalar(parser, event, mark) + } + } + if token.typ == yaml_BLOCK_END_TOKEN { + parser.state = parser.states[len(parser.states)-1] + parser.states = parser.states[:len(parser.states)-1] + parser.marks = parser.marks[:len(parser.marks)-1] + + *event = yaml_event_t{ + typ: yaml_SEQUENCE_END_EVENT, + start_mark: token.start_mark, + end_mark: token.end_mark, + } + + skip_token(parser) + return true + } + + context_mark := parser.marks[len(parser.marks)-1] + parser.marks = parser.marks[:len(parser.marks)-1] + return yaml_parser_set_parser_error_context(parser, + "while parsing a block collection", context_mark, + "did not find expected '-' indicator", token.start_mark) +} + +// Parse the productions: +// indentless_sequence ::= (BLOCK-ENTRY block_node?)+ +// *********** * +func yaml_parser_parse_indentless_sequence_entry(parser *yaml_parser_t, event *yaml_event_t) bool { + token := peek_token(parser) + if token == nil { + return false + } + + if token.typ == yaml_BLOCK_ENTRY_TOKEN { + mark := token.end_mark + skip_token(parser) + token = peek_token(parser) + if token == nil { + return false + } + if token.typ != yaml_BLOCK_ENTRY_TOKEN && + token.typ != yaml_KEY_TOKEN && + token.typ != yaml_VALUE_TOKEN && + token.typ != yaml_BLOCK_END_TOKEN { + parser.states = append(parser.states, yaml_PARSE_INDENTLESS_SEQUENCE_ENTRY_STATE) + return yaml_parser_parse_node(parser, event, true, false) + } + parser.state = yaml_PARSE_INDENTLESS_SEQUENCE_ENTRY_STATE + return yaml_parser_process_empty_scalar(parser, event, mark) + } + parser.state = parser.states[len(parser.states)-1] + parser.states = parser.states[:len(parser.states)-1] + + *event = yaml_event_t{ + typ: yaml_SEQUENCE_END_EVENT, + start_mark: token.start_mark, + end_mark: token.start_mark, // [Go] Shouldn't this be token.end_mark? + } + return true +} + +// Parse the productions: +// block_mapping ::= BLOCK-MAPPING_START +// ******************* +// ((KEY block_node_or_indentless_sequence?)? +// *** * +// (VALUE block_node_or_indentless_sequence?)?)* +// +// BLOCK-END +// ********* +// +func yaml_parser_parse_block_mapping_key(parser *yaml_parser_t, event *yaml_event_t, first bool) bool { + if first { + token := peek_token(parser) + parser.marks = append(parser.marks, token.start_mark) + skip_token(parser) + } + + token := peek_token(parser) + if token == nil { + return false + } + + if token.typ == yaml_KEY_TOKEN { + mark := token.end_mark + skip_token(parser) + token = peek_token(parser) + if token == nil { + return false + } + if token.typ != yaml_KEY_TOKEN && + token.typ != yaml_VALUE_TOKEN && + token.typ != yaml_BLOCK_END_TOKEN { + parser.states = append(parser.states, yaml_PARSE_BLOCK_MAPPING_VALUE_STATE) + return yaml_parser_parse_node(parser, event, true, true) + } else { + parser.state = yaml_PARSE_BLOCK_MAPPING_VALUE_STATE + return yaml_parser_process_empty_scalar(parser, event, mark) + } + } else if token.typ == yaml_BLOCK_END_TOKEN { + parser.state = parser.states[len(parser.states)-1] + parser.states = parser.states[:len(parser.states)-1] + parser.marks = parser.marks[:len(parser.marks)-1] + *event = yaml_event_t{ + typ: yaml_MAPPING_END_EVENT, + start_mark: token.start_mark, + end_mark: token.end_mark, + } + skip_token(parser) + return true + } + + context_mark := parser.marks[len(parser.marks)-1] + parser.marks = parser.marks[:len(parser.marks)-1] + return yaml_parser_set_parser_error_context(parser, + "while parsing a block mapping", context_mark, + "did not find expected key", token.start_mark) +} + +// Parse the productions: +// block_mapping ::= BLOCK-MAPPING_START +// +// ((KEY block_node_or_indentless_sequence?)? +// +// (VALUE block_node_or_indentless_sequence?)?)* +// ***** * +// BLOCK-END +// +// +func yaml_parser_parse_block_mapping_value(parser *yaml_parser_t, event *yaml_event_t) bool { + token := peek_token(parser) + if token == nil { + return false + } + if token.typ == yaml_VALUE_TOKEN { + mark := token.end_mark + skip_token(parser) + token = peek_token(parser) + if token == nil { + return false + } + if token.typ != yaml_KEY_TOKEN && + token.typ != yaml_VALUE_TOKEN && + token.typ != yaml_BLOCK_END_TOKEN { + parser.states = append(parser.states, yaml_PARSE_BLOCK_MAPPING_KEY_STATE) + return yaml_parser_parse_node(parser, event, true, true) + } + parser.state = yaml_PARSE_BLOCK_MAPPING_KEY_STATE + return yaml_parser_process_empty_scalar(parser, event, mark) + } + parser.state = yaml_PARSE_BLOCK_MAPPING_KEY_STATE + return yaml_parser_process_empty_scalar(parser, event, token.start_mark) +} + +// Parse the productions: +// flow_sequence ::= FLOW-SEQUENCE-START +// ******************* +// (flow_sequence_entry FLOW-ENTRY)* +// * ********** +// flow_sequence_entry? +// * +// FLOW-SEQUENCE-END +// ***************** +// flow_sequence_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? +// * +// +func yaml_parser_parse_flow_sequence_entry(parser *yaml_parser_t, event *yaml_event_t, first bool) bool { + if first { + token := peek_token(parser) + parser.marks = append(parser.marks, token.start_mark) + skip_token(parser) + } + token := peek_token(parser) + if token == nil { + return false + } + if token.typ != yaml_FLOW_SEQUENCE_END_TOKEN { + if !first { + if token.typ == yaml_FLOW_ENTRY_TOKEN { + skip_token(parser) + token = peek_token(parser) + if token == nil { + return false + } + } else { + context_mark := parser.marks[len(parser.marks)-1] + parser.marks = parser.marks[:len(parser.marks)-1] + return yaml_parser_set_parser_error_context(parser, + "while parsing a flow sequence", context_mark, + "did not find expected ',' or ']'", token.start_mark) + } + } + + if token.typ == yaml_KEY_TOKEN { + parser.state = yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_KEY_STATE + *event = yaml_event_t{ + typ: yaml_MAPPING_START_EVENT, + start_mark: token.start_mark, + end_mark: token.end_mark, + implicit: true, + style: yaml_style_t(yaml_FLOW_MAPPING_STYLE), + } + skip_token(parser) + return true + } else if token.typ != yaml_FLOW_SEQUENCE_END_TOKEN { + parser.states = append(parser.states, yaml_PARSE_FLOW_SEQUENCE_ENTRY_STATE) + return yaml_parser_parse_node(parser, event, false, false) + } + } + + parser.state = parser.states[len(parser.states)-1] + parser.states = parser.states[:len(parser.states)-1] + parser.marks = parser.marks[:len(parser.marks)-1] + + *event = yaml_event_t{ + typ: yaml_SEQUENCE_END_EVENT, + start_mark: token.start_mark, + end_mark: token.end_mark, + } + + skip_token(parser) + return true +} + +// +// Parse the productions: +// flow_sequence_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? +// *** * +// +func yaml_parser_parse_flow_sequence_entry_mapping_key(parser *yaml_parser_t, event *yaml_event_t) bool { + token := peek_token(parser) + if token == nil { + return false + } + if token.typ != yaml_VALUE_TOKEN && + token.typ != yaml_FLOW_ENTRY_TOKEN && + token.typ != yaml_FLOW_SEQUENCE_END_TOKEN { + parser.states = append(parser.states, yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_VALUE_STATE) + return yaml_parser_parse_node(parser, event, false, false) + } + mark := token.end_mark + skip_token(parser) + parser.state = yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_VALUE_STATE + return yaml_parser_process_empty_scalar(parser, event, mark) +} + +// Parse the productions: +// flow_sequence_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? +// ***** * +// +func yaml_parser_parse_flow_sequence_entry_mapping_value(parser *yaml_parser_t, event *yaml_event_t) bool { + token := peek_token(parser) + if token == nil { + return false + } + if token.typ == yaml_VALUE_TOKEN { + skip_token(parser) + token := peek_token(parser) + if token == nil { + return false + } + if token.typ != yaml_FLOW_ENTRY_TOKEN && token.typ != yaml_FLOW_SEQUENCE_END_TOKEN { + parser.states = append(parser.states, yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_END_STATE) + return yaml_parser_parse_node(parser, event, false, false) + } + } + parser.state = yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_END_STATE + return yaml_parser_process_empty_scalar(parser, event, token.start_mark) +} + +// Parse the productions: +// flow_sequence_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? +// * +// +func yaml_parser_parse_flow_sequence_entry_mapping_end(parser *yaml_parser_t, event *yaml_event_t) bool { + token := peek_token(parser) + if token == nil { + return false + } + parser.state = yaml_PARSE_FLOW_SEQUENCE_ENTRY_STATE + *event = yaml_event_t{ + typ: yaml_MAPPING_END_EVENT, + start_mark: token.start_mark, + end_mark: token.start_mark, // [Go] Shouldn't this be end_mark? + } + return true +} + +// Parse the productions: +// flow_mapping ::= FLOW-MAPPING-START +// ****************** +// (flow_mapping_entry FLOW-ENTRY)* +// * ********** +// flow_mapping_entry? +// ****************** +// FLOW-MAPPING-END +// **************** +// flow_mapping_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? +// * *** * +// +func yaml_parser_parse_flow_mapping_key(parser *yaml_parser_t, event *yaml_event_t, first bool) bool { + if first { + token := peek_token(parser) + parser.marks = append(parser.marks, token.start_mark) + skip_token(parser) + } + + token := peek_token(parser) + if token == nil { + return false + } + + if token.typ != yaml_FLOW_MAPPING_END_TOKEN { + if !first { + if token.typ == yaml_FLOW_ENTRY_TOKEN { + skip_token(parser) + token = peek_token(parser) + if token == nil { + return false + } + } else { + context_mark := parser.marks[len(parser.marks)-1] + parser.marks = parser.marks[:len(parser.marks)-1] + return yaml_parser_set_parser_error_context(parser, + "while parsing a flow mapping", context_mark, + "did not find expected ',' or '}'", token.start_mark) + } + } + + if token.typ == yaml_KEY_TOKEN { + skip_token(parser) + token = peek_token(parser) + if token == nil { + return false + } + if token.typ != yaml_VALUE_TOKEN && + token.typ != yaml_FLOW_ENTRY_TOKEN && + token.typ != yaml_FLOW_MAPPING_END_TOKEN { + parser.states = append(parser.states, yaml_PARSE_FLOW_MAPPING_VALUE_STATE) + return yaml_parser_parse_node(parser, event, false, false) + } else { + parser.state = yaml_PARSE_FLOW_MAPPING_VALUE_STATE + return yaml_parser_process_empty_scalar(parser, event, token.start_mark) + } + } else if token.typ != yaml_FLOW_MAPPING_END_TOKEN { + parser.states = append(parser.states, yaml_PARSE_FLOW_MAPPING_EMPTY_VALUE_STATE) + return yaml_parser_parse_node(parser, event, false, false) + } + } + + parser.state = parser.states[len(parser.states)-1] + parser.states = parser.states[:len(parser.states)-1] + parser.marks = parser.marks[:len(parser.marks)-1] + *event = yaml_event_t{ + typ: yaml_MAPPING_END_EVENT, + start_mark: token.start_mark, + end_mark: token.end_mark, + } + skip_token(parser) + return true +} + +// Parse the productions: +// flow_mapping_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? +// * ***** * +// +func yaml_parser_parse_flow_mapping_value(parser *yaml_parser_t, event *yaml_event_t, empty bool) bool { + token := peek_token(parser) + if token == nil { + return false + } + if empty { + parser.state = yaml_PARSE_FLOW_MAPPING_KEY_STATE + return yaml_parser_process_empty_scalar(parser, event, token.start_mark) + } + if token.typ == yaml_VALUE_TOKEN { + skip_token(parser) + token = peek_token(parser) + if token == nil { + return false + } + if token.typ != yaml_FLOW_ENTRY_TOKEN && token.typ != yaml_FLOW_MAPPING_END_TOKEN { + parser.states = append(parser.states, yaml_PARSE_FLOW_MAPPING_KEY_STATE) + return yaml_parser_parse_node(parser, event, false, false) + } + } + parser.state = yaml_PARSE_FLOW_MAPPING_KEY_STATE + return yaml_parser_process_empty_scalar(parser, event, token.start_mark) +} + +// Generate an empty scalar event. +func yaml_parser_process_empty_scalar(parser *yaml_parser_t, event *yaml_event_t, mark yaml_mark_t) bool { + *event = yaml_event_t{ + typ: yaml_SCALAR_EVENT, + start_mark: mark, + end_mark: mark, + value: nil, // Empty + implicit: true, + style: yaml_style_t(yaml_PLAIN_SCALAR_STYLE), + } + return true +} + +var default_tag_directives = []yaml_tag_directive_t{ + {[]byte("!"), []byte("!")}, + {[]byte("!!"), []byte("tag:yaml.org,2002:")}, +} + +// Parse directives. +func yaml_parser_process_directives(parser *yaml_parser_t, + version_directive_ref **yaml_version_directive_t, + tag_directives_ref *[]yaml_tag_directive_t) bool { + + var version_directive *yaml_version_directive_t + var tag_directives []yaml_tag_directive_t + + token := peek_token(parser) + if token == nil { + return false + } + + for token.typ == yaml_VERSION_DIRECTIVE_TOKEN || token.typ == yaml_TAG_DIRECTIVE_TOKEN { + if token.typ == yaml_VERSION_DIRECTIVE_TOKEN { + if version_directive != nil { + yaml_parser_set_parser_error(parser, + "found duplicate %YAML directive", token.start_mark) + return false + } + if token.major != 1 || token.minor != 1 { + yaml_parser_set_parser_error(parser, + "found incompatible YAML document", token.start_mark) + return false + } + version_directive = &yaml_version_directive_t{ + major: token.major, + minor: token.minor, + } + } else if token.typ == yaml_TAG_DIRECTIVE_TOKEN { + value := yaml_tag_directive_t{ + handle: token.value, + prefix: token.prefix, + } + if !yaml_parser_append_tag_directive(parser, value, false, token.start_mark) { + return false + } + tag_directives = append(tag_directives, value) + } + + skip_token(parser) + token = peek_token(parser) + if token == nil { + return false + } + } + + for i := range default_tag_directives { + if !yaml_parser_append_tag_directive(parser, default_tag_directives[i], true, token.start_mark) { + return false + } + } + + if version_directive_ref != nil { + *version_directive_ref = version_directive + } + if tag_directives_ref != nil { + *tag_directives_ref = tag_directives + } + return true +} + +// Append a tag directive to the directives stack. +func yaml_parser_append_tag_directive(parser *yaml_parser_t, value yaml_tag_directive_t, allow_duplicates bool, mark yaml_mark_t) bool { + for i := range parser.tag_directives { + if bytes.Equal(value.handle, parser.tag_directives[i].handle) { + if allow_duplicates { + return true + } + return yaml_parser_set_parser_error(parser, "found duplicate %TAG directive", mark) + } + } + + // [Go] I suspect the copy is unnecessary. This was likely done + // because there was no way to track ownership of the data. + value_copy := yaml_tag_directive_t{ + handle: make([]byte, len(value.handle)), + prefix: make([]byte, len(value.prefix)), + } + copy(value_copy.handle, value.handle) + copy(value_copy.prefix, value.prefix) + parser.tag_directives = append(parser.tag_directives, value_copy) + return true +} diff --git a/generator/vendor/gopkg.in/yaml.v2/readerc.go b/generator/vendor/gopkg.in/yaml.v2/readerc.go new file mode 100644 index 00000000..f4507917 --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/readerc.go @@ -0,0 +1,394 @@ +package yaml + +import ( + "io" +) + +// Set the reader error and return 0. +func yaml_parser_set_reader_error(parser *yaml_parser_t, problem string, offset int, value int) bool { + parser.error = yaml_READER_ERROR + parser.problem = problem + parser.problem_offset = offset + parser.problem_value = value + return false +} + +// Byte order marks. +const ( + bom_UTF8 = "\xef\xbb\xbf" + bom_UTF16LE = "\xff\xfe" + bom_UTF16BE = "\xfe\xff" +) + +// Determine the input stream encoding by checking the BOM symbol. If no BOM is +// found, the UTF-8 encoding is assumed. Return 1 on success, 0 on failure. +func yaml_parser_determine_encoding(parser *yaml_parser_t) bool { + // Ensure that we had enough bytes in the raw buffer. + for !parser.eof && len(parser.raw_buffer)-parser.raw_buffer_pos < 3 { + if !yaml_parser_update_raw_buffer(parser) { + return false + } + } + + // Determine the encoding. + buf := parser.raw_buffer + pos := parser.raw_buffer_pos + avail := len(buf) - pos + if avail >= 2 && buf[pos] == bom_UTF16LE[0] && buf[pos+1] == bom_UTF16LE[1] { + parser.encoding = yaml_UTF16LE_ENCODING + parser.raw_buffer_pos += 2 + parser.offset += 2 + } else if avail >= 2 && buf[pos] == bom_UTF16BE[0] && buf[pos+1] == bom_UTF16BE[1] { + parser.encoding = yaml_UTF16BE_ENCODING + parser.raw_buffer_pos += 2 + parser.offset += 2 + } else if avail >= 3 && buf[pos] == bom_UTF8[0] && buf[pos+1] == bom_UTF8[1] && buf[pos+2] == bom_UTF8[2] { + parser.encoding = yaml_UTF8_ENCODING + parser.raw_buffer_pos += 3 + parser.offset += 3 + } else { + parser.encoding = yaml_UTF8_ENCODING + } + return true +} + +// Update the raw buffer. +func yaml_parser_update_raw_buffer(parser *yaml_parser_t) bool { + size_read := 0 + + // Return if the raw buffer is full. + if parser.raw_buffer_pos == 0 && len(parser.raw_buffer) == cap(parser.raw_buffer) { + return true + } + + // Return on EOF. + if parser.eof { + return true + } + + // Move the remaining bytes in the raw buffer to the beginning. + if parser.raw_buffer_pos > 0 && parser.raw_buffer_pos < len(parser.raw_buffer) { + copy(parser.raw_buffer, parser.raw_buffer[parser.raw_buffer_pos:]) + } + parser.raw_buffer = parser.raw_buffer[:len(parser.raw_buffer)-parser.raw_buffer_pos] + parser.raw_buffer_pos = 0 + + // Call the read handler to fill the buffer. + size_read, err := parser.read_handler(parser, parser.raw_buffer[len(parser.raw_buffer):cap(parser.raw_buffer)]) + parser.raw_buffer = parser.raw_buffer[:len(parser.raw_buffer)+size_read] + if err == io.EOF { + parser.eof = true + } else if err != nil { + return yaml_parser_set_reader_error(parser, "input error: "+err.Error(), parser.offset, -1) + } + return true +} + +// Ensure that the buffer contains at least `length` characters. +// Return true on success, false on failure. +// +// The length is supposed to be significantly less that the buffer size. +func yaml_parser_update_buffer(parser *yaml_parser_t, length int) bool { + if parser.read_handler == nil { + panic("read handler must be set") + } + + // If the EOF flag is set and the raw buffer is empty, do nothing. + if parser.eof && parser.raw_buffer_pos == len(parser.raw_buffer) { + return true + } + + // Return if the buffer contains enough characters. + if parser.unread >= length { + return true + } + + // Determine the input encoding if it is not known yet. + if parser.encoding == yaml_ANY_ENCODING { + if !yaml_parser_determine_encoding(parser) { + return false + } + } + + // Move the unread characters to the beginning of the buffer. + buffer_len := len(parser.buffer) + if parser.buffer_pos > 0 && parser.buffer_pos < buffer_len { + copy(parser.buffer, parser.buffer[parser.buffer_pos:]) + buffer_len -= parser.buffer_pos + parser.buffer_pos = 0 + } else if parser.buffer_pos == buffer_len { + buffer_len = 0 + parser.buffer_pos = 0 + } + + // Open the whole buffer for writing, and cut it before returning. + parser.buffer = parser.buffer[:cap(parser.buffer)] + + // Fill the buffer until it has enough characters. + first := true + for parser.unread < length { + + // Fill the raw buffer if necessary. + if !first || parser.raw_buffer_pos == len(parser.raw_buffer) { + if !yaml_parser_update_raw_buffer(parser) { + parser.buffer = parser.buffer[:buffer_len] + return false + } + } + first = false + + // Decode the raw buffer. + inner: + for parser.raw_buffer_pos != len(parser.raw_buffer) { + var value rune + var width int + + raw_unread := len(parser.raw_buffer) - parser.raw_buffer_pos + + // Decode the next character. + switch parser.encoding { + case yaml_UTF8_ENCODING: + // Decode a UTF-8 character. Check RFC 3629 + // (http://www.ietf.org/rfc/rfc3629.txt) for more details. + // + // The following table (taken from the RFC) is used for + // decoding. + // + // Char. number range | UTF-8 octet sequence + // (hexadecimal) | (binary) + // --------------------+------------------------------------ + // 0000 0000-0000 007F | 0xxxxxxx + // 0000 0080-0000 07FF | 110xxxxx 10xxxxxx + // 0000 0800-0000 FFFF | 1110xxxx 10xxxxxx 10xxxxxx + // 0001 0000-0010 FFFF | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx + // + // Additionally, the characters in the range 0xD800-0xDFFF + // are prohibited as they are reserved for use with UTF-16 + // surrogate pairs. + + // Determine the length of the UTF-8 sequence. + octet := parser.raw_buffer[parser.raw_buffer_pos] + switch { + case octet&0x80 == 0x00: + width = 1 + case octet&0xE0 == 0xC0: + width = 2 + case octet&0xF0 == 0xE0: + width = 3 + case octet&0xF8 == 0xF0: + width = 4 + default: + // The leading octet is invalid. + return yaml_parser_set_reader_error(parser, + "invalid leading UTF-8 octet", + parser.offset, int(octet)) + } + + // Check if the raw buffer contains an incomplete character. + if width > raw_unread { + if parser.eof { + return yaml_parser_set_reader_error(parser, + "incomplete UTF-8 octet sequence", + parser.offset, -1) + } + break inner + } + + // Decode the leading octet. + switch { + case octet&0x80 == 0x00: + value = rune(octet & 0x7F) + case octet&0xE0 == 0xC0: + value = rune(octet & 0x1F) + case octet&0xF0 == 0xE0: + value = rune(octet & 0x0F) + case octet&0xF8 == 0xF0: + value = rune(octet & 0x07) + default: + value = 0 + } + + // Check and decode the trailing octets. + for k := 1; k < width; k++ { + octet = parser.raw_buffer[parser.raw_buffer_pos+k] + + // Check if the octet is valid. + if (octet & 0xC0) != 0x80 { + return yaml_parser_set_reader_error(parser, + "invalid trailing UTF-8 octet", + parser.offset+k, int(octet)) + } + + // Decode the octet. + value = (value << 6) + rune(octet&0x3F) + } + + // Check the length of the sequence against the value. + switch { + case width == 1: + case width == 2 && value >= 0x80: + case width == 3 && value >= 0x800: + case width == 4 && value >= 0x10000: + default: + return yaml_parser_set_reader_error(parser, + "invalid length of a UTF-8 sequence", + parser.offset, -1) + } + + // Check the range of the value. + if value >= 0xD800 && value <= 0xDFFF || value > 0x10FFFF { + return yaml_parser_set_reader_error(parser, + "invalid Unicode character", + parser.offset, int(value)) + } + + case yaml_UTF16LE_ENCODING, yaml_UTF16BE_ENCODING: + var low, high int + if parser.encoding == yaml_UTF16LE_ENCODING { + low, high = 0, 1 + } else { + low, high = 1, 0 + } + + // The UTF-16 encoding is not as simple as one might + // naively think. Check RFC 2781 + // (http://www.ietf.org/rfc/rfc2781.txt). + // + // Normally, two subsequent bytes describe a Unicode + // character. However a special technique (called a + // surrogate pair) is used for specifying character + // values larger than 0xFFFF. + // + // A surrogate pair consists of two pseudo-characters: + // high surrogate area (0xD800-0xDBFF) + // low surrogate area (0xDC00-0xDFFF) + // + // The following formulas are used for decoding + // and encoding characters using surrogate pairs: + // + // U = U' + 0x10000 (0x01 00 00 <= U <= 0x10 FF FF) + // U' = yyyyyyyyyyxxxxxxxxxx (0 <= U' <= 0x0F FF FF) + // W1 = 110110yyyyyyyyyy + // W2 = 110111xxxxxxxxxx + // + // where U is the character value, W1 is the high surrogate + // area, W2 is the low surrogate area. + + // Check for incomplete UTF-16 character. + if raw_unread < 2 { + if parser.eof { + return yaml_parser_set_reader_error(parser, + "incomplete UTF-16 character", + parser.offset, -1) + } + break inner + } + + // Get the character. + value = rune(parser.raw_buffer[parser.raw_buffer_pos+low]) + + (rune(parser.raw_buffer[parser.raw_buffer_pos+high]) << 8) + + // Check for unexpected low surrogate area. + if value&0xFC00 == 0xDC00 { + return yaml_parser_set_reader_error(parser, + "unexpected low surrogate area", + parser.offset, int(value)) + } + + // Check for a high surrogate area. + if value&0xFC00 == 0xD800 { + width = 4 + + // Check for incomplete surrogate pair. + if raw_unread < 4 { + if parser.eof { + return yaml_parser_set_reader_error(parser, + "incomplete UTF-16 surrogate pair", + parser.offset, -1) + } + break inner + } + + // Get the next character. + value2 := rune(parser.raw_buffer[parser.raw_buffer_pos+low+2]) + + (rune(parser.raw_buffer[parser.raw_buffer_pos+high+2]) << 8) + + // Check for a low surrogate area. + if value2&0xFC00 != 0xDC00 { + return yaml_parser_set_reader_error(parser, + "expected low surrogate area", + parser.offset+2, int(value2)) + } + + // Generate the value of the surrogate pair. + value = 0x10000 + ((value & 0x3FF) << 10) + (value2 & 0x3FF) + } else { + width = 2 + } + + default: + panic("impossible") + } + + // Check if the character is in the allowed range: + // #x9 | #xA | #xD | [#x20-#x7E] (8 bit) + // | #x85 | [#xA0-#xD7FF] | [#xE000-#xFFFD] (16 bit) + // | [#x10000-#x10FFFF] (32 bit) + switch { + case value == 0x09: + case value == 0x0A: + case value == 0x0D: + case value >= 0x20 && value <= 0x7E: + case value == 0x85: + case value >= 0xA0 && value <= 0xD7FF: + case value >= 0xE000 && value <= 0xFFFD: + case value >= 0x10000 && value <= 0x10FFFF: + default: + return yaml_parser_set_reader_error(parser, + "control characters are not allowed", + parser.offset, int(value)) + } + + // Move the raw pointers. + parser.raw_buffer_pos += width + parser.offset += width + + // Finally put the character into the buffer. + if value <= 0x7F { + // 0000 0000-0000 007F . 0xxxxxxx + parser.buffer[buffer_len+0] = byte(value) + buffer_len += 1 + } else if value <= 0x7FF { + // 0000 0080-0000 07FF . 110xxxxx 10xxxxxx + parser.buffer[buffer_len+0] = byte(0xC0 + (value >> 6)) + parser.buffer[buffer_len+1] = byte(0x80 + (value & 0x3F)) + buffer_len += 2 + } else if value <= 0xFFFF { + // 0000 0800-0000 FFFF . 1110xxxx 10xxxxxx 10xxxxxx + parser.buffer[buffer_len+0] = byte(0xE0 + (value >> 12)) + parser.buffer[buffer_len+1] = byte(0x80 + ((value >> 6) & 0x3F)) + parser.buffer[buffer_len+2] = byte(0x80 + (value & 0x3F)) + buffer_len += 3 + } else { + // 0001 0000-0010 FFFF . 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx + parser.buffer[buffer_len+0] = byte(0xF0 + (value >> 18)) + parser.buffer[buffer_len+1] = byte(0x80 + ((value >> 12) & 0x3F)) + parser.buffer[buffer_len+2] = byte(0x80 + ((value >> 6) & 0x3F)) + parser.buffer[buffer_len+3] = byte(0x80 + (value & 0x3F)) + buffer_len += 4 + } + + parser.unread++ + } + + // On EOF, put NUL into the buffer and return. + if parser.eof { + parser.buffer[buffer_len] = 0 + buffer_len++ + parser.unread++ + break + } + } + parser.buffer = parser.buffer[:buffer_len] + return true +} diff --git a/generator/vendor/gopkg.in/yaml.v2/resolve.go b/generator/vendor/gopkg.in/yaml.v2/resolve.go new file mode 100644 index 00000000..232313cc --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/resolve.go @@ -0,0 +1,208 @@ +package yaml + +import ( + "encoding/base64" + "math" + "regexp" + "strconv" + "strings" + "unicode/utf8" +) + +type resolveMapItem struct { + value interface{} + tag string +} + +var resolveTable = make([]byte, 256) +var resolveMap = make(map[string]resolveMapItem) + +func init() { + t := resolveTable + t[int('+')] = 'S' // Sign + t[int('-')] = 'S' + for _, c := range "0123456789" { + t[int(c)] = 'D' // Digit + } + for _, c := range "yYnNtTfFoO~" { + t[int(c)] = 'M' // In map + } + t[int('.')] = '.' // Float (potentially in map) + + var resolveMapList = []struct { + v interface{} + tag string + l []string + }{ + {true, yaml_BOOL_TAG, []string{"y", "Y", "yes", "Yes", "YES"}}, + {true, yaml_BOOL_TAG, []string{"true", "True", "TRUE"}}, + {true, yaml_BOOL_TAG, []string{"on", "On", "ON"}}, + {false, yaml_BOOL_TAG, []string{"n", "N", "no", "No", "NO"}}, + {false, yaml_BOOL_TAG, []string{"false", "False", "FALSE"}}, + {false, yaml_BOOL_TAG, []string{"off", "Off", "OFF"}}, + {nil, yaml_NULL_TAG, []string{"", "~", "null", "Null", "NULL"}}, + {math.NaN(), yaml_FLOAT_TAG, []string{".nan", ".NaN", ".NAN"}}, + {math.Inf(+1), yaml_FLOAT_TAG, []string{".inf", ".Inf", ".INF"}}, + {math.Inf(+1), yaml_FLOAT_TAG, []string{"+.inf", "+.Inf", "+.INF"}}, + {math.Inf(-1), yaml_FLOAT_TAG, []string{"-.inf", "-.Inf", "-.INF"}}, + {"<<", yaml_MERGE_TAG, []string{"<<"}}, + } + + m := resolveMap + for _, item := range resolveMapList { + for _, s := range item.l { + m[s] = resolveMapItem{item.v, item.tag} + } + } +} + +const longTagPrefix = "tag:yaml.org,2002:" + +func shortTag(tag string) string { + // TODO This can easily be made faster and produce less garbage. + if strings.HasPrefix(tag, longTagPrefix) { + return "!!" + tag[len(longTagPrefix):] + } + return tag +} + +func longTag(tag string) string { + if strings.HasPrefix(tag, "!!") { + return longTagPrefix + tag[2:] + } + return tag +} + +func resolvableTag(tag string) bool { + switch tag { + case "", yaml_STR_TAG, yaml_BOOL_TAG, yaml_INT_TAG, yaml_FLOAT_TAG, yaml_NULL_TAG: + return true + } + return false +} + +var yamlStyleFloat = regexp.MustCompile(`^[-+]?[0-9]*\.?[0-9]+([eE][-+][0-9]+)?$`) + +func resolve(tag string, in string) (rtag string, out interface{}) { + if !resolvableTag(tag) { + return tag, in + } + + defer func() { + switch tag { + case "", rtag, yaml_STR_TAG, yaml_BINARY_TAG: + return + } + failf("cannot decode %s `%s` as a %s", shortTag(rtag), in, shortTag(tag)) + }() + + // Any data is accepted as a !!str or !!binary. + // Otherwise, the prefix is enough of a hint about what it might be. + hint := byte('N') + if in != "" { + hint = resolveTable[in[0]] + } + if hint != 0 && tag != yaml_STR_TAG && tag != yaml_BINARY_TAG { + // Handle things we can lookup in a map. + if item, ok := resolveMap[in]; ok { + return item.tag, item.value + } + + // Base 60 floats are a bad idea, were dropped in YAML 1.2, and + // are purposefully unsupported here. They're still quoted on + // the way out for compatibility with other parser, though. + + switch hint { + case 'M': + // We've already checked the map above. + + case '.': + // Not in the map, so maybe a normal float. + floatv, err := strconv.ParseFloat(in, 64) + if err == nil { + return yaml_FLOAT_TAG, floatv + } + + case 'D', 'S': + // Int, float, or timestamp. + plain := strings.Replace(in, "_", "", -1) + intv, err := strconv.ParseInt(plain, 0, 64) + if err == nil { + if intv == int64(int(intv)) { + return yaml_INT_TAG, int(intv) + } else { + return yaml_INT_TAG, intv + } + } + uintv, err := strconv.ParseUint(plain, 0, 64) + if err == nil { + return yaml_INT_TAG, uintv + } + if yamlStyleFloat.MatchString(plain) { + floatv, err := strconv.ParseFloat(plain, 64) + if err == nil { + return yaml_FLOAT_TAG, floatv + } + } + if strings.HasPrefix(plain, "0b") { + intv, err := strconv.ParseInt(plain[2:], 2, 64) + if err == nil { + if intv == int64(int(intv)) { + return yaml_INT_TAG, int(intv) + } else { + return yaml_INT_TAG, intv + } + } + uintv, err := strconv.ParseUint(plain[2:], 2, 64) + if err == nil { + return yaml_INT_TAG, uintv + } + } else if strings.HasPrefix(plain, "-0b") { + intv, err := strconv.ParseInt(plain[3:], 2, 64) + if err == nil { + if intv == int64(int(intv)) { + return yaml_INT_TAG, -int(intv) + } else { + return yaml_INT_TAG, -intv + } + } + } + // XXX Handle timestamps here. + + default: + panic("resolveTable item not yet handled: " + string(rune(hint)) + " (with " + in + ")") + } + } + if tag == yaml_BINARY_TAG { + return yaml_BINARY_TAG, in + } + if utf8.ValidString(in) { + return yaml_STR_TAG, in + } + return yaml_BINARY_TAG, encodeBase64(in) +} + +// encodeBase64 encodes s as base64 that is broken up into multiple lines +// as appropriate for the resulting length. +func encodeBase64(s string) string { + const lineLen = 70 + encLen := base64.StdEncoding.EncodedLen(len(s)) + lines := encLen/lineLen + 1 + buf := make([]byte, encLen*2+lines) + in := buf[0:encLen] + out := buf[encLen:] + base64.StdEncoding.Encode(in, []byte(s)) + k := 0 + for i := 0; i < len(in); i += lineLen { + j := i + lineLen + if j > len(in) { + j = len(in) + } + k += copy(out[k:], in[i:j]) + if lines > 1 { + out[k] = '\n' + k++ + } + } + return string(out[:k]) +} diff --git a/generator/vendor/gopkg.in/yaml.v2/scannerc.go b/generator/vendor/gopkg.in/yaml.v2/scannerc.go new file mode 100644 index 00000000..07448445 --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/scannerc.go @@ -0,0 +1,2711 @@ +package yaml + +import ( + "bytes" + "fmt" +) + +// Introduction +// ************ +// +// The following notes assume that you are familiar with the YAML specification +// (http://yaml.org/spec/1.2/spec.html). We mostly follow it, although in +// some cases we are less restrictive that it requires. +// +// The process of transforming a YAML stream into a sequence of events is +// divided on two steps: Scanning and Parsing. +// +// The Scanner transforms the input stream into a sequence of tokens, while the +// parser transform the sequence of tokens produced by the Scanner into a +// sequence of parsing events. +// +// The Scanner is rather clever and complicated. The Parser, on the contrary, +// is a straightforward implementation of a recursive-descendant parser (or, +// LL(1) parser, as it is usually called). +// +// Actually there are two issues of Scanning that might be called "clever", the +// rest is quite straightforward. The issues are "block collection start" and +// "simple keys". Both issues are explained below in details. +// +// Here the Scanning step is explained and implemented. We start with the list +// of all the tokens produced by the Scanner together with short descriptions. +// +// Now, tokens: +// +// STREAM-START(encoding) # The stream start. +// STREAM-END # The stream end. +// VERSION-DIRECTIVE(major,minor) # The '%YAML' directive. +// TAG-DIRECTIVE(handle,prefix) # The '%TAG' directive. +// DOCUMENT-START # '---' +// DOCUMENT-END # '...' +// BLOCK-SEQUENCE-START # Indentation increase denoting a block +// BLOCK-MAPPING-START # sequence or a block mapping. +// BLOCK-END # Indentation decrease. +// FLOW-SEQUENCE-START # '[' +// FLOW-SEQUENCE-END # ']' +// BLOCK-SEQUENCE-START # '{' +// BLOCK-SEQUENCE-END # '}' +// BLOCK-ENTRY # '-' +// FLOW-ENTRY # ',' +// KEY # '?' or nothing (simple keys). +// VALUE # ':' +// ALIAS(anchor) # '*anchor' +// ANCHOR(anchor) # '&anchor' +// TAG(handle,suffix) # '!handle!suffix' +// SCALAR(value,style) # A scalar. +// +// The following two tokens are "virtual" tokens denoting the beginning and the +// end of the stream: +// +// STREAM-START(encoding) +// STREAM-END +// +// We pass the information about the input stream encoding with the +// STREAM-START token. +// +// The next two tokens are responsible for tags: +// +// VERSION-DIRECTIVE(major,minor) +// TAG-DIRECTIVE(handle,prefix) +// +// Example: +// +// %YAML 1.1 +// %TAG ! !foo +// %TAG !yaml! tag:yaml.org,2002: +// --- +// +// The correspoding sequence of tokens: +// +// STREAM-START(utf-8) +// VERSION-DIRECTIVE(1,1) +// TAG-DIRECTIVE("!","!foo") +// TAG-DIRECTIVE("!yaml","tag:yaml.org,2002:") +// DOCUMENT-START +// STREAM-END +// +// Note that the VERSION-DIRECTIVE and TAG-DIRECTIVE tokens occupy a whole +// line. +// +// The document start and end indicators are represented by: +// +// DOCUMENT-START +// DOCUMENT-END +// +// Note that if a YAML stream contains an implicit document (without '---' +// and '...' indicators), no DOCUMENT-START and DOCUMENT-END tokens will be +// produced. +// +// In the following examples, we present whole documents together with the +// produced tokens. +// +// 1. An implicit document: +// +// 'a scalar' +// +// Tokens: +// +// STREAM-START(utf-8) +// SCALAR("a scalar",single-quoted) +// STREAM-END +// +// 2. An explicit document: +// +// --- +// 'a scalar' +// ... +// +// Tokens: +// +// STREAM-START(utf-8) +// DOCUMENT-START +// SCALAR("a scalar",single-quoted) +// DOCUMENT-END +// STREAM-END +// +// 3. Several documents in a stream: +// +// 'a scalar' +// --- +// 'another scalar' +// --- +// 'yet another scalar' +// +// Tokens: +// +// STREAM-START(utf-8) +// SCALAR("a scalar",single-quoted) +// DOCUMENT-START +// SCALAR("another scalar",single-quoted) +// DOCUMENT-START +// SCALAR("yet another scalar",single-quoted) +// STREAM-END +// +// We have already introduced the SCALAR token above. The following tokens are +// used to describe aliases, anchors, tag, and scalars: +// +// ALIAS(anchor) +// ANCHOR(anchor) +// TAG(handle,suffix) +// SCALAR(value,style) +// +// The following series of examples illustrate the usage of these tokens: +// +// 1. A recursive sequence: +// +// &A [ *A ] +// +// Tokens: +// +// STREAM-START(utf-8) +// ANCHOR("A") +// FLOW-SEQUENCE-START +// ALIAS("A") +// FLOW-SEQUENCE-END +// STREAM-END +// +// 2. A tagged scalar: +// +// !!float "3.14" # A good approximation. +// +// Tokens: +// +// STREAM-START(utf-8) +// TAG("!!","float") +// SCALAR("3.14",double-quoted) +// STREAM-END +// +// 3. Various scalar styles: +// +// --- # Implicit empty plain scalars do not produce tokens. +// --- a plain scalar +// --- 'a single-quoted scalar' +// --- "a double-quoted scalar" +// --- |- +// a literal scalar +// --- >- +// a folded +// scalar +// +// Tokens: +// +// STREAM-START(utf-8) +// DOCUMENT-START +// DOCUMENT-START +// SCALAR("a plain scalar",plain) +// DOCUMENT-START +// SCALAR("a single-quoted scalar",single-quoted) +// DOCUMENT-START +// SCALAR("a double-quoted scalar",double-quoted) +// DOCUMENT-START +// SCALAR("a literal scalar",literal) +// DOCUMENT-START +// SCALAR("a folded scalar",folded) +// STREAM-END +// +// Now it's time to review collection-related tokens. We will start with +// flow collections: +// +// FLOW-SEQUENCE-START +// FLOW-SEQUENCE-END +// FLOW-MAPPING-START +// FLOW-MAPPING-END +// FLOW-ENTRY +// KEY +// VALUE +// +// The tokens FLOW-SEQUENCE-START, FLOW-SEQUENCE-END, FLOW-MAPPING-START, and +// FLOW-MAPPING-END represent the indicators '[', ']', '{', and '}' +// correspondingly. FLOW-ENTRY represent the ',' indicator. Finally the +// indicators '?' and ':', which are used for denoting mapping keys and values, +// are represented by the KEY and VALUE tokens. +// +// The following examples show flow collections: +// +// 1. A flow sequence: +// +// [item 1, item 2, item 3] +// +// Tokens: +// +// STREAM-START(utf-8) +// FLOW-SEQUENCE-START +// SCALAR("item 1",plain) +// FLOW-ENTRY +// SCALAR("item 2",plain) +// FLOW-ENTRY +// SCALAR("item 3",plain) +// FLOW-SEQUENCE-END +// STREAM-END +// +// 2. A flow mapping: +// +// { +// a simple key: a value, # Note that the KEY token is produced. +// ? a complex key: another value, +// } +// +// Tokens: +// +// STREAM-START(utf-8) +// FLOW-MAPPING-START +// KEY +// SCALAR("a simple key",plain) +// VALUE +// SCALAR("a value",plain) +// FLOW-ENTRY +// KEY +// SCALAR("a complex key",plain) +// VALUE +// SCALAR("another value",plain) +// FLOW-ENTRY +// FLOW-MAPPING-END +// STREAM-END +// +// A simple key is a key which is not denoted by the '?' indicator. Note that +// the Scanner still produce the KEY token whenever it encounters a simple key. +// +// For scanning block collections, the following tokens are used (note that we +// repeat KEY and VALUE here): +// +// BLOCK-SEQUENCE-START +// BLOCK-MAPPING-START +// BLOCK-END +// BLOCK-ENTRY +// KEY +// VALUE +// +// The tokens BLOCK-SEQUENCE-START and BLOCK-MAPPING-START denote indentation +// increase that precedes a block collection (cf. the INDENT token in Python). +// The token BLOCK-END denote indentation decrease that ends a block collection +// (cf. the DEDENT token in Python). However YAML has some syntax pecularities +// that makes detections of these tokens more complex. +// +// The tokens BLOCK-ENTRY, KEY, and VALUE are used to represent the indicators +// '-', '?', and ':' correspondingly. +// +// The following examples show how the tokens BLOCK-SEQUENCE-START, +// BLOCK-MAPPING-START, and BLOCK-END are emitted by the Scanner: +// +// 1. Block sequences: +// +// - item 1 +// - item 2 +// - +// - item 3.1 +// - item 3.2 +// - +// key 1: value 1 +// key 2: value 2 +// +// Tokens: +// +// STREAM-START(utf-8) +// BLOCK-SEQUENCE-START +// BLOCK-ENTRY +// SCALAR("item 1",plain) +// BLOCK-ENTRY +// SCALAR("item 2",plain) +// BLOCK-ENTRY +// BLOCK-SEQUENCE-START +// BLOCK-ENTRY +// SCALAR("item 3.1",plain) +// BLOCK-ENTRY +// SCALAR("item 3.2",plain) +// BLOCK-END +// BLOCK-ENTRY +// BLOCK-MAPPING-START +// KEY +// SCALAR("key 1",plain) +// VALUE +// SCALAR("value 1",plain) +// KEY +// SCALAR("key 2",plain) +// VALUE +// SCALAR("value 2",plain) +// BLOCK-END +// BLOCK-END +// STREAM-END +// +// 2. Block mappings: +// +// a simple key: a value # The KEY token is produced here. +// ? a complex key +// : another value +// a mapping: +// key 1: value 1 +// key 2: value 2 +// a sequence: +// - item 1 +// - item 2 +// +// Tokens: +// +// STREAM-START(utf-8) +// BLOCK-MAPPING-START +// KEY +// SCALAR("a simple key",plain) +// VALUE +// SCALAR("a value",plain) +// KEY +// SCALAR("a complex key",plain) +// VALUE +// SCALAR("another value",plain) +// KEY +// SCALAR("a mapping",plain) +// BLOCK-MAPPING-START +// KEY +// SCALAR("key 1",plain) +// VALUE +// SCALAR("value 1",plain) +// KEY +// SCALAR("key 2",plain) +// VALUE +// SCALAR("value 2",plain) +// BLOCK-END +// KEY +// SCALAR("a sequence",plain) +// VALUE +// BLOCK-SEQUENCE-START +// BLOCK-ENTRY +// SCALAR("item 1",plain) +// BLOCK-ENTRY +// SCALAR("item 2",plain) +// BLOCK-END +// BLOCK-END +// STREAM-END +// +// YAML does not always require to start a new block collection from a new +// line. If the current line contains only '-', '?', and ':' indicators, a new +// block collection may start at the current line. The following examples +// illustrate this case: +// +// 1. Collections in a sequence: +// +// - - item 1 +// - item 2 +// - key 1: value 1 +// key 2: value 2 +// - ? complex key +// : complex value +// +// Tokens: +// +// STREAM-START(utf-8) +// BLOCK-SEQUENCE-START +// BLOCK-ENTRY +// BLOCK-SEQUENCE-START +// BLOCK-ENTRY +// SCALAR("item 1",plain) +// BLOCK-ENTRY +// SCALAR("item 2",plain) +// BLOCK-END +// BLOCK-ENTRY +// BLOCK-MAPPING-START +// KEY +// SCALAR("key 1",plain) +// VALUE +// SCALAR("value 1",plain) +// KEY +// SCALAR("key 2",plain) +// VALUE +// SCALAR("value 2",plain) +// BLOCK-END +// BLOCK-ENTRY +// BLOCK-MAPPING-START +// KEY +// SCALAR("complex key") +// VALUE +// SCALAR("complex value") +// BLOCK-END +// BLOCK-END +// STREAM-END +// +// 2. Collections in a mapping: +// +// ? a sequence +// : - item 1 +// - item 2 +// ? a mapping +// : key 1: value 1 +// key 2: value 2 +// +// Tokens: +// +// STREAM-START(utf-8) +// BLOCK-MAPPING-START +// KEY +// SCALAR("a sequence",plain) +// VALUE +// BLOCK-SEQUENCE-START +// BLOCK-ENTRY +// SCALAR("item 1",plain) +// BLOCK-ENTRY +// SCALAR("item 2",plain) +// BLOCK-END +// KEY +// SCALAR("a mapping",plain) +// VALUE +// BLOCK-MAPPING-START +// KEY +// SCALAR("key 1",plain) +// VALUE +// SCALAR("value 1",plain) +// KEY +// SCALAR("key 2",plain) +// VALUE +// SCALAR("value 2",plain) +// BLOCK-END +// BLOCK-END +// STREAM-END +// +// YAML also permits non-indented sequences if they are included into a block +// mapping. In this case, the token BLOCK-SEQUENCE-START is not produced: +// +// key: +// - item 1 # BLOCK-SEQUENCE-START is NOT produced here. +// - item 2 +// +// Tokens: +// +// STREAM-START(utf-8) +// BLOCK-MAPPING-START +// KEY +// SCALAR("key",plain) +// VALUE +// BLOCK-ENTRY +// SCALAR("item 1",plain) +// BLOCK-ENTRY +// SCALAR("item 2",plain) +// BLOCK-END +// + +// Ensure that the buffer contains the required number of characters. +// Return true on success, false on failure (reader error or memory error). +func cache(parser *yaml_parser_t, length int) bool { + // [Go] This was inlined: !cache(A, B) -> unread < B && !update(A, B) + return parser.unread >= length || yaml_parser_update_buffer(parser, length) +} + +// Advance the buffer pointer. +func skip(parser *yaml_parser_t) { + parser.mark.index++ + parser.mark.column++ + parser.unread-- + parser.buffer_pos += width(parser.buffer[parser.buffer_pos]) +} + +func skip_line(parser *yaml_parser_t) { + if is_crlf(parser.buffer, parser.buffer_pos) { + parser.mark.index += 2 + parser.mark.column = 0 + parser.mark.line++ + parser.unread -= 2 + parser.buffer_pos += 2 + } else if is_break(parser.buffer, parser.buffer_pos) { + parser.mark.index++ + parser.mark.column = 0 + parser.mark.line++ + parser.unread-- + parser.buffer_pos += width(parser.buffer[parser.buffer_pos]) + } +} + +// Copy a character to a string buffer and advance pointers. +func read(parser *yaml_parser_t, s []byte) []byte { + w := width(parser.buffer[parser.buffer_pos]) + if w == 0 { + panic("invalid character sequence") + } + if len(s) == 0 { + s = make([]byte, 0, 32) + } + if w == 1 && len(s)+w <= cap(s) { + s = s[:len(s)+1] + s[len(s)-1] = parser.buffer[parser.buffer_pos] + parser.buffer_pos++ + } else { + s = append(s, parser.buffer[parser.buffer_pos:parser.buffer_pos+w]...) + parser.buffer_pos += w + } + parser.mark.index++ + parser.mark.column++ + parser.unread-- + return s +} + +// Copy a line break character to a string buffer and advance pointers. +func read_line(parser *yaml_parser_t, s []byte) []byte { + buf := parser.buffer + pos := parser.buffer_pos + switch { + case buf[pos] == '\r' && buf[pos+1] == '\n': + // CR LF . LF + s = append(s, '\n') + parser.buffer_pos += 2 + parser.mark.index++ + parser.unread-- + case buf[pos] == '\r' || buf[pos] == '\n': + // CR|LF . LF + s = append(s, '\n') + parser.buffer_pos += 1 + case buf[pos] == '\xC2' && buf[pos+1] == '\x85': + // NEL . LF + s = append(s, '\n') + parser.buffer_pos += 2 + case buf[pos] == '\xE2' && buf[pos+1] == '\x80' && (buf[pos+2] == '\xA8' || buf[pos+2] == '\xA9'): + // LS|PS . LS|PS + s = append(s, buf[parser.buffer_pos:pos+3]...) + parser.buffer_pos += 3 + default: + return s + } + parser.mark.index++ + parser.mark.column = 0 + parser.mark.line++ + parser.unread-- + return s +} + +// Get the next token. +func yaml_parser_scan(parser *yaml_parser_t, token *yaml_token_t) bool { + // Erase the token object. + *token = yaml_token_t{} // [Go] Is this necessary? + + // No tokens after STREAM-END or error. + if parser.stream_end_produced || parser.error != yaml_NO_ERROR { + return true + } + + // Ensure that the tokens queue contains enough tokens. + if !parser.token_available { + if !yaml_parser_fetch_more_tokens(parser) { + return false + } + } + + // Fetch the next token from the queue. + *token = parser.tokens[parser.tokens_head] + parser.tokens_head++ + parser.tokens_parsed++ + parser.token_available = false + + if token.typ == yaml_STREAM_END_TOKEN { + parser.stream_end_produced = true + } + return true +} + +// Set the scanner error and return false. +func yaml_parser_set_scanner_error(parser *yaml_parser_t, context string, context_mark yaml_mark_t, problem string) bool { + parser.error = yaml_SCANNER_ERROR + parser.context = context + parser.context_mark = context_mark + parser.problem = problem + parser.problem_mark = parser.mark + return false +} + +func yaml_parser_set_scanner_tag_error(parser *yaml_parser_t, directive bool, context_mark yaml_mark_t, problem string) bool { + context := "while parsing a tag" + if directive { + context = "while parsing a %TAG directive" + } + return yaml_parser_set_scanner_error(parser, context, context_mark, problem) +} + +func trace(args ...interface{}) func() { + pargs := append([]interface{}{"+++"}, args...) + fmt.Println(pargs...) + pargs = append([]interface{}{"---"}, args...) + return func() { fmt.Println(pargs...) } +} + +// Ensure that the tokens queue contains at least one token which can be +// returned to the Parser. +func yaml_parser_fetch_more_tokens(parser *yaml_parser_t) bool { + // While we need more tokens to fetch, do it. + for { + // Check if we really need to fetch more tokens. + need_more_tokens := false + + if parser.tokens_head == len(parser.tokens) { + // Queue is empty. + need_more_tokens = true + } else { + // Check if any potential simple key may occupy the head position. + if !yaml_parser_stale_simple_keys(parser) { + return false + } + + for i := range parser.simple_keys { + simple_key := &parser.simple_keys[i] + if simple_key.possible && simple_key.token_number == parser.tokens_parsed { + need_more_tokens = true + break + } + } + } + + // We are finished. + if !need_more_tokens { + break + } + // Fetch the next token. + if !yaml_parser_fetch_next_token(parser) { + return false + } + } + + parser.token_available = true + return true +} + +// The dispatcher for token fetchers. +func yaml_parser_fetch_next_token(parser *yaml_parser_t) bool { + // Ensure that the buffer is initialized. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + + // Check if we just started scanning. Fetch STREAM-START then. + if !parser.stream_start_produced { + return yaml_parser_fetch_stream_start(parser) + } + + // Eat whitespaces and comments until we reach the next token. + if !yaml_parser_scan_to_next_token(parser) { + return false + } + + // Remove obsolete potential simple keys. + if !yaml_parser_stale_simple_keys(parser) { + return false + } + + // Check the indentation level against the current column. + if !yaml_parser_unroll_indent(parser, parser.mark.column) { + return false + } + + // Ensure that the buffer contains at least 4 characters. 4 is the length + // of the longest indicators ('--- ' and '... '). + if parser.unread < 4 && !yaml_parser_update_buffer(parser, 4) { + return false + } + + // Is it the end of the stream? + if is_z(parser.buffer, parser.buffer_pos) { + return yaml_parser_fetch_stream_end(parser) + } + + // Is it a directive? + if parser.mark.column == 0 && parser.buffer[parser.buffer_pos] == '%' { + return yaml_parser_fetch_directive(parser) + } + + buf := parser.buffer + pos := parser.buffer_pos + + // Is it the document start indicator? + if parser.mark.column == 0 && buf[pos] == '-' && buf[pos+1] == '-' && buf[pos+2] == '-' && is_blankz(buf, pos+3) { + return yaml_parser_fetch_document_indicator(parser, yaml_DOCUMENT_START_TOKEN) + } + + // Is it the document end indicator? + if parser.mark.column == 0 && buf[pos] == '.' && buf[pos+1] == '.' && buf[pos+2] == '.' && is_blankz(buf, pos+3) { + return yaml_parser_fetch_document_indicator(parser, yaml_DOCUMENT_END_TOKEN) + } + + // Is it the flow sequence start indicator? + if buf[pos] == '[' { + return yaml_parser_fetch_flow_collection_start(parser, yaml_FLOW_SEQUENCE_START_TOKEN) + } + + // Is it the flow mapping start indicator? + if parser.buffer[parser.buffer_pos] == '{' { + return yaml_parser_fetch_flow_collection_start(parser, yaml_FLOW_MAPPING_START_TOKEN) + } + + // Is it the flow sequence end indicator? + if parser.buffer[parser.buffer_pos] == ']' { + return yaml_parser_fetch_flow_collection_end(parser, + yaml_FLOW_SEQUENCE_END_TOKEN) + } + + // Is it the flow mapping end indicator? + if parser.buffer[parser.buffer_pos] == '}' { + return yaml_parser_fetch_flow_collection_end(parser, + yaml_FLOW_MAPPING_END_TOKEN) + } + + // Is it the flow entry indicator? + if parser.buffer[parser.buffer_pos] == ',' { + return yaml_parser_fetch_flow_entry(parser) + } + + // Is it the block entry indicator? + if parser.buffer[parser.buffer_pos] == '-' && is_blankz(parser.buffer, parser.buffer_pos+1) { + return yaml_parser_fetch_block_entry(parser) + } + + // Is it the key indicator? + if parser.buffer[parser.buffer_pos] == '?' && (parser.flow_level > 0 || is_blankz(parser.buffer, parser.buffer_pos+1)) { + return yaml_parser_fetch_key(parser) + } + + // Is it the value indicator? + if parser.buffer[parser.buffer_pos] == ':' && (parser.flow_level > 0 || is_blankz(parser.buffer, parser.buffer_pos+1)) { + return yaml_parser_fetch_value(parser) + } + + // Is it an alias? + if parser.buffer[parser.buffer_pos] == '*' { + return yaml_parser_fetch_anchor(parser, yaml_ALIAS_TOKEN) + } + + // Is it an anchor? + if parser.buffer[parser.buffer_pos] == '&' { + return yaml_parser_fetch_anchor(parser, yaml_ANCHOR_TOKEN) + } + + // Is it a tag? + if parser.buffer[parser.buffer_pos] == '!' { + return yaml_parser_fetch_tag(parser) + } + + // Is it a literal scalar? + if parser.buffer[parser.buffer_pos] == '|' && parser.flow_level == 0 { + return yaml_parser_fetch_block_scalar(parser, true) + } + + // Is it a folded scalar? + if parser.buffer[parser.buffer_pos] == '>' && parser.flow_level == 0 { + return yaml_parser_fetch_block_scalar(parser, false) + } + + // Is it a single-quoted scalar? + if parser.buffer[parser.buffer_pos] == '\'' { + return yaml_parser_fetch_flow_scalar(parser, true) + } + + // Is it a double-quoted scalar? + if parser.buffer[parser.buffer_pos] == '"' { + return yaml_parser_fetch_flow_scalar(parser, false) + } + + // Is it a plain scalar? + // + // A plain scalar may start with any non-blank characters except + // + // '-', '?', ':', ',', '[', ']', '{', '}', + // '#', '&', '*', '!', '|', '>', '\'', '\"', + // '%', '@', '`'. + // + // In the block context (and, for the '-' indicator, in the flow context + // too), it may also start with the characters + // + // '-', '?', ':' + // + // if it is followed by a non-space character. + // + // The last rule is more restrictive than the specification requires. + // [Go] Make this logic more reasonable. + //switch parser.buffer[parser.buffer_pos] { + //case '-', '?', ':', ',', '?', '-', ',', ':', ']', '[', '}', '{', '&', '#', '!', '*', '>', '|', '"', '\'', '@', '%', '-', '`': + //} + if !(is_blankz(parser.buffer, parser.buffer_pos) || parser.buffer[parser.buffer_pos] == '-' || + parser.buffer[parser.buffer_pos] == '?' || parser.buffer[parser.buffer_pos] == ':' || + parser.buffer[parser.buffer_pos] == ',' || parser.buffer[parser.buffer_pos] == '[' || + parser.buffer[parser.buffer_pos] == ']' || parser.buffer[parser.buffer_pos] == '{' || + parser.buffer[parser.buffer_pos] == '}' || parser.buffer[parser.buffer_pos] == '#' || + parser.buffer[parser.buffer_pos] == '&' || parser.buffer[parser.buffer_pos] == '*' || + parser.buffer[parser.buffer_pos] == '!' || parser.buffer[parser.buffer_pos] == '|' || + parser.buffer[parser.buffer_pos] == '>' || parser.buffer[parser.buffer_pos] == '\'' || + parser.buffer[parser.buffer_pos] == '"' || parser.buffer[parser.buffer_pos] == '%' || + parser.buffer[parser.buffer_pos] == '@' || parser.buffer[parser.buffer_pos] == '`') || + (parser.buffer[parser.buffer_pos] == '-' && !is_blank(parser.buffer, parser.buffer_pos+1)) || + (parser.flow_level == 0 && + (parser.buffer[parser.buffer_pos] == '?' || parser.buffer[parser.buffer_pos] == ':') && + !is_blankz(parser.buffer, parser.buffer_pos+1)) { + return yaml_parser_fetch_plain_scalar(parser) + } + + // If we don't determine the token type so far, it is an error. + return yaml_parser_set_scanner_error(parser, + "while scanning for the next token", parser.mark, + "found character that cannot start any token") +} + +// Check the list of potential simple keys and remove the positions that +// cannot contain simple keys anymore. +func yaml_parser_stale_simple_keys(parser *yaml_parser_t) bool { + // Check for a potential simple key for each flow level. + for i := range parser.simple_keys { + simple_key := &parser.simple_keys[i] + + // The specification requires that a simple key + // + // - is limited to a single line, + // - is shorter than 1024 characters. + if simple_key.possible && (simple_key.mark.line < parser.mark.line || simple_key.mark.index+1024 < parser.mark.index) { + + // Check if the potential simple key to be removed is required. + if simple_key.required { + return yaml_parser_set_scanner_error(parser, + "while scanning a simple key", simple_key.mark, + "could not find expected ':'") + } + simple_key.possible = false + } + } + return true +} + +// Check if a simple key may start at the current position and add it if +// needed. +func yaml_parser_save_simple_key(parser *yaml_parser_t) bool { + // A simple key is required at the current position if the scanner is in + // the block context and the current column coincides with the indentation + // level. + + required := parser.flow_level == 0 && parser.indent == parser.mark.column + + // A simple key is required only when it is the first token in the current + // line. Therefore it is always allowed. But we add a check anyway. + if required && !parser.simple_key_allowed { + panic("should not happen") + } + + // + // If the current position may start a simple key, save it. + // + if parser.simple_key_allowed { + simple_key := yaml_simple_key_t{ + possible: true, + required: required, + token_number: parser.tokens_parsed + (len(parser.tokens) - parser.tokens_head), + } + simple_key.mark = parser.mark + + if !yaml_parser_remove_simple_key(parser) { + return false + } + parser.simple_keys[len(parser.simple_keys)-1] = simple_key + } + return true +} + +// Remove a potential simple key at the current flow level. +func yaml_parser_remove_simple_key(parser *yaml_parser_t) bool { + i := len(parser.simple_keys) - 1 + if parser.simple_keys[i].possible { + // If the key is required, it is an error. + if parser.simple_keys[i].required { + return yaml_parser_set_scanner_error(parser, + "while scanning a simple key", parser.simple_keys[i].mark, + "could not find expected ':'") + } + } + // Remove the key from the stack. + parser.simple_keys[i].possible = false + return true +} + +// Increase the flow level and resize the simple key list if needed. +func yaml_parser_increase_flow_level(parser *yaml_parser_t) bool { + // Reset the simple key on the next level. + parser.simple_keys = append(parser.simple_keys, yaml_simple_key_t{}) + + // Increase the flow level. + parser.flow_level++ + return true +} + +// Decrease the flow level. +func yaml_parser_decrease_flow_level(parser *yaml_parser_t) bool { + if parser.flow_level > 0 { + parser.flow_level-- + parser.simple_keys = parser.simple_keys[:len(parser.simple_keys)-1] + } + return true +} + +// Push the current indentation level to the stack and set the new level +// the current column is greater than the indentation level. In this case, +// append or insert the specified token into the token queue. +func yaml_parser_roll_indent(parser *yaml_parser_t, column, number int, typ yaml_token_type_t, mark yaml_mark_t) bool { + // In the flow context, do nothing. + if parser.flow_level > 0 { + return true + } + + if parser.indent < column { + // Push the current indentation level to the stack and set the new + // indentation level. + parser.indents = append(parser.indents, parser.indent) + parser.indent = column + + // Create a token and insert it into the queue. + token := yaml_token_t{ + typ: typ, + start_mark: mark, + end_mark: mark, + } + if number > -1 { + number -= parser.tokens_parsed + } + yaml_insert_token(parser, number, &token) + } + return true +} + +// Pop indentation levels from the indents stack until the current level +// becomes less or equal to the column. For each indentation level, append +// the BLOCK-END token. +func yaml_parser_unroll_indent(parser *yaml_parser_t, column int) bool { + // In the flow context, do nothing. + if parser.flow_level > 0 { + return true + } + + // Loop through the indentation levels in the stack. + for parser.indent > column { + // Create a token and append it to the queue. + token := yaml_token_t{ + typ: yaml_BLOCK_END_TOKEN, + start_mark: parser.mark, + end_mark: parser.mark, + } + yaml_insert_token(parser, -1, &token) + + // Pop the indentation level. + parser.indent = parser.indents[len(parser.indents)-1] + parser.indents = parser.indents[:len(parser.indents)-1] + } + return true +} + +// Initialize the scanner and produce the STREAM-START token. +func yaml_parser_fetch_stream_start(parser *yaml_parser_t) bool { + + // Set the initial indentation. + parser.indent = -1 + + // Initialize the simple key stack. + parser.simple_keys = append(parser.simple_keys, yaml_simple_key_t{}) + + // A simple key is allowed at the beginning of the stream. + parser.simple_key_allowed = true + + // We have started. + parser.stream_start_produced = true + + // Create the STREAM-START token and append it to the queue. + token := yaml_token_t{ + typ: yaml_STREAM_START_TOKEN, + start_mark: parser.mark, + end_mark: parser.mark, + encoding: parser.encoding, + } + yaml_insert_token(parser, -1, &token) + return true +} + +// Produce the STREAM-END token and shut down the scanner. +func yaml_parser_fetch_stream_end(parser *yaml_parser_t) bool { + + // Force new line. + if parser.mark.column != 0 { + parser.mark.column = 0 + parser.mark.line++ + } + + // Reset the indentation level. + if !yaml_parser_unroll_indent(parser, -1) { + return false + } + + // Reset simple keys. + if !yaml_parser_remove_simple_key(parser) { + return false + } + + parser.simple_key_allowed = false + + // Create the STREAM-END token and append it to the queue. + token := yaml_token_t{ + typ: yaml_STREAM_END_TOKEN, + start_mark: parser.mark, + end_mark: parser.mark, + } + yaml_insert_token(parser, -1, &token) + return true +} + +// Produce a VERSION-DIRECTIVE or TAG-DIRECTIVE token. +func yaml_parser_fetch_directive(parser *yaml_parser_t) bool { + // Reset the indentation level. + if !yaml_parser_unroll_indent(parser, -1) { + return false + } + + // Reset simple keys. + if !yaml_parser_remove_simple_key(parser) { + return false + } + + parser.simple_key_allowed = false + + // Create the YAML-DIRECTIVE or TAG-DIRECTIVE token. + token := yaml_token_t{} + if !yaml_parser_scan_directive(parser, &token) { + return false + } + // Append the token to the queue. + yaml_insert_token(parser, -1, &token) + return true +} + +// Produce the DOCUMENT-START or DOCUMENT-END token. +func yaml_parser_fetch_document_indicator(parser *yaml_parser_t, typ yaml_token_type_t) bool { + // Reset the indentation level. + if !yaml_parser_unroll_indent(parser, -1) { + return false + } + + // Reset simple keys. + if !yaml_parser_remove_simple_key(parser) { + return false + } + + parser.simple_key_allowed = false + + // Consume the token. + start_mark := parser.mark + + skip(parser) + skip(parser) + skip(parser) + + end_mark := parser.mark + + // Create the DOCUMENT-START or DOCUMENT-END token. + token := yaml_token_t{ + typ: typ, + start_mark: start_mark, + end_mark: end_mark, + } + // Append the token to the queue. + yaml_insert_token(parser, -1, &token) + return true +} + +// Produce the FLOW-SEQUENCE-START or FLOW-MAPPING-START token. +func yaml_parser_fetch_flow_collection_start(parser *yaml_parser_t, typ yaml_token_type_t) bool { + // The indicators '[' and '{' may start a simple key. + if !yaml_parser_save_simple_key(parser) { + return false + } + + // Increase the flow level. + if !yaml_parser_increase_flow_level(parser) { + return false + } + + // A simple key may follow the indicators '[' and '{'. + parser.simple_key_allowed = true + + // Consume the token. + start_mark := parser.mark + skip(parser) + end_mark := parser.mark + + // Create the FLOW-SEQUENCE-START of FLOW-MAPPING-START token. + token := yaml_token_t{ + typ: typ, + start_mark: start_mark, + end_mark: end_mark, + } + // Append the token to the queue. + yaml_insert_token(parser, -1, &token) + return true +} + +// Produce the FLOW-SEQUENCE-END or FLOW-MAPPING-END token. +func yaml_parser_fetch_flow_collection_end(parser *yaml_parser_t, typ yaml_token_type_t) bool { + // Reset any potential simple key on the current flow level. + if !yaml_parser_remove_simple_key(parser) { + return false + } + + // Decrease the flow level. + if !yaml_parser_decrease_flow_level(parser) { + return false + } + + // No simple keys after the indicators ']' and '}'. + parser.simple_key_allowed = false + + // Consume the token. + + start_mark := parser.mark + skip(parser) + end_mark := parser.mark + + // Create the FLOW-SEQUENCE-END of FLOW-MAPPING-END token. + token := yaml_token_t{ + typ: typ, + start_mark: start_mark, + end_mark: end_mark, + } + // Append the token to the queue. + yaml_insert_token(parser, -1, &token) + return true +} + +// Produce the FLOW-ENTRY token. +func yaml_parser_fetch_flow_entry(parser *yaml_parser_t) bool { + // Reset any potential simple keys on the current flow level. + if !yaml_parser_remove_simple_key(parser) { + return false + } + + // Simple keys are allowed after ','. + parser.simple_key_allowed = true + + // Consume the token. + start_mark := parser.mark + skip(parser) + end_mark := parser.mark + + // Create the FLOW-ENTRY token and append it to the queue. + token := yaml_token_t{ + typ: yaml_FLOW_ENTRY_TOKEN, + start_mark: start_mark, + end_mark: end_mark, + } + yaml_insert_token(parser, -1, &token) + return true +} + +// Produce the BLOCK-ENTRY token. +func yaml_parser_fetch_block_entry(parser *yaml_parser_t) bool { + // Check if the scanner is in the block context. + if parser.flow_level == 0 { + // Check if we are allowed to start a new entry. + if !parser.simple_key_allowed { + return yaml_parser_set_scanner_error(parser, "", parser.mark, + "block sequence entries are not allowed in this context") + } + // Add the BLOCK-SEQUENCE-START token if needed. + if !yaml_parser_roll_indent(parser, parser.mark.column, -1, yaml_BLOCK_SEQUENCE_START_TOKEN, parser.mark) { + return false + } + } else { + // It is an error for the '-' indicator to occur in the flow context, + // but we let the Parser detect and report about it because the Parser + // is able to point to the context. + } + + // Reset any potential simple keys on the current flow level. + if !yaml_parser_remove_simple_key(parser) { + return false + } + + // Simple keys are allowed after '-'. + parser.simple_key_allowed = true + + // Consume the token. + start_mark := parser.mark + skip(parser) + end_mark := parser.mark + + // Create the BLOCK-ENTRY token and append it to the queue. + token := yaml_token_t{ + typ: yaml_BLOCK_ENTRY_TOKEN, + start_mark: start_mark, + end_mark: end_mark, + } + yaml_insert_token(parser, -1, &token) + return true +} + +// Produce the KEY token. +func yaml_parser_fetch_key(parser *yaml_parser_t) bool { + + // In the block context, additional checks are required. + if parser.flow_level == 0 { + // Check if we are allowed to start a new key (not nessesary simple). + if !parser.simple_key_allowed { + return yaml_parser_set_scanner_error(parser, "", parser.mark, + "mapping keys are not allowed in this context") + } + // Add the BLOCK-MAPPING-START token if needed. + if !yaml_parser_roll_indent(parser, parser.mark.column, -1, yaml_BLOCK_MAPPING_START_TOKEN, parser.mark) { + return false + } + } + + // Reset any potential simple keys on the current flow level. + if !yaml_parser_remove_simple_key(parser) { + return false + } + + // Simple keys are allowed after '?' in the block context. + parser.simple_key_allowed = parser.flow_level == 0 + + // Consume the token. + start_mark := parser.mark + skip(parser) + end_mark := parser.mark + + // Create the KEY token and append it to the queue. + token := yaml_token_t{ + typ: yaml_KEY_TOKEN, + start_mark: start_mark, + end_mark: end_mark, + } + yaml_insert_token(parser, -1, &token) + return true +} + +// Produce the VALUE token. +func yaml_parser_fetch_value(parser *yaml_parser_t) bool { + + simple_key := &parser.simple_keys[len(parser.simple_keys)-1] + + // Have we found a simple key? + if simple_key.possible { + // Create the KEY token and insert it into the queue. + token := yaml_token_t{ + typ: yaml_KEY_TOKEN, + start_mark: simple_key.mark, + end_mark: simple_key.mark, + } + yaml_insert_token(parser, simple_key.token_number-parser.tokens_parsed, &token) + + // In the block context, we may need to add the BLOCK-MAPPING-START token. + if !yaml_parser_roll_indent(parser, simple_key.mark.column, + simple_key.token_number, + yaml_BLOCK_MAPPING_START_TOKEN, simple_key.mark) { + return false + } + + // Remove the simple key. + simple_key.possible = false + + // A simple key cannot follow another simple key. + parser.simple_key_allowed = false + + } else { + // The ':' indicator follows a complex key. + + // In the block context, extra checks are required. + if parser.flow_level == 0 { + + // Check if we are allowed to start a complex value. + if !parser.simple_key_allowed { + return yaml_parser_set_scanner_error(parser, "", parser.mark, + "mapping values are not allowed in this context") + } + + // Add the BLOCK-MAPPING-START token if needed. + if !yaml_parser_roll_indent(parser, parser.mark.column, -1, yaml_BLOCK_MAPPING_START_TOKEN, parser.mark) { + return false + } + } + + // Simple keys after ':' are allowed in the block context. + parser.simple_key_allowed = parser.flow_level == 0 + } + + // Consume the token. + start_mark := parser.mark + skip(parser) + end_mark := parser.mark + + // Create the VALUE token and append it to the queue. + token := yaml_token_t{ + typ: yaml_VALUE_TOKEN, + start_mark: start_mark, + end_mark: end_mark, + } + yaml_insert_token(parser, -1, &token) + return true +} + +// Produce the ALIAS or ANCHOR token. +func yaml_parser_fetch_anchor(parser *yaml_parser_t, typ yaml_token_type_t) bool { + // An anchor or an alias could be a simple key. + if !yaml_parser_save_simple_key(parser) { + return false + } + + // A simple key cannot follow an anchor or an alias. + parser.simple_key_allowed = false + + // Create the ALIAS or ANCHOR token and append it to the queue. + var token yaml_token_t + if !yaml_parser_scan_anchor(parser, &token, typ) { + return false + } + yaml_insert_token(parser, -1, &token) + return true +} + +// Produce the TAG token. +func yaml_parser_fetch_tag(parser *yaml_parser_t) bool { + // A tag could be a simple key. + if !yaml_parser_save_simple_key(parser) { + return false + } + + // A simple key cannot follow a tag. + parser.simple_key_allowed = false + + // Create the TAG token and append it to the queue. + var token yaml_token_t + if !yaml_parser_scan_tag(parser, &token) { + return false + } + yaml_insert_token(parser, -1, &token) + return true +} + +// Produce the SCALAR(...,literal) or SCALAR(...,folded) tokens. +func yaml_parser_fetch_block_scalar(parser *yaml_parser_t, literal bool) bool { + // Remove any potential simple keys. + if !yaml_parser_remove_simple_key(parser) { + return false + } + + // A simple key may follow a block scalar. + parser.simple_key_allowed = true + + // Create the SCALAR token and append it to the queue. + var token yaml_token_t + if !yaml_parser_scan_block_scalar(parser, &token, literal) { + return false + } + yaml_insert_token(parser, -1, &token) + return true +} + +// Produce the SCALAR(...,single-quoted) or SCALAR(...,double-quoted) tokens. +func yaml_parser_fetch_flow_scalar(parser *yaml_parser_t, single bool) bool { + // A plain scalar could be a simple key. + if !yaml_parser_save_simple_key(parser) { + return false + } + + // A simple key cannot follow a flow scalar. + parser.simple_key_allowed = false + + // Create the SCALAR token and append it to the queue. + var token yaml_token_t + if !yaml_parser_scan_flow_scalar(parser, &token, single) { + return false + } + yaml_insert_token(parser, -1, &token) + return true +} + +// Produce the SCALAR(...,plain) token. +func yaml_parser_fetch_plain_scalar(parser *yaml_parser_t) bool { + // A plain scalar could be a simple key. + if !yaml_parser_save_simple_key(parser) { + return false + } + + // A simple key cannot follow a flow scalar. + parser.simple_key_allowed = false + + // Create the SCALAR token and append it to the queue. + var token yaml_token_t + if !yaml_parser_scan_plain_scalar(parser, &token) { + return false + } + yaml_insert_token(parser, -1, &token) + return true +} + +// Eat whitespaces and comments until the next token is found. +func yaml_parser_scan_to_next_token(parser *yaml_parser_t) bool { + + // Until the next token is not found. + for { + // Allow the BOM mark to start a line. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + if parser.mark.column == 0 && is_bom(parser.buffer, parser.buffer_pos) { + skip(parser) + } + + // Eat whitespaces. + // Tabs are allowed: + // - in the flow context + // - in the block context, but not at the beginning of the line or + // after '-', '?', or ':' (complex value). + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + + for parser.buffer[parser.buffer_pos] == ' ' || ((parser.flow_level > 0 || !parser.simple_key_allowed) && parser.buffer[parser.buffer_pos] == '\t') { + skip(parser) + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + + // Eat a comment until a line break. + if parser.buffer[parser.buffer_pos] == '#' { + for !is_breakz(parser.buffer, parser.buffer_pos) { + skip(parser) + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + } + + // If it is a line break, eat it. + if is_break(parser.buffer, parser.buffer_pos) { + if parser.unread < 2 && !yaml_parser_update_buffer(parser, 2) { + return false + } + skip_line(parser) + + // In the block context, a new line may start a simple key. + if parser.flow_level == 0 { + parser.simple_key_allowed = true + } + } else { + break // We have found a token. + } + } + + return true +} + +// Scan a YAML-DIRECTIVE or TAG-DIRECTIVE token. +// +// Scope: +// %YAML 1.1 # a comment \n +// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +// %TAG !yaml! tag:yaml.org,2002: \n +// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +// +func yaml_parser_scan_directive(parser *yaml_parser_t, token *yaml_token_t) bool { + // Eat '%'. + start_mark := parser.mark + skip(parser) + + // Scan the directive name. + var name []byte + if !yaml_parser_scan_directive_name(parser, start_mark, &name) { + return false + } + + // Is it a YAML directive? + if bytes.Equal(name, []byte("YAML")) { + // Scan the VERSION directive value. + var major, minor int8 + if !yaml_parser_scan_version_directive_value(parser, start_mark, &major, &minor) { + return false + } + end_mark := parser.mark + + // Create a VERSION-DIRECTIVE token. + *token = yaml_token_t{ + typ: yaml_VERSION_DIRECTIVE_TOKEN, + start_mark: start_mark, + end_mark: end_mark, + major: major, + minor: minor, + } + + // Is it a TAG directive? + } else if bytes.Equal(name, []byte("TAG")) { + // Scan the TAG directive value. + var handle, prefix []byte + if !yaml_parser_scan_tag_directive_value(parser, start_mark, &handle, &prefix) { + return false + } + end_mark := parser.mark + + // Create a TAG-DIRECTIVE token. + *token = yaml_token_t{ + typ: yaml_TAG_DIRECTIVE_TOKEN, + start_mark: start_mark, + end_mark: end_mark, + value: handle, + prefix: prefix, + } + + // Unknown directive. + } else { + yaml_parser_set_scanner_error(parser, "while scanning a directive", + start_mark, "found unknown directive name") + return false + } + + // Eat the rest of the line including any comments. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + + for is_blank(parser.buffer, parser.buffer_pos) { + skip(parser) + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + + if parser.buffer[parser.buffer_pos] == '#' { + for !is_breakz(parser.buffer, parser.buffer_pos) { + skip(parser) + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + } + + // Check if we are at the end of the line. + if !is_breakz(parser.buffer, parser.buffer_pos) { + yaml_parser_set_scanner_error(parser, "while scanning a directive", + start_mark, "did not find expected comment or line break") + return false + } + + // Eat a line break. + if is_break(parser.buffer, parser.buffer_pos) { + if parser.unread < 2 && !yaml_parser_update_buffer(parser, 2) { + return false + } + skip_line(parser) + } + + return true +} + +// Scan the directive name. +// +// Scope: +// %YAML 1.1 # a comment \n +// ^^^^ +// %TAG !yaml! tag:yaml.org,2002: \n +// ^^^ +// +func yaml_parser_scan_directive_name(parser *yaml_parser_t, start_mark yaml_mark_t, name *[]byte) bool { + // Consume the directive name. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + + var s []byte + for is_alpha(parser.buffer, parser.buffer_pos) { + s = read(parser, s) + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + + // Check if the name is empty. + if len(s) == 0 { + yaml_parser_set_scanner_error(parser, "while scanning a directive", + start_mark, "could not find expected directive name") + return false + } + + // Check for an blank character after the name. + if !is_blankz(parser.buffer, parser.buffer_pos) { + yaml_parser_set_scanner_error(parser, "while scanning a directive", + start_mark, "found unexpected non-alphabetical character") + return false + } + *name = s + return true +} + +// Scan the value of VERSION-DIRECTIVE. +// +// Scope: +// %YAML 1.1 # a comment \n +// ^^^^^^ +func yaml_parser_scan_version_directive_value(parser *yaml_parser_t, start_mark yaml_mark_t, major, minor *int8) bool { + // Eat whitespaces. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + for is_blank(parser.buffer, parser.buffer_pos) { + skip(parser) + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + + // Consume the major version number. + if !yaml_parser_scan_version_directive_number(parser, start_mark, major) { + return false + } + + // Eat '.'. + if parser.buffer[parser.buffer_pos] != '.' { + return yaml_parser_set_scanner_error(parser, "while scanning a %YAML directive", + start_mark, "did not find expected digit or '.' character") + } + + skip(parser) + + // Consume the minor version number. + if !yaml_parser_scan_version_directive_number(parser, start_mark, minor) { + return false + } + return true +} + +const max_number_length = 2 + +// Scan the version number of VERSION-DIRECTIVE. +// +// Scope: +// %YAML 1.1 # a comment \n +// ^ +// %YAML 1.1 # a comment \n +// ^ +func yaml_parser_scan_version_directive_number(parser *yaml_parser_t, start_mark yaml_mark_t, number *int8) bool { + + // Repeat while the next character is digit. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + var value, length int8 + for is_digit(parser.buffer, parser.buffer_pos) { + // Check if the number is too long. + length++ + if length > max_number_length { + return yaml_parser_set_scanner_error(parser, "while scanning a %YAML directive", + start_mark, "found extremely long version number") + } + value = value*10 + int8(as_digit(parser.buffer, parser.buffer_pos)) + skip(parser) + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + + // Check if the number was present. + if length == 0 { + return yaml_parser_set_scanner_error(parser, "while scanning a %YAML directive", + start_mark, "did not find expected version number") + } + *number = value + return true +} + +// Scan the value of a TAG-DIRECTIVE token. +// +// Scope: +// %TAG !yaml! tag:yaml.org,2002: \n +// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +// +func yaml_parser_scan_tag_directive_value(parser *yaml_parser_t, start_mark yaml_mark_t, handle, prefix *[]byte) bool { + var handle_value, prefix_value []byte + + // Eat whitespaces. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + + for is_blank(parser.buffer, parser.buffer_pos) { + skip(parser) + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + + // Scan a handle. + if !yaml_parser_scan_tag_handle(parser, true, start_mark, &handle_value) { + return false + } + + // Expect a whitespace. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + if !is_blank(parser.buffer, parser.buffer_pos) { + yaml_parser_set_scanner_error(parser, "while scanning a %TAG directive", + start_mark, "did not find expected whitespace") + return false + } + + // Eat whitespaces. + for is_blank(parser.buffer, parser.buffer_pos) { + skip(parser) + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + + // Scan a prefix. + if !yaml_parser_scan_tag_uri(parser, true, nil, start_mark, &prefix_value) { + return false + } + + // Expect a whitespace or line break. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + if !is_blankz(parser.buffer, parser.buffer_pos) { + yaml_parser_set_scanner_error(parser, "while scanning a %TAG directive", + start_mark, "did not find expected whitespace or line break") + return false + } + + *handle = handle_value + *prefix = prefix_value + return true +} + +func yaml_parser_scan_anchor(parser *yaml_parser_t, token *yaml_token_t, typ yaml_token_type_t) bool { + var s []byte + + // Eat the indicator character. + start_mark := parser.mark + skip(parser) + + // Consume the value. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + + for is_alpha(parser.buffer, parser.buffer_pos) { + s = read(parser, s) + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + + end_mark := parser.mark + + /* + * Check if length of the anchor is greater than 0 and it is followed by + * a whitespace character or one of the indicators: + * + * '?', ':', ',', ']', '}', '%', '@', '`'. + */ + + if len(s) == 0 || + !(is_blankz(parser.buffer, parser.buffer_pos) || parser.buffer[parser.buffer_pos] == '?' || + parser.buffer[parser.buffer_pos] == ':' || parser.buffer[parser.buffer_pos] == ',' || + parser.buffer[parser.buffer_pos] == ']' || parser.buffer[parser.buffer_pos] == '}' || + parser.buffer[parser.buffer_pos] == '%' || parser.buffer[parser.buffer_pos] == '@' || + parser.buffer[parser.buffer_pos] == '`') { + context := "while scanning an alias" + if typ == yaml_ANCHOR_TOKEN { + context = "while scanning an anchor" + } + yaml_parser_set_scanner_error(parser, context, start_mark, + "did not find expected alphabetic or numeric character") + return false + } + + // Create a token. + *token = yaml_token_t{ + typ: typ, + start_mark: start_mark, + end_mark: end_mark, + value: s, + } + + return true +} + +/* + * Scan a TAG token. + */ + +func yaml_parser_scan_tag(parser *yaml_parser_t, token *yaml_token_t) bool { + var handle, suffix []byte + + start_mark := parser.mark + + // Check if the tag is in the canonical form. + if parser.unread < 2 && !yaml_parser_update_buffer(parser, 2) { + return false + } + + if parser.buffer[parser.buffer_pos+1] == '<' { + // Keep the handle as '' + + // Eat '!<' + skip(parser) + skip(parser) + + // Consume the tag value. + if !yaml_parser_scan_tag_uri(parser, false, nil, start_mark, &suffix) { + return false + } + + // Check for '>' and eat it. + if parser.buffer[parser.buffer_pos] != '>' { + yaml_parser_set_scanner_error(parser, "while scanning a tag", + start_mark, "did not find the expected '>'") + return false + } + + skip(parser) + } else { + // The tag has either the '!suffix' or the '!handle!suffix' form. + + // First, try to scan a handle. + if !yaml_parser_scan_tag_handle(parser, false, start_mark, &handle) { + return false + } + + // Check if it is, indeed, handle. + if handle[0] == '!' && len(handle) > 1 && handle[len(handle)-1] == '!' { + // Scan the suffix now. + if !yaml_parser_scan_tag_uri(parser, false, nil, start_mark, &suffix) { + return false + } + } else { + // It wasn't a handle after all. Scan the rest of the tag. + if !yaml_parser_scan_tag_uri(parser, false, handle, start_mark, &suffix) { + return false + } + + // Set the handle to '!'. + handle = []byte{'!'} + + // A special case: the '!' tag. Set the handle to '' and the + // suffix to '!'. + if len(suffix) == 0 { + handle, suffix = suffix, handle + } + } + } + + // Check the character which ends the tag. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + if !is_blankz(parser.buffer, parser.buffer_pos) { + yaml_parser_set_scanner_error(parser, "while scanning a tag", + start_mark, "did not find expected whitespace or line break") + return false + } + + end_mark := parser.mark + + // Create a token. + *token = yaml_token_t{ + typ: yaml_TAG_TOKEN, + start_mark: start_mark, + end_mark: end_mark, + value: handle, + suffix: suffix, + } + return true +} + +// Scan a tag handle. +func yaml_parser_scan_tag_handle(parser *yaml_parser_t, directive bool, start_mark yaml_mark_t, handle *[]byte) bool { + // Check the initial '!' character. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + if parser.buffer[parser.buffer_pos] != '!' { + yaml_parser_set_scanner_tag_error(parser, directive, + start_mark, "did not find expected '!'") + return false + } + + var s []byte + + // Copy the '!' character. + s = read(parser, s) + + // Copy all subsequent alphabetical and numerical characters. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + for is_alpha(parser.buffer, parser.buffer_pos) { + s = read(parser, s) + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + + // Check if the trailing character is '!' and copy it. + if parser.buffer[parser.buffer_pos] == '!' { + s = read(parser, s) + } else { + // It's either the '!' tag or not really a tag handle. If it's a %TAG + // directive, it's an error. If it's a tag token, it must be a part of URI. + if directive && string(s) != "!" { + yaml_parser_set_scanner_tag_error(parser, directive, + start_mark, "did not find expected '!'") + return false + } + } + + *handle = s + return true +} + +// Scan a tag. +func yaml_parser_scan_tag_uri(parser *yaml_parser_t, directive bool, head []byte, start_mark yaml_mark_t, uri *[]byte) bool { + //size_t length = head ? strlen((char *)head) : 0 + var s []byte + hasTag := len(head) > 0 + + // Copy the head if needed. + // + // Note that we don't copy the leading '!' character. + if len(head) > 1 { + s = append(s, head[1:]...) + } + + // Scan the tag. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + + // The set of characters that may appear in URI is as follows: + // + // '0'-'9', 'A'-'Z', 'a'-'z', '_', '-', ';', '/', '?', ':', '@', '&', + // '=', '+', '$', ',', '.', '!', '~', '*', '\'', '(', ')', '[', ']', + // '%'. + // [Go] Convert this into more reasonable logic. + for is_alpha(parser.buffer, parser.buffer_pos) || parser.buffer[parser.buffer_pos] == ';' || + parser.buffer[parser.buffer_pos] == '/' || parser.buffer[parser.buffer_pos] == '?' || + parser.buffer[parser.buffer_pos] == ':' || parser.buffer[parser.buffer_pos] == '@' || + parser.buffer[parser.buffer_pos] == '&' || parser.buffer[parser.buffer_pos] == '=' || + parser.buffer[parser.buffer_pos] == '+' || parser.buffer[parser.buffer_pos] == '$' || + parser.buffer[parser.buffer_pos] == ',' || parser.buffer[parser.buffer_pos] == '.' || + parser.buffer[parser.buffer_pos] == '!' || parser.buffer[parser.buffer_pos] == '~' || + parser.buffer[parser.buffer_pos] == '*' || parser.buffer[parser.buffer_pos] == '\'' || + parser.buffer[parser.buffer_pos] == '(' || parser.buffer[parser.buffer_pos] == ')' || + parser.buffer[parser.buffer_pos] == '[' || parser.buffer[parser.buffer_pos] == ']' || + parser.buffer[parser.buffer_pos] == '%' { + // Check if it is a URI-escape sequence. + if parser.buffer[parser.buffer_pos] == '%' { + if !yaml_parser_scan_uri_escapes(parser, directive, start_mark, &s) { + return false + } + } else { + s = read(parser, s) + } + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + hasTag = true + } + + if !hasTag { + yaml_parser_set_scanner_tag_error(parser, directive, + start_mark, "did not find expected tag URI") + return false + } + *uri = s + return true +} + +// Decode an URI-escape sequence corresponding to a single UTF-8 character. +func yaml_parser_scan_uri_escapes(parser *yaml_parser_t, directive bool, start_mark yaml_mark_t, s *[]byte) bool { + + // Decode the required number of characters. + w := 1024 + for w > 0 { + // Check for a URI-escaped octet. + if parser.unread < 3 && !yaml_parser_update_buffer(parser, 3) { + return false + } + + if !(parser.buffer[parser.buffer_pos] == '%' && + is_hex(parser.buffer, parser.buffer_pos+1) && + is_hex(parser.buffer, parser.buffer_pos+2)) { + return yaml_parser_set_scanner_tag_error(parser, directive, + start_mark, "did not find URI escaped octet") + } + + // Get the octet. + octet := byte((as_hex(parser.buffer, parser.buffer_pos+1) << 4) + as_hex(parser.buffer, parser.buffer_pos+2)) + + // If it is the leading octet, determine the length of the UTF-8 sequence. + if w == 1024 { + w = width(octet) + if w == 0 { + return yaml_parser_set_scanner_tag_error(parser, directive, + start_mark, "found an incorrect leading UTF-8 octet") + } + } else { + // Check if the trailing octet is correct. + if octet&0xC0 != 0x80 { + return yaml_parser_set_scanner_tag_error(parser, directive, + start_mark, "found an incorrect trailing UTF-8 octet") + } + } + + // Copy the octet and move the pointers. + *s = append(*s, octet) + skip(parser) + skip(parser) + skip(parser) + w-- + } + return true +} + +// Scan a block scalar. +func yaml_parser_scan_block_scalar(parser *yaml_parser_t, token *yaml_token_t, literal bool) bool { + // Eat the indicator '|' or '>'. + start_mark := parser.mark + skip(parser) + + // Scan the additional block scalar indicators. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + + // Check for a chomping indicator. + var chomping, increment int + if parser.buffer[parser.buffer_pos] == '+' || parser.buffer[parser.buffer_pos] == '-' { + // Set the chomping method and eat the indicator. + if parser.buffer[parser.buffer_pos] == '+' { + chomping = +1 + } else { + chomping = -1 + } + skip(parser) + + // Check for an indentation indicator. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + if is_digit(parser.buffer, parser.buffer_pos) { + // Check that the indentation is greater than 0. + if parser.buffer[parser.buffer_pos] == '0' { + yaml_parser_set_scanner_error(parser, "while scanning a block scalar", + start_mark, "found an indentation indicator equal to 0") + return false + } + + // Get the indentation level and eat the indicator. + increment = as_digit(parser.buffer, parser.buffer_pos) + skip(parser) + } + + } else if is_digit(parser.buffer, parser.buffer_pos) { + // Do the same as above, but in the opposite order. + + if parser.buffer[parser.buffer_pos] == '0' { + yaml_parser_set_scanner_error(parser, "while scanning a block scalar", + start_mark, "found an indentation indicator equal to 0") + return false + } + increment = as_digit(parser.buffer, parser.buffer_pos) + skip(parser) + + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + if parser.buffer[parser.buffer_pos] == '+' || parser.buffer[parser.buffer_pos] == '-' { + if parser.buffer[parser.buffer_pos] == '+' { + chomping = +1 + } else { + chomping = -1 + } + skip(parser) + } + } + + // Eat whitespaces and comments to the end of the line. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + for is_blank(parser.buffer, parser.buffer_pos) { + skip(parser) + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + if parser.buffer[parser.buffer_pos] == '#' { + for !is_breakz(parser.buffer, parser.buffer_pos) { + skip(parser) + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + } + + // Check if we are at the end of the line. + if !is_breakz(parser.buffer, parser.buffer_pos) { + yaml_parser_set_scanner_error(parser, "while scanning a block scalar", + start_mark, "did not find expected comment or line break") + return false + } + + // Eat a line break. + if is_break(parser.buffer, parser.buffer_pos) { + if parser.unread < 2 && !yaml_parser_update_buffer(parser, 2) { + return false + } + skip_line(parser) + } + + end_mark := parser.mark + + // Set the indentation level if it was specified. + var indent int + if increment > 0 { + if parser.indent >= 0 { + indent = parser.indent + increment + } else { + indent = increment + } + } + + // Scan the leading line breaks and determine the indentation level if needed. + var s, leading_break, trailing_breaks []byte + if !yaml_parser_scan_block_scalar_breaks(parser, &indent, &trailing_breaks, start_mark, &end_mark) { + return false + } + + // Scan the block scalar content. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + var leading_blank, trailing_blank bool + for parser.mark.column == indent && !is_z(parser.buffer, parser.buffer_pos) { + // We are at the beginning of a non-empty line. + + // Is it a trailing whitespace? + trailing_blank = is_blank(parser.buffer, parser.buffer_pos) + + // Check if we need to fold the leading line break. + if !literal && !leading_blank && !trailing_blank && len(leading_break) > 0 && leading_break[0] == '\n' { + // Do we need to join the lines by space? + if len(trailing_breaks) == 0 { + s = append(s, ' ') + } + } else { + s = append(s, leading_break...) + } + leading_break = leading_break[:0] + + // Append the remaining line breaks. + s = append(s, trailing_breaks...) + trailing_breaks = trailing_breaks[:0] + + // Is it a leading whitespace? + leading_blank = is_blank(parser.buffer, parser.buffer_pos) + + // Consume the current line. + for !is_breakz(parser.buffer, parser.buffer_pos) { + s = read(parser, s) + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + + // Consume the line break. + if parser.unread < 2 && !yaml_parser_update_buffer(parser, 2) { + return false + } + + leading_break = read_line(parser, leading_break) + + // Eat the following indentation spaces and line breaks. + if !yaml_parser_scan_block_scalar_breaks(parser, &indent, &trailing_breaks, start_mark, &end_mark) { + return false + } + } + + // Chomp the tail. + if chomping != -1 { + s = append(s, leading_break...) + } + if chomping == 1 { + s = append(s, trailing_breaks...) + } + + // Create a token. + *token = yaml_token_t{ + typ: yaml_SCALAR_TOKEN, + start_mark: start_mark, + end_mark: end_mark, + value: s, + style: yaml_LITERAL_SCALAR_STYLE, + } + if !literal { + token.style = yaml_FOLDED_SCALAR_STYLE + } + return true +} + +// Scan indentation spaces and line breaks for a block scalar. Determine the +// indentation level if needed. +func yaml_parser_scan_block_scalar_breaks(parser *yaml_parser_t, indent *int, breaks *[]byte, start_mark yaml_mark_t, end_mark *yaml_mark_t) bool { + *end_mark = parser.mark + + // Eat the indentation spaces and line breaks. + max_indent := 0 + for { + // Eat the indentation spaces. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + for (*indent == 0 || parser.mark.column < *indent) && is_space(parser.buffer, parser.buffer_pos) { + skip(parser) + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + if parser.mark.column > max_indent { + max_indent = parser.mark.column + } + + // Check for a tab character messing the indentation. + if (*indent == 0 || parser.mark.column < *indent) && is_tab(parser.buffer, parser.buffer_pos) { + return yaml_parser_set_scanner_error(parser, "while scanning a block scalar", + start_mark, "found a tab character where an indentation space is expected") + } + + // Have we found a non-empty line? + if !is_break(parser.buffer, parser.buffer_pos) { + break + } + + // Consume the line break. + if parser.unread < 2 && !yaml_parser_update_buffer(parser, 2) { + return false + } + // [Go] Should really be returning breaks instead. + *breaks = read_line(parser, *breaks) + *end_mark = parser.mark + } + + // Determine the indentation level if needed. + if *indent == 0 { + *indent = max_indent + if *indent < parser.indent+1 { + *indent = parser.indent + 1 + } + if *indent < 1 { + *indent = 1 + } + } + return true +} + +// Scan a quoted scalar. +func yaml_parser_scan_flow_scalar(parser *yaml_parser_t, token *yaml_token_t, single bool) bool { + // Eat the left quote. + start_mark := parser.mark + skip(parser) + + // Consume the content of the quoted scalar. + var s, leading_break, trailing_breaks, whitespaces []byte + for { + // Check that there are no document indicators at the beginning of the line. + if parser.unread < 4 && !yaml_parser_update_buffer(parser, 4) { + return false + } + + if parser.mark.column == 0 && + ((parser.buffer[parser.buffer_pos+0] == '-' && + parser.buffer[parser.buffer_pos+1] == '-' && + parser.buffer[parser.buffer_pos+2] == '-') || + (parser.buffer[parser.buffer_pos+0] == '.' && + parser.buffer[parser.buffer_pos+1] == '.' && + parser.buffer[parser.buffer_pos+2] == '.')) && + is_blankz(parser.buffer, parser.buffer_pos+3) { + yaml_parser_set_scanner_error(parser, "while scanning a quoted scalar", + start_mark, "found unexpected document indicator") + return false + } + + // Check for EOF. + if is_z(parser.buffer, parser.buffer_pos) { + yaml_parser_set_scanner_error(parser, "while scanning a quoted scalar", + start_mark, "found unexpected end of stream") + return false + } + + // Consume non-blank characters. + leading_blanks := false + for !is_blankz(parser.buffer, parser.buffer_pos) { + if single && parser.buffer[parser.buffer_pos] == '\'' && parser.buffer[parser.buffer_pos+1] == '\'' { + // Is is an escaped single quote. + s = append(s, '\'') + skip(parser) + skip(parser) + + } else if single && parser.buffer[parser.buffer_pos] == '\'' { + // It is a right single quote. + break + } else if !single && parser.buffer[parser.buffer_pos] == '"' { + // It is a right double quote. + break + + } else if !single && parser.buffer[parser.buffer_pos] == '\\' && is_break(parser.buffer, parser.buffer_pos+1) { + // It is an escaped line break. + if parser.unread < 3 && !yaml_parser_update_buffer(parser, 3) { + return false + } + skip(parser) + skip_line(parser) + leading_blanks = true + break + + } else if !single && parser.buffer[parser.buffer_pos] == '\\' { + // It is an escape sequence. + code_length := 0 + + // Check the escape character. + switch parser.buffer[parser.buffer_pos+1] { + case '0': + s = append(s, 0) + case 'a': + s = append(s, '\x07') + case 'b': + s = append(s, '\x08') + case 't', '\t': + s = append(s, '\x09') + case 'n': + s = append(s, '\x0A') + case 'v': + s = append(s, '\x0B') + case 'f': + s = append(s, '\x0C') + case 'r': + s = append(s, '\x0D') + case 'e': + s = append(s, '\x1B') + case ' ': + s = append(s, '\x20') + case '"': + s = append(s, '"') + case '\'': + s = append(s, '\'') + case '\\': + s = append(s, '\\') + case 'N': // NEL (#x85) + s = append(s, '\xC2') + s = append(s, '\x85') + case '_': // #xA0 + s = append(s, '\xC2') + s = append(s, '\xA0') + case 'L': // LS (#x2028) + s = append(s, '\xE2') + s = append(s, '\x80') + s = append(s, '\xA8') + case 'P': // PS (#x2029) + s = append(s, '\xE2') + s = append(s, '\x80') + s = append(s, '\xA9') + case 'x': + code_length = 2 + case 'u': + code_length = 4 + case 'U': + code_length = 8 + default: + yaml_parser_set_scanner_error(parser, "while parsing a quoted scalar", + start_mark, "found unknown escape character") + return false + } + + skip(parser) + skip(parser) + + // Consume an arbitrary escape code. + if code_length > 0 { + var value int + + // Scan the character value. + if parser.unread < code_length && !yaml_parser_update_buffer(parser, code_length) { + return false + } + for k := 0; k < code_length; k++ { + if !is_hex(parser.buffer, parser.buffer_pos+k) { + yaml_parser_set_scanner_error(parser, "while parsing a quoted scalar", + start_mark, "did not find expected hexdecimal number") + return false + } + value = (value << 4) + as_hex(parser.buffer, parser.buffer_pos+k) + } + + // Check the value and write the character. + if (value >= 0xD800 && value <= 0xDFFF) || value > 0x10FFFF { + yaml_parser_set_scanner_error(parser, "while parsing a quoted scalar", + start_mark, "found invalid Unicode character escape code") + return false + } + if value <= 0x7F { + s = append(s, byte(value)) + } else if value <= 0x7FF { + s = append(s, byte(0xC0+(value>>6))) + s = append(s, byte(0x80+(value&0x3F))) + } else if value <= 0xFFFF { + s = append(s, byte(0xE0+(value>>12))) + s = append(s, byte(0x80+((value>>6)&0x3F))) + s = append(s, byte(0x80+(value&0x3F))) + } else { + s = append(s, byte(0xF0+(value>>18))) + s = append(s, byte(0x80+((value>>12)&0x3F))) + s = append(s, byte(0x80+((value>>6)&0x3F))) + s = append(s, byte(0x80+(value&0x3F))) + } + + // Advance the pointer. + for k := 0; k < code_length; k++ { + skip(parser) + } + } + } else { + // It is a non-escaped non-blank character. + s = read(parser, s) + } + if parser.unread < 2 && !yaml_parser_update_buffer(parser, 2) { + return false + } + } + + // Check if we are at the end of the scalar. + if single { + if parser.buffer[parser.buffer_pos] == '\'' { + break + } + } else { + if parser.buffer[parser.buffer_pos] == '"' { + break + } + } + + // Consume blank characters. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + + for is_blank(parser.buffer, parser.buffer_pos) || is_break(parser.buffer, parser.buffer_pos) { + if is_blank(parser.buffer, parser.buffer_pos) { + // Consume a space or a tab character. + if !leading_blanks { + whitespaces = read(parser, whitespaces) + } else { + skip(parser) + } + } else { + if parser.unread < 2 && !yaml_parser_update_buffer(parser, 2) { + return false + } + + // Check if it is a first line break. + if !leading_blanks { + whitespaces = whitespaces[:0] + leading_break = read_line(parser, leading_break) + leading_blanks = true + } else { + trailing_breaks = read_line(parser, trailing_breaks) + } + } + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + + // Join the whitespaces or fold line breaks. + if leading_blanks { + // Do we need to fold line breaks? + if len(leading_break) > 0 && leading_break[0] == '\n' { + if len(trailing_breaks) == 0 { + s = append(s, ' ') + } else { + s = append(s, trailing_breaks...) + } + } else { + s = append(s, leading_break...) + s = append(s, trailing_breaks...) + } + trailing_breaks = trailing_breaks[:0] + leading_break = leading_break[:0] + } else { + s = append(s, whitespaces...) + whitespaces = whitespaces[:0] + } + } + + // Eat the right quote. + skip(parser) + end_mark := parser.mark + + // Create a token. + *token = yaml_token_t{ + typ: yaml_SCALAR_TOKEN, + start_mark: start_mark, + end_mark: end_mark, + value: s, + style: yaml_SINGLE_QUOTED_SCALAR_STYLE, + } + if !single { + token.style = yaml_DOUBLE_QUOTED_SCALAR_STYLE + } + return true +} + +// Scan a plain scalar. +func yaml_parser_scan_plain_scalar(parser *yaml_parser_t, token *yaml_token_t) bool { + + var s, leading_break, trailing_breaks, whitespaces []byte + var leading_blanks bool + var indent = parser.indent + 1 + + start_mark := parser.mark + end_mark := parser.mark + + // Consume the content of the plain scalar. + for { + // Check for a document indicator. + if parser.unread < 4 && !yaml_parser_update_buffer(parser, 4) { + return false + } + if parser.mark.column == 0 && + ((parser.buffer[parser.buffer_pos+0] == '-' && + parser.buffer[parser.buffer_pos+1] == '-' && + parser.buffer[parser.buffer_pos+2] == '-') || + (parser.buffer[parser.buffer_pos+0] == '.' && + parser.buffer[parser.buffer_pos+1] == '.' && + parser.buffer[parser.buffer_pos+2] == '.')) && + is_blankz(parser.buffer, parser.buffer_pos+3) { + break + } + + // Check for a comment. + if parser.buffer[parser.buffer_pos] == '#' { + break + } + + // Consume non-blank characters. + for !is_blankz(parser.buffer, parser.buffer_pos) { + + // Check for 'x:x' in the flow context. TODO: Fix the test "spec-08-13". + if parser.flow_level > 0 && + parser.buffer[parser.buffer_pos] == ':' && + !is_blankz(parser.buffer, parser.buffer_pos+1) { + yaml_parser_set_scanner_error(parser, "while scanning a plain scalar", + start_mark, "found unexpected ':'") + return false + } + + // Check for indicators that may end a plain scalar. + if (parser.buffer[parser.buffer_pos] == ':' && is_blankz(parser.buffer, parser.buffer_pos+1)) || + (parser.flow_level > 0 && + (parser.buffer[parser.buffer_pos] == ',' || parser.buffer[parser.buffer_pos] == ':' || + parser.buffer[parser.buffer_pos] == '?' || parser.buffer[parser.buffer_pos] == '[' || + parser.buffer[parser.buffer_pos] == ']' || parser.buffer[parser.buffer_pos] == '{' || + parser.buffer[parser.buffer_pos] == '}')) { + break + } + + // Check if we need to join whitespaces and breaks. + if leading_blanks || len(whitespaces) > 0 { + if leading_blanks { + // Do we need to fold line breaks? + if leading_break[0] == '\n' { + if len(trailing_breaks) == 0 { + s = append(s, ' ') + } else { + s = append(s, trailing_breaks...) + } + } else { + s = append(s, leading_break...) + s = append(s, trailing_breaks...) + } + trailing_breaks = trailing_breaks[:0] + leading_break = leading_break[:0] + leading_blanks = false + } else { + s = append(s, whitespaces...) + whitespaces = whitespaces[:0] + } + } + + // Copy the character. + s = read(parser, s) + + end_mark = parser.mark + if parser.unread < 2 && !yaml_parser_update_buffer(parser, 2) { + return false + } + } + + // Is it the end? + if !(is_blank(parser.buffer, parser.buffer_pos) || is_break(parser.buffer, parser.buffer_pos)) { + break + } + + // Consume blank characters. + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + + for is_blank(parser.buffer, parser.buffer_pos) || is_break(parser.buffer, parser.buffer_pos) { + if is_blank(parser.buffer, parser.buffer_pos) { + + // Check for tab character that abuse indentation. + if leading_blanks && parser.mark.column < indent && is_tab(parser.buffer, parser.buffer_pos) { + yaml_parser_set_scanner_error(parser, "while scanning a plain scalar", + start_mark, "found a tab character that violate indentation") + return false + } + + // Consume a space or a tab character. + if !leading_blanks { + whitespaces = read(parser, whitespaces) + } else { + skip(parser) + } + } else { + if parser.unread < 2 && !yaml_parser_update_buffer(parser, 2) { + return false + } + + // Check if it is a first line break. + if !leading_blanks { + whitespaces = whitespaces[:0] + leading_break = read_line(parser, leading_break) + leading_blanks = true + } else { + trailing_breaks = read_line(parser, trailing_breaks) + } + } + if parser.unread < 1 && !yaml_parser_update_buffer(parser, 1) { + return false + } + } + + // Check indentation level. + if parser.flow_level == 0 && parser.mark.column < indent { + break + } + } + + // Create a token. + *token = yaml_token_t{ + typ: yaml_SCALAR_TOKEN, + start_mark: start_mark, + end_mark: end_mark, + value: s, + style: yaml_PLAIN_SCALAR_STYLE, + } + + // Note that we change the 'simple_key_allowed' flag. + if leading_blanks { + parser.simple_key_allowed = true + } + return true +} diff --git a/generator/vendor/gopkg.in/yaml.v2/sorter.go b/generator/vendor/gopkg.in/yaml.v2/sorter.go new file mode 100644 index 00000000..5958822f --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/sorter.go @@ -0,0 +1,104 @@ +package yaml + +import ( + "reflect" + "unicode" +) + +type keyList []reflect.Value + +func (l keyList) Len() int { return len(l) } +func (l keyList) Swap(i, j int) { l[i], l[j] = l[j], l[i] } +func (l keyList) Less(i, j int) bool { + a := l[i] + b := l[j] + ak := a.Kind() + bk := b.Kind() + for (ak == reflect.Interface || ak == reflect.Ptr) && !a.IsNil() { + a = a.Elem() + ak = a.Kind() + } + for (bk == reflect.Interface || bk == reflect.Ptr) && !b.IsNil() { + b = b.Elem() + bk = b.Kind() + } + af, aok := keyFloat(a) + bf, bok := keyFloat(b) + if aok && bok { + if af != bf { + return af < bf + } + if ak != bk { + return ak < bk + } + return numLess(a, b) + } + if ak != reflect.String || bk != reflect.String { + return ak < bk + } + ar, br := []rune(a.String()), []rune(b.String()) + for i := 0; i < len(ar) && i < len(br); i++ { + if ar[i] == br[i] { + continue + } + al := unicode.IsLetter(ar[i]) + bl := unicode.IsLetter(br[i]) + if al && bl { + return ar[i] < br[i] + } + if al || bl { + return bl + } + var ai, bi int + var an, bn int64 + for ai = i; ai < len(ar) && unicode.IsDigit(ar[ai]); ai++ { + an = an*10 + int64(ar[ai]-'0') + } + for bi = i; bi < len(br) && unicode.IsDigit(br[bi]); bi++ { + bn = bn*10 + int64(br[bi]-'0') + } + if an != bn { + return an < bn + } + if ai != bi { + return ai < bi + } + return ar[i] < br[i] + } + return len(ar) < len(br) +} + +// keyFloat returns a float value for v if it is a number/bool +// and whether it is a number/bool or not. +func keyFloat(v reflect.Value) (f float64, ok bool) { + switch v.Kind() { + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + return float64(v.Int()), true + case reflect.Float32, reflect.Float64: + return v.Float(), true + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: + return float64(v.Uint()), true + case reflect.Bool: + if v.Bool() { + return 1, true + } + return 0, true + } + return 0, false +} + +// numLess returns whether a < b. +// a and b must necessarily have the same kind. +func numLess(a, b reflect.Value) bool { + switch a.Kind() { + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + return a.Int() < b.Int() + case reflect.Float32, reflect.Float64: + return a.Float() < b.Float() + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: + return a.Uint() < b.Uint() + case reflect.Bool: + return !a.Bool() && b.Bool() + } + panic("not a number") +} diff --git a/generator/vendor/gopkg.in/yaml.v2/suite_test.go b/generator/vendor/gopkg.in/yaml.v2/suite_test.go new file mode 100644 index 00000000..c5cf1ed4 --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/suite_test.go @@ -0,0 +1,12 @@ +package yaml_test + +import ( + . "gopkg.in/check.v1" + "testing" +) + +func Test(t *testing.T) { TestingT(t) } + +type S struct{} + +var _ = Suite(&S{}) diff --git a/generator/vendor/gopkg.in/yaml.v2/writerc.go b/generator/vendor/gopkg.in/yaml.v2/writerc.go new file mode 100644 index 00000000..190362f2 --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/writerc.go @@ -0,0 +1,89 @@ +package yaml + +// Set the writer error and return false. +func yaml_emitter_set_writer_error(emitter *yaml_emitter_t, problem string) bool { + emitter.error = yaml_WRITER_ERROR + emitter.problem = problem + return false +} + +// Flush the output buffer. +func yaml_emitter_flush(emitter *yaml_emitter_t) bool { + if emitter.write_handler == nil { + panic("write handler not set") + } + + // Check if the buffer is empty. + if emitter.buffer_pos == 0 { + return true + } + + // If the output encoding is UTF-8, we don't need to recode the buffer. + if emitter.encoding == yaml_UTF8_ENCODING { + if err := emitter.write_handler(emitter, emitter.buffer[:emitter.buffer_pos]); err != nil { + return yaml_emitter_set_writer_error(emitter, "write error: "+err.Error()) + } + emitter.buffer_pos = 0 + return true + } + + // Recode the buffer into the raw buffer. + var low, high int + if emitter.encoding == yaml_UTF16LE_ENCODING { + low, high = 0, 1 + } else { + high, low = 1, 0 + } + + pos := 0 + for pos < emitter.buffer_pos { + // See the "reader.c" code for more details on UTF-8 encoding. Note + // that we assume that the buffer contains a valid UTF-8 sequence. + + // Read the next UTF-8 character. + octet := emitter.buffer[pos] + + var w int + var value rune + switch { + case octet&0x80 == 0x00: + w, value = 1, rune(octet&0x7F) + case octet&0xE0 == 0xC0: + w, value = 2, rune(octet&0x1F) + case octet&0xF0 == 0xE0: + w, value = 3, rune(octet&0x0F) + case octet&0xF8 == 0xF0: + w, value = 4, rune(octet&0x07) + } + for k := 1; k < w; k++ { + octet = emitter.buffer[pos+k] + value = (value << 6) + (rune(octet) & 0x3F) + } + pos += w + + // Write the character. + if value < 0x10000 { + var b [2]byte + b[high] = byte(value >> 8) + b[low] = byte(value & 0xFF) + emitter.raw_buffer = append(emitter.raw_buffer, b[0], b[1]) + } else { + // Write the character using a surrogate pair (check "reader.c"). + var b [4]byte + value -= 0x10000 + b[high] = byte(0xD8 + (value >> 18)) + b[low] = byte((value >> 10) & 0xFF) + b[high+2] = byte(0xDC + ((value >> 8) & 0xFF)) + b[low+2] = byte(value & 0xFF) + emitter.raw_buffer = append(emitter.raw_buffer, b[0], b[1], b[2], b[3]) + } + } + + // Write the raw buffer. + if err := emitter.write_handler(emitter, emitter.raw_buffer); err != nil { + return yaml_emitter_set_writer_error(emitter, "write error: "+err.Error()) + } + emitter.buffer_pos = 0 + emitter.raw_buffer = emitter.raw_buffer[:0] + return true +} diff --git a/generator/vendor/gopkg.in/yaml.v2/yaml.go b/generator/vendor/gopkg.in/yaml.v2/yaml.go new file mode 100644 index 00000000..bf18884e --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/yaml.go @@ -0,0 +1,357 @@ +// Package yaml implements YAML support for the Go language. +// +// Source code and other details for the project are available at GitHub: +// +// https://github.com/go-yaml/yaml +// +package yaml + +import ( + "errors" + "fmt" + "reflect" + "strings" + "sync" +) + +// MapSlice encodes and decodes as a YAML map. +// The order of keys is preserved when encoding and decoding. +type MapSlice []MapItem + +// MapItem is an item in a MapSlice. +type MapItem struct { + Key, Value interface{} +} + +// The Unmarshaler interface may be implemented by types to customize their +// behavior when being unmarshaled from a YAML document. The UnmarshalYAML +// method receives a function that may be called to unmarshal the original +// YAML value into a field or variable. It is safe to call the unmarshal +// function parameter more than once if necessary. +type Unmarshaler interface { + UnmarshalYAML(unmarshal func(interface{}) error) error +} + +// The Marshaler interface may be implemented by types to customize their +// behavior when being marshaled into a YAML document. The returned value +// is marshaled in place of the original value implementing Marshaler. +// +// If an error is returned by MarshalYAML, the marshaling procedure stops +// and returns with the provided error. +type Marshaler interface { + MarshalYAML() (interface{}, error) +} + +// Unmarshal decodes the first document found within the in byte slice +// and assigns decoded values into the out value. +// +// Maps and pointers (to a struct, string, int, etc) are accepted as out +// values. If an internal pointer within a struct is not initialized, +// the yaml package will initialize it if necessary for unmarshalling +// the provided data. The out parameter must not be nil. +// +// The type of the decoded values should be compatible with the respective +// values in out. If one or more values cannot be decoded due to a type +// mismatches, decoding continues partially until the end of the YAML +// content, and a *yaml.TypeError is returned with details for all +// missed values. +// +// Struct fields are only unmarshalled if they are exported (have an +// upper case first letter), and are unmarshalled using the field name +// lowercased as the default key. Custom keys may be defined via the +// "yaml" name in the field tag: the content preceding the first comma +// is used as the key, and the following comma-separated options are +// used to tweak the marshalling process (see Marshal). +// Conflicting names result in a runtime error. +// +// For example: +// +// type T struct { +// F int `yaml:"a,omitempty"` +// B int +// } +// var t T +// yaml.Unmarshal([]byte("a: 1\nb: 2"), &t) +// +// See the documentation of Marshal for the format of tags and a list of +// supported tag options. +// +func Unmarshal(in []byte, out interface{}) (err error) { + return unmarshal(in, out, false) +} + +// UnmarshalStrict is like Unmarshal except that any fields that are found +// in the data that do not have corresponding struct members will result in +// an error. +func UnmarshalStrict(in []byte, out interface{}) (err error) { + return unmarshal(in, out, true) +} + +func unmarshal(in []byte, out interface{}, strict bool) (err error) { + defer handleErr(&err) + d := newDecoder(strict) + p := newParser(in) + defer p.destroy() + node := p.parse() + if node != nil { + v := reflect.ValueOf(out) + if v.Kind() == reflect.Ptr && !v.IsNil() { + v = v.Elem() + } + d.unmarshal(node, v) + } + if len(d.terrors) > 0 { + return &TypeError{d.terrors} + } + return nil +} + +// Marshal serializes the value provided into a YAML document. The structure +// of the generated document will reflect the structure of the value itself. +// Maps and pointers (to struct, string, int, etc) are accepted as the in value. +// +// Struct fields are only unmarshalled if they are exported (have an upper case +// first letter), and are unmarshalled using the field name lowercased as the +// default key. Custom keys may be defined via the "yaml" name in the field +// tag: the content preceding the first comma is used as the key, and the +// following comma-separated options are used to tweak the marshalling process. +// Conflicting names result in a runtime error. +// +// The field tag format accepted is: +// +// `(...) yaml:"[<key>][,<flag1>[,<flag2>]]" (...)` +// +// The following flags are currently supported: +// +// omitempty Only include the field if it's not set to the zero +// value for the type or to empty slices or maps. +// Does not apply to zero valued structs. +// +// flow Marshal using a flow style (useful for structs, +// sequences and maps). +// +// inline Inline the field, which must be a struct or a map, +// causing all of its fields or keys to be processed as if +// they were part of the outer struct. For maps, keys must +// not conflict with the yaml keys of other struct fields. +// +// In addition, if the key is "-", the field is ignored. +// +// For example: +// +// type T struct { +// F int "a,omitempty" +// B int +// } +// yaml.Marshal(&T{B: 2}) // Returns "b: 2\n" +// yaml.Marshal(&T{F: 1}} // Returns "a: 1\nb: 0\n" +// +func Marshal(in interface{}) (out []byte, err error) { + defer handleErr(&err) + e := newEncoder() + defer e.destroy() + e.marshal("", reflect.ValueOf(in)) + e.finish() + out = e.out + return +} + +func handleErr(err *error) { + if v := recover(); v != nil { + if e, ok := v.(yamlError); ok { + *err = e.err + } else { + panic(v) + } + } +} + +type yamlError struct { + err error +} + +func fail(err error) { + panic(yamlError{err}) +} + +func failf(format string, args ...interface{}) { + panic(yamlError{fmt.Errorf("yaml: "+format, args...)}) +} + +// A TypeError is returned by Unmarshal when one or more fields in +// the YAML document cannot be properly decoded into the requested +// types. When this error is returned, the value is still +// unmarshaled partially. +type TypeError struct { + Errors []string +} + +func (e *TypeError) Error() string { + return fmt.Sprintf("yaml: unmarshal errors:\n %s", strings.Join(e.Errors, "\n ")) +} + +// -------------------------------------------------------------------------- +// Maintain a mapping of keys to structure field indexes + +// The code in this section was copied from mgo/bson. + +// structInfo holds details for the serialization of fields of +// a given struct. +type structInfo struct { + FieldsMap map[string]fieldInfo + FieldsList []fieldInfo + + // InlineMap is the number of the field in the struct that + // contains an ,inline map, or -1 if there's none. + InlineMap int +} + +type fieldInfo struct { + Key string + Num int + OmitEmpty bool + Flow bool + + // Inline holds the field index if the field is part of an inlined struct. + Inline []int +} + +var structMap = make(map[reflect.Type]*structInfo) +var fieldMapMutex sync.RWMutex + +func getStructInfo(st reflect.Type) (*structInfo, error) { + fieldMapMutex.RLock() + sinfo, found := structMap[st] + fieldMapMutex.RUnlock() + if found { + return sinfo, nil + } + + n := st.NumField() + fieldsMap := make(map[string]fieldInfo) + fieldsList := make([]fieldInfo, 0, n) + inlineMap := -1 + for i := 0; i != n; i++ { + field := st.Field(i) + if field.PkgPath != "" && !field.Anonymous { + continue // Private field + } + + info := fieldInfo{Num: i} + + tag := field.Tag.Get("yaml") + if tag == "" && strings.Index(string(field.Tag), ":") < 0 { + tag = string(field.Tag) + } + if tag == "-" { + continue + } + + inline := false + fields := strings.Split(tag, ",") + if len(fields) > 1 { + for _, flag := range fields[1:] { + switch flag { + case "omitempty": + info.OmitEmpty = true + case "flow": + info.Flow = true + case "inline": + inline = true + default: + return nil, errors.New(fmt.Sprintf("Unsupported flag %q in tag %q of type %s", flag, tag, st)) + } + } + tag = fields[0] + } + + if inline { + switch field.Type.Kind() { + case reflect.Map: + if inlineMap >= 0 { + return nil, errors.New("Multiple ,inline maps in struct " + st.String()) + } + if field.Type.Key() != reflect.TypeOf("") { + return nil, errors.New("Option ,inline needs a map with string keys in struct " + st.String()) + } + inlineMap = info.Num + case reflect.Struct: + sinfo, err := getStructInfo(field.Type) + if err != nil { + return nil, err + } + for _, finfo := range sinfo.FieldsList { + if _, found := fieldsMap[finfo.Key]; found { + msg := "Duplicated key '" + finfo.Key + "' in struct " + st.String() + return nil, errors.New(msg) + } + if finfo.Inline == nil { + finfo.Inline = []int{i, finfo.Num} + } else { + finfo.Inline = append([]int{i}, finfo.Inline...) + } + fieldsMap[finfo.Key] = finfo + fieldsList = append(fieldsList, finfo) + } + default: + //return nil, errors.New("Option ,inline needs a struct value or map field") + return nil, errors.New("Option ,inline needs a struct value field") + } + continue + } + + if tag != "" { + info.Key = tag + } else { + info.Key = strings.ToLower(field.Name) + } + + if _, found = fieldsMap[info.Key]; found { + msg := "Duplicated key '" + info.Key + "' in struct " + st.String() + return nil, errors.New(msg) + } + + fieldsList = append(fieldsList, info) + fieldsMap[info.Key] = info + } + + sinfo = &structInfo{fieldsMap, fieldsList, inlineMap} + + fieldMapMutex.Lock() + structMap[st] = sinfo + fieldMapMutex.Unlock() + return sinfo, nil +} + +func isZero(v reflect.Value) bool { + switch v.Kind() { + case reflect.String: + return len(v.String()) == 0 + case reflect.Interface, reflect.Ptr: + return v.IsNil() + case reflect.Slice: + return v.Len() == 0 + case reflect.Map: + return v.Len() == 0 + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + return v.Int() == 0 + case reflect.Float32, reflect.Float64: + return v.Float() == 0 + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: + return v.Uint() == 0 + case reflect.Bool: + return !v.Bool() + case reflect.Struct: + vt := v.Type() + for i := v.NumField() - 1; i >= 0; i-- { + if vt.Field(i).PkgPath != "" { + continue // Private field + } + if !isZero(v.Field(i)) { + return false + } + } + return true + } + return false +} diff --git a/generator/vendor/gopkg.in/yaml.v2/yamlh.go b/generator/vendor/gopkg.in/yaml.v2/yamlh.go new file mode 100644 index 00000000..3caeca04 --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/yamlh.go @@ -0,0 +1,716 @@ +package yaml + +import ( + "io" +) + +// The version directive data. +type yaml_version_directive_t struct { + major int8 // The major version number. + minor int8 // The minor version number. +} + +// The tag directive data. +type yaml_tag_directive_t struct { + handle []byte // The tag handle. + prefix []byte // The tag prefix. +} + +type yaml_encoding_t int + +// The stream encoding. +const ( + // Let the parser choose the encoding. + yaml_ANY_ENCODING yaml_encoding_t = iota + + yaml_UTF8_ENCODING // The default UTF-8 encoding. + yaml_UTF16LE_ENCODING // The UTF-16-LE encoding with BOM. + yaml_UTF16BE_ENCODING // The UTF-16-BE encoding with BOM. +) + +type yaml_break_t int + +// Line break types. +const ( + // Let the parser choose the break type. + yaml_ANY_BREAK yaml_break_t = iota + + yaml_CR_BREAK // Use CR for line breaks (Mac style). + yaml_LN_BREAK // Use LN for line breaks (Unix style). + yaml_CRLN_BREAK // Use CR LN for line breaks (DOS style). +) + +type yaml_error_type_t int + +// Many bad things could happen with the parser and emitter. +const ( + // No error is produced. + yaml_NO_ERROR yaml_error_type_t = iota + + yaml_MEMORY_ERROR // Cannot allocate or reallocate a block of memory. + yaml_READER_ERROR // Cannot read or decode the input stream. + yaml_SCANNER_ERROR // Cannot scan the input stream. + yaml_PARSER_ERROR // Cannot parse the input stream. + yaml_COMPOSER_ERROR // Cannot compose a YAML document. + yaml_WRITER_ERROR // Cannot write to the output stream. + yaml_EMITTER_ERROR // Cannot emit a YAML stream. +) + +// The pointer position. +type yaml_mark_t struct { + index int // The position index. + line int // The position line. + column int // The position column. +} + +// Node Styles + +type yaml_style_t int8 + +type yaml_scalar_style_t yaml_style_t + +// Scalar styles. +const ( + // Let the emitter choose the style. + yaml_ANY_SCALAR_STYLE yaml_scalar_style_t = iota + + yaml_PLAIN_SCALAR_STYLE // The plain scalar style. + yaml_SINGLE_QUOTED_SCALAR_STYLE // The single-quoted scalar style. + yaml_DOUBLE_QUOTED_SCALAR_STYLE // The double-quoted scalar style. + yaml_LITERAL_SCALAR_STYLE // The literal scalar style. + yaml_FOLDED_SCALAR_STYLE // The folded scalar style. +) + +type yaml_sequence_style_t yaml_style_t + +// Sequence styles. +const ( + // Let the emitter choose the style. + yaml_ANY_SEQUENCE_STYLE yaml_sequence_style_t = iota + + yaml_BLOCK_SEQUENCE_STYLE // The block sequence style. + yaml_FLOW_SEQUENCE_STYLE // The flow sequence style. +) + +type yaml_mapping_style_t yaml_style_t + +// Mapping styles. +const ( + // Let the emitter choose the style. + yaml_ANY_MAPPING_STYLE yaml_mapping_style_t = iota + + yaml_BLOCK_MAPPING_STYLE // The block mapping style. + yaml_FLOW_MAPPING_STYLE // The flow mapping style. +) + +// Tokens + +type yaml_token_type_t int + +// Token types. +const ( + // An empty token. + yaml_NO_TOKEN yaml_token_type_t = iota + + yaml_STREAM_START_TOKEN // A STREAM-START token. + yaml_STREAM_END_TOKEN // A STREAM-END token. + + yaml_VERSION_DIRECTIVE_TOKEN // A VERSION-DIRECTIVE token. + yaml_TAG_DIRECTIVE_TOKEN // A TAG-DIRECTIVE token. + yaml_DOCUMENT_START_TOKEN // A DOCUMENT-START token. + yaml_DOCUMENT_END_TOKEN // A DOCUMENT-END token. + + yaml_BLOCK_SEQUENCE_START_TOKEN // A BLOCK-SEQUENCE-START token. + yaml_BLOCK_MAPPING_START_TOKEN // A BLOCK-SEQUENCE-END token. + yaml_BLOCK_END_TOKEN // A BLOCK-END token. + + yaml_FLOW_SEQUENCE_START_TOKEN // A FLOW-SEQUENCE-START token. + yaml_FLOW_SEQUENCE_END_TOKEN // A FLOW-SEQUENCE-END token. + yaml_FLOW_MAPPING_START_TOKEN // A FLOW-MAPPING-START token. + yaml_FLOW_MAPPING_END_TOKEN // A FLOW-MAPPING-END token. + + yaml_BLOCK_ENTRY_TOKEN // A BLOCK-ENTRY token. + yaml_FLOW_ENTRY_TOKEN // A FLOW-ENTRY token. + yaml_KEY_TOKEN // A KEY token. + yaml_VALUE_TOKEN // A VALUE token. + + yaml_ALIAS_TOKEN // An ALIAS token. + yaml_ANCHOR_TOKEN // An ANCHOR token. + yaml_TAG_TOKEN // A TAG token. + yaml_SCALAR_TOKEN // A SCALAR token. +) + +func (tt yaml_token_type_t) String() string { + switch tt { + case yaml_NO_TOKEN: + return "yaml_NO_TOKEN" + case yaml_STREAM_START_TOKEN: + return "yaml_STREAM_START_TOKEN" + case yaml_STREAM_END_TOKEN: + return "yaml_STREAM_END_TOKEN" + case yaml_VERSION_DIRECTIVE_TOKEN: + return "yaml_VERSION_DIRECTIVE_TOKEN" + case yaml_TAG_DIRECTIVE_TOKEN: + return "yaml_TAG_DIRECTIVE_TOKEN" + case yaml_DOCUMENT_START_TOKEN: + return "yaml_DOCUMENT_START_TOKEN" + case yaml_DOCUMENT_END_TOKEN: + return "yaml_DOCUMENT_END_TOKEN" + case yaml_BLOCK_SEQUENCE_START_TOKEN: + return "yaml_BLOCK_SEQUENCE_START_TOKEN" + case yaml_BLOCK_MAPPING_START_TOKEN: + return "yaml_BLOCK_MAPPING_START_TOKEN" + case yaml_BLOCK_END_TOKEN: + return "yaml_BLOCK_END_TOKEN" + case yaml_FLOW_SEQUENCE_START_TOKEN: + return "yaml_FLOW_SEQUENCE_START_TOKEN" + case yaml_FLOW_SEQUENCE_END_TOKEN: + return "yaml_FLOW_SEQUENCE_END_TOKEN" + case yaml_FLOW_MAPPING_START_TOKEN: + return "yaml_FLOW_MAPPING_START_TOKEN" + case yaml_FLOW_MAPPING_END_TOKEN: + return "yaml_FLOW_MAPPING_END_TOKEN" + case yaml_BLOCK_ENTRY_TOKEN: + return "yaml_BLOCK_ENTRY_TOKEN" + case yaml_FLOW_ENTRY_TOKEN: + return "yaml_FLOW_ENTRY_TOKEN" + case yaml_KEY_TOKEN: + return "yaml_KEY_TOKEN" + case yaml_VALUE_TOKEN: + return "yaml_VALUE_TOKEN" + case yaml_ALIAS_TOKEN: + return "yaml_ALIAS_TOKEN" + case yaml_ANCHOR_TOKEN: + return "yaml_ANCHOR_TOKEN" + case yaml_TAG_TOKEN: + return "yaml_TAG_TOKEN" + case yaml_SCALAR_TOKEN: + return "yaml_SCALAR_TOKEN" + } + return "<unknown token>" +} + +// The token structure. +type yaml_token_t struct { + // The token type. + typ yaml_token_type_t + + // The start/end of the token. + start_mark, end_mark yaml_mark_t + + // The stream encoding (for yaml_STREAM_START_TOKEN). + encoding yaml_encoding_t + + // The alias/anchor/scalar value or tag/tag directive handle + // (for yaml_ALIAS_TOKEN, yaml_ANCHOR_TOKEN, yaml_SCALAR_TOKEN, yaml_TAG_TOKEN, yaml_TAG_DIRECTIVE_TOKEN). + value []byte + + // The tag suffix (for yaml_TAG_TOKEN). + suffix []byte + + // The tag directive prefix (for yaml_TAG_DIRECTIVE_TOKEN). + prefix []byte + + // The scalar style (for yaml_SCALAR_TOKEN). + style yaml_scalar_style_t + + // The version directive major/minor (for yaml_VERSION_DIRECTIVE_TOKEN). + major, minor int8 +} + +// Events + +type yaml_event_type_t int8 + +// Event types. +const ( + // An empty event. + yaml_NO_EVENT yaml_event_type_t = iota + + yaml_STREAM_START_EVENT // A STREAM-START event. + yaml_STREAM_END_EVENT // A STREAM-END event. + yaml_DOCUMENT_START_EVENT // A DOCUMENT-START event. + yaml_DOCUMENT_END_EVENT // A DOCUMENT-END event. + yaml_ALIAS_EVENT // An ALIAS event. + yaml_SCALAR_EVENT // A SCALAR event. + yaml_SEQUENCE_START_EVENT // A SEQUENCE-START event. + yaml_SEQUENCE_END_EVENT // A SEQUENCE-END event. + yaml_MAPPING_START_EVENT // A MAPPING-START event. + yaml_MAPPING_END_EVENT // A MAPPING-END event. +) + +// The event structure. +type yaml_event_t struct { + + // The event type. + typ yaml_event_type_t + + // The start and end of the event. + start_mark, end_mark yaml_mark_t + + // The document encoding (for yaml_STREAM_START_EVENT). + encoding yaml_encoding_t + + // The version directive (for yaml_DOCUMENT_START_EVENT). + version_directive *yaml_version_directive_t + + // The list of tag directives (for yaml_DOCUMENT_START_EVENT). + tag_directives []yaml_tag_directive_t + + // The anchor (for yaml_SCALAR_EVENT, yaml_SEQUENCE_START_EVENT, yaml_MAPPING_START_EVENT, yaml_ALIAS_EVENT). + anchor []byte + + // The tag (for yaml_SCALAR_EVENT, yaml_SEQUENCE_START_EVENT, yaml_MAPPING_START_EVENT). + tag []byte + + // The scalar value (for yaml_SCALAR_EVENT). + value []byte + + // Is the document start/end indicator implicit, or the tag optional? + // (for yaml_DOCUMENT_START_EVENT, yaml_DOCUMENT_END_EVENT, yaml_SEQUENCE_START_EVENT, yaml_MAPPING_START_EVENT, yaml_SCALAR_EVENT). + implicit bool + + // Is the tag optional for any non-plain style? (for yaml_SCALAR_EVENT). + quoted_implicit bool + + // The style (for yaml_SCALAR_EVENT, yaml_SEQUENCE_START_EVENT, yaml_MAPPING_START_EVENT). + style yaml_style_t +} + +func (e *yaml_event_t) scalar_style() yaml_scalar_style_t { return yaml_scalar_style_t(e.style) } +func (e *yaml_event_t) sequence_style() yaml_sequence_style_t { return yaml_sequence_style_t(e.style) } +func (e *yaml_event_t) mapping_style() yaml_mapping_style_t { return yaml_mapping_style_t(e.style) } + +// Nodes + +const ( + yaml_NULL_TAG = "tag:yaml.org,2002:null" // The tag !!null with the only possible value: null. + yaml_BOOL_TAG = "tag:yaml.org,2002:bool" // The tag !!bool with the values: true and false. + yaml_STR_TAG = "tag:yaml.org,2002:str" // The tag !!str for string values. + yaml_INT_TAG = "tag:yaml.org,2002:int" // The tag !!int for integer values. + yaml_FLOAT_TAG = "tag:yaml.org,2002:float" // The tag !!float for float values. + yaml_TIMESTAMP_TAG = "tag:yaml.org,2002:timestamp" // The tag !!timestamp for date and time values. + + yaml_SEQ_TAG = "tag:yaml.org,2002:seq" // The tag !!seq is used to denote sequences. + yaml_MAP_TAG = "tag:yaml.org,2002:map" // The tag !!map is used to denote mapping. + + // Not in original libyaml. + yaml_BINARY_TAG = "tag:yaml.org,2002:binary" + yaml_MERGE_TAG = "tag:yaml.org,2002:merge" + + yaml_DEFAULT_SCALAR_TAG = yaml_STR_TAG // The default scalar tag is !!str. + yaml_DEFAULT_SEQUENCE_TAG = yaml_SEQ_TAG // The default sequence tag is !!seq. + yaml_DEFAULT_MAPPING_TAG = yaml_MAP_TAG // The default mapping tag is !!map. +) + +type yaml_node_type_t int + +// Node types. +const ( + // An empty node. + yaml_NO_NODE yaml_node_type_t = iota + + yaml_SCALAR_NODE // A scalar node. + yaml_SEQUENCE_NODE // A sequence node. + yaml_MAPPING_NODE // A mapping node. +) + +// An element of a sequence node. +type yaml_node_item_t int + +// An element of a mapping node. +type yaml_node_pair_t struct { + key int // The key of the element. + value int // The value of the element. +} + +// The node structure. +type yaml_node_t struct { + typ yaml_node_type_t // The node type. + tag []byte // The node tag. + + // The node data. + + // The scalar parameters (for yaml_SCALAR_NODE). + scalar struct { + value []byte // The scalar value. + length int // The length of the scalar value. + style yaml_scalar_style_t // The scalar style. + } + + // The sequence parameters (for YAML_SEQUENCE_NODE). + sequence struct { + items_data []yaml_node_item_t // The stack of sequence items. + style yaml_sequence_style_t // The sequence style. + } + + // The mapping parameters (for yaml_MAPPING_NODE). + mapping struct { + pairs_data []yaml_node_pair_t // The stack of mapping pairs (key, value). + pairs_start *yaml_node_pair_t // The beginning of the stack. + pairs_end *yaml_node_pair_t // The end of the stack. + pairs_top *yaml_node_pair_t // The top of the stack. + style yaml_mapping_style_t // The mapping style. + } + + start_mark yaml_mark_t // The beginning of the node. + end_mark yaml_mark_t // The end of the node. + +} + +// The document structure. +type yaml_document_t struct { + + // The document nodes. + nodes []yaml_node_t + + // The version directive. + version_directive *yaml_version_directive_t + + // The list of tag directives. + tag_directives_data []yaml_tag_directive_t + tag_directives_start int // The beginning of the tag directives list. + tag_directives_end int // The end of the tag directives list. + + start_implicit int // Is the document start indicator implicit? + end_implicit int // Is the document end indicator implicit? + + // The start/end of the document. + start_mark, end_mark yaml_mark_t +} + +// The prototype of a read handler. +// +// The read handler is called when the parser needs to read more bytes from the +// source. The handler should write not more than size bytes to the buffer. +// The number of written bytes should be set to the size_read variable. +// +// [in,out] data A pointer to an application data specified by +// yaml_parser_set_input(). +// [out] buffer The buffer to write the data from the source. +// [in] size The size of the buffer. +// [out] size_read The actual number of bytes read from the source. +// +// On success, the handler should return 1. If the handler failed, +// the returned value should be 0. On EOF, the handler should set the +// size_read to 0 and return 1. +type yaml_read_handler_t func(parser *yaml_parser_t, buffer []byte) (n int, err error) + +// This structure holds information about a potential simple key. +type yaml_simple_key_t struct { + possible bool // Is a simple key possible? + required bool // Is a simple key required? + token_number int // The number of the token. + mark yaml_mark_t // The position mark. +} + +// The states of the parser. +type yaml_parser_state_t int + +const ( + yaml_PARSE_STREAM_START_STATE yaml_parser_state_t = iota + + yaml_PARSE_IMPLICIT_DOCUMENT_START_STATE // Expect the beginning of an implicit document. + yaml_PARSE_DOCUMENT_START_STATE // Expect DOCUMENT-START. + yaml_PARSE_DOCUMENT_CONTENT_STATE // Expect the content of a document. + yaml_PARSE_DOCUMENT_END_STATE // Expect DOCUMENT-END. + yaml_PARSE_BLOCK_NODE_STATE // Expect a block node. + yaml_PARSE_BLOCK_NODE_OR_INDENTLESS_SEQUENCE_STATE // Expect a block node or indentless sequence. + yaml_PARSE_FLOW_NODE_STATE // Expect a flow node. + yaml_PARSE_BLOCK_SEQUENCE_FIRST_ENTRY_STATE // Expect the first entry of a block sequence. + yaml_PARSE_BLOCK_SEQUENCE_ENTRY_STATE // Expect an entry of a block sequence. + yaml_PARSE_INDENTLESS_SEQUENCE_ENTRY_STATE // Expect an entry of an indentless sequence. + yaml_PARSE_BLOCK_MAPPING_FIRST_KEY_STATE // Expect the first key of a block mapping. + yaml_PARSE_BLOCK_MAPPING_KEY_STATE // Expect a block mapping key. + yaml_PARSE_BLOCK_MAPPING_VALUE_STATE // Expect a block mapping value. + yaml_PARSE_FLOW_SEQUENCE_FIRST_ENTRY_STATE // Expect the first entry of a flow sequence. + yaml_PARSE_FLOW_SEQUENCE_ENTRY_STATE // Expect an entry of a flow sequence. + yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_KEY_STATE // Expect a key of an ordered mapping. + yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_VALUE_STATE // Expect a value of an ordered mapping. + yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_END_STATE // Expect the and of an ordered mapping entry. + yaml_PARSE_FLOW_MAPPING_FIRST_KEY_STATE // Expect the first key of a flow mapping. + yaml_PARSE_FLOW_MAPPING_KEY_STATE // Expect a key of a flow mapping. + yaml_PARSE_FLOW_MAPPING_VALUE_STATE // Expect a value of a flow mapping. + yaml_PARSE_FLOW_MAPPING_EMPTY_VALUE_STATE // Expect an empty value of a flow mapping. + yaml_PARSE_END_STATE // Expect nothing. +) + +func (ps yaml_parser_state_t) String() string { + switch ps { + case yaml_PARSE_STREAM_START_STATE: + return "yaml_PARSE_STREAM_START_STATE" + case yaml_PARSE_IMPLICIT_DOCUMENT_START_STATE: + return "yaml_PARSE_IMPLICIT_DOCUMENT_START_STATE" + case yaml_PARSE_DOCUMENT_START_STATE: + return "yaml_PARSE_DOCUMENT_START_STATE" + case yaml_PARSE_DOCUMENT_CONTENT_STATE: + return "yaml_PARSE_DOCUMENT_CONTENT_STATE" + case yaml_PARSE_DOCUMENT_END_STATE: + return "yaml_PARSE_DOCUMENT_END_STATE" + case yaml_PARSE_BLOCK_NODE_STATE: + return "yaml_PARSE_BLOCK_NODE_STATE" + case yaml_PARSE_BLOCK_NODE_OR_INDENTLESS_SEQUENCE_STATE: + return "yaml_PARSE_BLOCK_NODE_OR_INDENTLESS_SEQUENCE_STATE" + case yaml_PARSE_FLOW_NODE_STATE: + return "yaml_PARSE_FLOW_NODE_STATE" + case yaml_PARSE_BLOCK_SEQUENCE_FIRST_ENTRY_STATE: + return "yaml_PARSE_BLOCK_SEQUENCE_FIRST_ENTRY_STATE" + case yaml_PARSE_BLOCK_SEQUENCE_ENTRY_STATE: + return "yaml_PARSE_BLOCK_SEQUENCE_ENTRY_STATE" + case yaml_PARSE_INDENTLESS_SEQUENCE_ENTRY_STATE: + return "yaml_PARSE_INDENTLESS_SEQUENCE_ENTRY_STATE" + case yaml_PARSE_BLOCK_MAPPING_FIRST_KEY_STATE: + return "yaml_PARSE_BLOCK_MAPPING_FIRST_KEY_STATE" + case yaml_PARSE_BLOCK_MAPPING_KEY_STATE: + return "yaml_PARSE_BLOCK_MAPPING_KEY_STATE" + case yaml_PARSE_BLOCK_MAPPING_VALUE_STATE: + return "yaml_PARSE_BLOCK_MAPPING_VALUE_STATE" + case yaml_PARSE_FLOW_SEQUENCE_FIRST_ENTRY_STATE: + return "yaml_PARSE_FLOW_SEQUENCE_FIRST_ENTRY_STATE" + case yaml_PARSE_FLOW_SEQUENCE_ENTRY_STATE: + return "yaml_PARSE_FLOW_SEQUENCE_ENTRY_STATE" + case yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_KEY_STATE: + return "yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_KEY_STATE" + case yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_VALUE_STATE: + return "yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_VALUE_STATE" + case yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_END_STATE: + return "yaml_PARSE_FLOW_SEQUENCE_ENTRY_MAPPING_END_STATE" + case yaml_PARSE_FLOW_MAPPING_FIRST_KEY_STATE: + return "yaml_PARSE_FLOW_MAPPING_FIRST_KEY_STATE" + case yaml_PARSE_FLOW_MAPPING_KEY_STATE: + return "yaml_PARSE_FLOW_MAPPING_KEY_STATE" + case yaml_PARSE_FLOW_MAPPING_VALUE_STATE: + return "yaml_PARSE_FLOW_MAPPING_VALUE_STATE" + case yaml_PARSE_FLOW_MAPPING_EMPTY_VALUE_STATE: + return "yaml_PARSE_FLOW_MAPPING_EMPTY_VALUE_STATE" + case yaml_PARSE_END_STATE: + return "yaml_PARSE_END_STATE" + } + return "<unknown parser state>" +} + +// This structure holds aliases data. +type yaml_alias_data_t struct { + anchor []byte // The anchor. + index int // The node id. + mark yaml_mark_t // The anchor mark. +} + +// The parser structure. +// +// All members are internal. Manage the structure using the +// yaml_parser_ family of functions. +type yaml_parser_t struct { + + // Error handling + + error yaml_error_type_t // Error type. + + problem string // Error description. + + // The byte about which the problem occurred. + problem_offset int + problem_value int + problem_mark yaml_mark_t + + // The error context. + context string + context_mark yaml_mark_t + + // Reader stuff + + read_handler yaml_read_handler_t // Read handler. + + input_file io.Reader // File input data. + input []byte // String input data. + input_pos int + + eof bool // EOF flag + + buffer []byte // The working buffer. + buffer_pos int // The current position of the buffer. + + unread int // The number of unread characters in the buffer. + + raw_buffer []byte // The raw buffer. + raw_buffer_pos int // The current position of the buffer. + + encoding yaml_encoding_t // The input encoding. + + offset int // The offset of the current position (in bytes). + mark yaml_mark_t // The mark of the current position. + + // Scanner stuff + + stream_start_produced bool // Have we started to scan the input stream? + stream_end_produced bool // Have we reached the end of the input stream? + + flow_level int // The number of unclosed '[' and '{' indicators. + + tokens []yaml_token_t // The tokens queue. + tokens_head int // The head of the tokens queue. + tokens_parsed int // The number of tokens fetched from the queue. + token_available bool // Does the tokens queue contain a token ready for dequeueing. + + indent int // The current indentation level. + indents []int // The indentation levels stack. + + simple_key_allowed bool // May a simple key occur at the current position? + simple_keys []yaml_simple_key_t // The stack of simple keys. + + // Parser stuff + + state yaml_parser_state_t // The current parser state. + states []yaml_parser_state_t // The parser states stack. + marks []yaml_mark_t // The stack of marks. + tag_directives []yaml_tag_directive_t // The list of TAG directives. + + // Dumper stuff + + aliases []yaml_alias_data_t // The alias data. + + document *yaml_document_t // The currently parsed document. +} + +// Emitter Definitions + +// The prototype of a write handler. +// +// The write handler is called when the emitter needs to flush the accumulated +// characters to the output. The handler should write @a size bytes of the +// @a buffer to the output. +// +// @param[in,out] data A pointer to an application data specified by +// yaml_emitter_set_output(). +// @param[in] buffer The buffer with bytes to be written. +// @param[in] size The size of the buffer. +// +// @returns On success, the handler should return @c 1. If the handler failed, +// the returned value should be @c 0. +// +type yaml_write_handler_t func(emitter *yaml_emitter_t, buffer []byte) error + +type yaml_emitter_state_t int + +// The emitter states. +const ( + // Expect STREAM-START. + yaml_EMIT_STREAM_START_STATE yaml_emitter_state_t = iota + + yaml_EMIT_FIRST_DOCUMENT_START_STATE // Expect the first DOCUMENT-START or STREAM-END. + yaml_EMIT_DOCUMENT_START_STATE // Expect DOCUMENT-START or STREAM-END. + yaml_EMIT_DOCUMENT_CONTENT_STATE // Expect the content of a document. + yaml_EMIT_DOCUMENT_END_STATE // Expect DOCUMENT-END. + yaml_EMIT_FLOW_SEQUENCE_FIRST_ITEM_STATE // Expect the first item of a flow sequence. + yaml_EMIT_FLOW_SEQUENCE_ITEM_STATE // Expect an item of a flow sequence. + yaml_EMIT_FLOW_MAPPING_FIRST_KEY_STATE // Expect the first key of a flow mapping. + yaml_EMIT_FLOW_MAPPING_KEY_STATE // Expect a key of a flow mapping. + yaml_EMIT_FLOW_MAPPING_SIMPLE_VALUE_STATE // Expect a value for a simple key of a flow mapping. + yaml_EMIT_FLOW_MAPPING_VALUE_STATE // Expect a value of a flow mapping. + yaml_EMIT_BLOCK_SEQUENCE_FIRST_ITEM_STATE // Expect the first item of a block sequence. + yaml_EMIT_BLOCK_SEQUENCE_ITEM_STATE // Expect an item of a block sequence. + yaml_EMIT_BLOCK_MAPPING_FIRST_KEY_STATE // Expect the first key of a block mapping. + yaml_EMIT_BLOCK_MAPPING_KEY_STATE // Expect the key of a block mapping. + yaml_EMIT_BLOCK_MAPPING_SIMPLE_VALUE_STATE // Expect a value for a simple key of a block mapping. + yaml_EMIT_BLOCK_MAPPING_VALUE_STATE // Expect a value of a block mapping. + yaml_EMIT_END_STATE // Expect nothing. +) + +// The emitter structure. +// +// All members are internal. Manage the structure using the @c yaml_emitter_ +// family of functions. +type yaml_emitter_t struct { + + // Error handling + + error yaml_error_type_t // Error type. + problem string // Error description. + + // Writer stuff + + write_handler yaml_write_handler_t // Write handler. + + output_buffer *[]byte // String output data. + output_file io.Writer // File output data. + + buffer []byte // The working buffer. + buffer_pos int // The current position of the buffer. + + raw_buffer []byte // The raw buffer. + raw_buffer_pos int // The current position of the buffer. + + encoding yaml_encoding_t // The stream encoding. + + // Emitter stuff + + canonical bool // If the output is in the canonical style? + best_indent int // The number of indentation spaces. + best_width int // The preferred width of the output lines. + unicode bool // Allow unescaped non-ASCII characters? + line_break yaml_break_t // The preferred line break. + + state yaml_emitter_state_t // The current emitter state. + states []yaml_emitter_state_t // The stack of states. + + events []yaml_event_t // The event queue. + events_head int // The head of the event queue. + + indents []int // The stack of indentation levels. + + tag_directives []yaml_tag_directive_t // The list of tag directives. + + indent int // The current indentation level. + + flow_level int // The current flow level. + + root_context bool // Is it the document root context? + sequence_context bool // Is it a sequence context? + mapping_context bool // Is it a mapping context? + simple_key_context bool // Is it a simple mapping key context? + + line int // The current line. + column int // The current column. + whitespace bool // If the last character was a whitespace? + indention bool // If the last character was an indentation character (' ', '-', '?', ':')? + open_ended bool // If an explicit document end is required? + + // Anchor analysis. + anchor_data struct { + anchor []byte // The anchor value. + alias bool // Is it an alias? + } + + // Tag analysis. + tag_data struct { + handle []byte // The tag handle. + suffix []byte // The tag suffix. + } + + // Scalar analysis. + scalar_data struct { + value []byte // The scalar value. + multiline bool // Does the scalar contain line breaks? + flow_plain_allowed bool // Can the scalar be expessed in the flow plain style? + block_plain_allowed bool // Can the scalar be expressed in the block plain style? + single_quoted_allowed bool // Can the scalar be expressed in the single quoted style? + block_allowed bool // Can the scalar be expressed in the literal or folded styles? + style yaml_scalar_style_t // The output style. + } + + // Dumper stuff + + opened bool // If the stream was already opened? + closed bool // If the stream was already closed? + + // The information associated with the document nodes. + anchors *struct { + references int // The number of references. + anchor int // The anchor id. + serialized bool // If the node has been emitted? + } + + last_anchor_id int // The last assigned anchor id. + + document *yaml_document_t // The currently emitted document. +} diff --git a/generator/vendor/gopkg.in/yaml.v2/yamlprivateh.go b/generator/vendor/gopkg.in/yaml.v2/yamlprivateh.go new file mode 100644 index 00000000..8110ce3c --- /dev/null +++ b/generator/vendor/gopkg.in/yaml.v2/yamlprivateh.go @@ -0,0 +1,173 @@ +package yaml + +const ( + // The size of the input raw buffer. + input_raw_buffer_size = 512 + + // The size of the input buffer. + // It should be possible to decode the whole raw buffer. + input_buffer_size = input_raw_buffer_size * 3 + + // The size of the output buffer. + output_buffer_size = 128 + + // The size of the output raw buffer. + // It should be possible to encode the whole output buffer. + output_raw_buffer_size = (output_buffer_size*2 + 2) + + // The size of other stacks and queues. + initial_stack_size = 16 + initial_queue_size = 16 + initial_string_size = 16 +) + +// Check if the character at the specified position is an alphabetical +// character, a digit, '_', or '-'. +func is_alpha(b []byte, i int) bool { + return b[i] >= '0' && b[i] <= '9' || b[i] >= 'A' && b[i] <= 'Z' || b[i] >= 'a' && b[i] <= 'z' || b[i] == '_' || b[i] == '-' +} + +// Check if the character at the specified position is a digit. +func is_digit(b []byte, i int) bool { + return b[i] >= '0' && b[i] <= '9' +} + +// Get the value of a digit. +func as_digit(b []byte, i int) int { + return int(b[i]) - '0' +} + +// Check if the character at the specified position is a hex-digit. +func is_hex(b []byte, i int) bool { + return b[i] >= '0' && b[i] <= '9' || b[i] >= 'A' && b[i] <= 'F' || b[i] >= 'a' && b[i] <= 'f' +} + +// Get the value of a hex-digit. +func as_hex(b []byte, i int) int { + bi := b[i] + if bi >= 'A' && bi <= 'F' { + return int(bi) - 'A' + 10 + } + if bi >= 'a' && bi <= 'f' { + return int(bi) - 'a' + 10 + } + return int(bi) - '0' +} + +// Check if the character is ASCII. +func is_ascii(b []byte, i int) bool { + return b[i] <= 0x7F +} + +// Check if the character at the start of the buffer can be printed unescaped. +func is_printable(b []byte, i int) bool { + return ((b[i] == 0x0A) || // . == #x0A + (b[i] >= 0x20 && b[i] <= 0x7E) || // #x20 <= . <= #x7E + (b[i] == 0xC2 && b[i+1] >= 0xA0) || // #0xA0 <= . <= #xD7FF + (b[i] > 0xC2 && b[i] < 0xED) || + (b[i] == 0xED && b[i+1] < 0xA0) || + (b[i] == 0xEE) || + (b[i] == 0xEF && // #xE000 <= . <= #xFFFD + !(b[i+1] == 0xBB && b[i+2] == 0xBF) && // && . != #xFEFF + !(b[i+1] == 0xBF && (b[i+2] == 0xBE || b[i+2] == 0xBF)))) +} + +// Check if the character at the specified position is NUL. +func is_z(b []byte, i int) bool { + return b[i] == 0x00 +} + +// Check if the beginning of the buffer is a BOM. +func is_bom(b []byte, i int) bool { + return b[0] == 0xEF && b[1] == 0xBB && b[2] == 0xBF +} + +// Check if the character at the specified position is space. +func is_space(b []byte, i int) bool { + return b[i] == ' ' +} + +// Check if the character at the specified position is tab. +func is_tab(b []byte, i int) bool { + return b[i] == '\t' +} + +// Check if the character at the specified position is blank (space or tab). +func is_blank(b []byte, i int) bool { + //return is_space(b, i) || is_tab(b, i) + return b[i] == ' ' || b[i] == '\t' +} + +// Check if the character at the specified position is a line break. +func is_break(b []byte, i int) bool { + return (b[i] == '\r' || // CR (#xD) + b[i] == '\n' || // LF (#xA) + b[i] == 0xC2 && b[i+1] == 0x85 || // NEL (#x85) + b[i] == 0xE2 && b[i+1] == 0x80 && b[i+2] == 0xA8 || // LS (#x2028) + b[i] == 0xE2 && b[i+1] == 0x80 && b[i+2] == 0xA9) // PS (#x2029) +} + +func is_crlf(b []byte, i int) bool { + return b[i] == '\r' && b[i+1] == '\n' +} + +// Check if the character is a line break or NUL. +func is_breakz(b []byte, i int) bool { + //return is_break(b, i) || is_z(b, i) + return ( // is_break: + b[i] == '\r' || // CR (#xD) + b[i] == '\n' || // LF (#xA) + b[i] == 0xC2 && b[i+1] == 0x85 || // NEL (#x85) + b[i] == 0xE2 && b[i+1] == 0x80 && b[i+2] == 0xA8 || // LS (#x2028) + b[i] == 0xE2 && b[i+1] == 0x80 && b[i+2] == 0xA9 || // PS (#x2029) + // is_z: + b[i] == 0) +} + +// Check if the character is a line break, space, or NUL. +func is_spacez(b []byte, i int) bool { + //return is_space(b, i) || is_breakz(b, i) + return ( // is_space: + b[i] == ' ' || + // is_breakz: + b[i] == '\r' || // CR (#xD) + b[i] == '\n' || // LF (#xA) + b[i] == 0xC2 && b[i+1] == 0x85 || // NEL (#x85) + b[i] == 0xE2 && b[i+1] == 0x80 && b[i+2] == 0xA8 || // LS (#x2028) + b[i] == 0xE2 && b[i+1] == 0x80 && b[i+2] == 0xA9 || // PS (#x2029) + b[i] == 0) +} + +// Check if the character is a line break, space, tab, or NUL. +func is_blankz(b []byte, i int) bool { + //return is_blank(b, i) || is_breakz(b, i) + return ( // is_blank: + b[i] == ' ' || b[i] == '\t' || + // is_breakz: + b[i] == '\r' || // CR (#xD) + b[i] == '\n' || // LF (#xA) + b[i] == 0xC2 && b[i+1] == 0x85 || // NEL (#x85) + b[i] == 0xE2 && b[i+1] == 0x80 && b[i+2] == 0xA8 || // LS (#x2028) + b[i] == 0xE2 && b[i+1] == 0x80 && b[i+2] == 0xA9 || // PS (#x2029) + b[i] == 0) +} + +// Determine the width of the character. +func width(b byte) int { + // Don't replace these by a switch without first + // confirming that it is being inlined. + if b&0x80 == 0x00 { + return 1 + } + if b&0xE0 == 0xC0 { + return 2 + } + if b&0xF0 == 0xE0 { + return 3 + } + if b&0xF8 == 0xF0 { + return 4 + } + return 0 + +} diff --git a/generator/wg_readme.tmpl b/generator/wg_readme.tmpl index 4af37286..cd9cc2fd 100644 --- a/generator/wg_readme.tmpl +++ b/generator/wg_readme.tmpl @@ -4,23 +4,36 @@ {{ .MissionStatement }} ## Meetings {{- range .Meetings }} -* [{{.Day}}s at {{.UTC}} UTC]({{$.MeetingURL}}) ({{.Frequency}}). [Convert to your timezone](http://www.thetimezoneconverter.com/?t={{.UTC}}&tz=UTC). +* [{{.Day}}s at {{.Time}} {{.TZ}}]({{$.MeetingURL}}) ({{.Frequency}}). [Convert to your timezone](http://www.thetimezoneconverter.com/?t={{.Time}}&tz={{.TZ | tzUrlEncode}}). {{- end }} +{{ if .MeetingArchiveURL -}} Meeting notes and Agenda can be found [here]({{.MeetingArchiveURL}}). +{{- end }} +{{ if .MeetingRecordingsURL -}} Meeting recordings can be found [here]({{.MeetingRecordingsURL}}). +{{- end }} ## Organizers {{- range .Leads }} -* [{{.Name}}](https://github.com/{{.GitHub}}){{if .Company}}, {{.Company}}{{end}} +* {{.Name}} (**[@{{.GitHub}}](https://github.com/{{.GitHub}})**){{if .Company}}, {{.Company}}{{end}} {{- end }} ## Contact * [Slack](https://kubernetes.slack.com/messages/{{.Contact.Slack}}) * [Mailing list]({{.Contact.MailingList}}) -{{if .Contact.FullGitHubTeams}} -## GitHub Teams -{{range .Contact.GithubTeamNames -}} -* [@{{.}}](https://github.com/kubernetes/teams/{{.}}) -{{end}} -{{end}} +{{- if .Label }} +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/wg%2F{{.Label}}) +{{- end }} +{{ if .Contact.GithubTeams }} +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +{{- range .Contact.GithubTeams }} +| @kubernetes/{{.Name}} | [link](https://github.com/orgs/kubernetes/teams/{{.Name}}) | {{.Description}} | +{{- end }} +{{ end }} diff --git a/governance.md b/governance.md index 89732d38..083211fd 100644 --- a/governance.md +++ b/governance.md @@ -6,7 +6,7 @@ The Kubernetes community adheres to the following principles: * Open: Kubernetes is open source. See repository guidelines and CLA, below. * Welcoming and respectful: See Code of Conduct, below. * Transparent and accessible: Work and collaboration should be done in public. See SIG governance, below. -* Merit: Ideas and contributions are accepted according to their technical merit and alignment with project objectives, [scope](http://kubernetes.io/docs/whatisk8s/), and [design principles](contributors/design-proposals/principles.md). +* Merit: Ideas and contributions are accepted according to their technical merit and alignment with project objectives, [scope](http://kubernetes.io/docs/whatisk8s/), and [design principles](contributors/design-proposals/architecture/principles.md). # Code of Conduct @@ -14,9 +14,9 @@ The Kubernetes community abides by the CNCF [code of conduct](https://github.com _As contributors and maintainers of this project, and in the interest of fostering an open and welcoming community, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities._ -As a member of the Kubernetes project, you represent the project and your fellow contributors. +As a member of the Kubernetes project, you represent the project and your fellow contributors. We value our community tremendously and we'd like to keep cultivating a friendly and collaborative -environment for our contributors and users. We want everyone in the community to have +environment for our contributors and users. We want everyone in the community to have [positive experiences](https://www.cncf.io/blog/2016/12/14/diversity-scholarship-series-one-software-engineers-unexpected-cloudnativecon-kubecon-experience). # Community membership @@ -44,7 +44,7 @@ and code ownership, as well as providing focused forums for getting work done, making decisions, and onboarding new contributors. Every identifiable subpart of the project (e.g., github org, repository, subdirectory, API, test, issue, PR) is intended to be owned by some -SIG. +SIG. Areas covered by SIGs may be vertically focused on particular components or functions, cross-cutting/horizontal, spanning many/all @@ -73,7 +73,7 @@ relatively free to customize or change how they operate, within some broad guidelines and constraints imposed by cross-SIG processes (e.g., the release process) and assets (e.g., the kubernetes repo). -A primary reason that SIGs exist is as forums for collaboration. +A primary reason that SIGs exist is as forums for collaboration. Much work in a SIG should stay local within that SIG. However, SIGs must communicate in the open, ensure other SIGs and community members can find notes of meetings, discussions, designs, and decisions, and @@ -137,7 +137,7 @@ community meeting. All repositories under Kubernetes github orgs, such as kubernetes and kubernetes-incubator, should follow the procedures outlined in the [incubator document](incubator.md). All code projects use the [Apache Licence version 2.0](LICENSE). Documentation repositories should use the -[Creative Commons License version 4.0](https://github.com/kubernetes/kubernetes.github.io/blob/master/LICENSE). +[Creative Commons License version 4.0](https://git.k8s.io/website/LICENSE). # Incubator process diff --git a/hack/verify.sh b/hack/verify.sh index 5f855aa7..8eb75982 100755 --- a/hack/verify.sh +++ b/hack/verify.sh @@ -1,38 +1,44 @@ #!/bin/bash -export CRTDIR=$(pwd) -export TMPDIR=/tmp/testgendocs -mkdir $TMPDIR +CRT_DIR=$(pwd) +VERIFY_TEMP=$(mktemp -d 2>/dev/null || mktemp -d -t k8s-community.XXXXXX) +WORKING_DIR=${VERIFY_TEMP}/src/testgendocs +GOPATH=${VERIFY_TEMP} +mkdir -p ${WORKING_DIR} -cp -r sig* Makefile generator $TMPDIR +function cleanup { + rm -rf "${VERIFY_TEMP}" +} +trap cleanup EXIT -cd $TMPDIR +cp -r sigs.yaml sig-* wg-* Makefile generator ${WORKING_DIR}/ + +cd ${WORKING_DIR} make 1>/dev/null mismatches=0 break=$(printf "=%.0s" $(seq 1 68)) -for file in $(ls $CRTDIR/sig-*/README.md $CRTDIR/sig-list.md); do - real=${file#$CRTDIR/} - if ! diff -q $file $TMPDIR/$real &>/dev/null; then - echo "$file does not match $TMPDIR/$real"; +for file in $(ls ${CRT_DIR}/sig-*/README.md ${CRT_DIR}/wg-*/README.md ${CRT_DIR}/sig-list.md); do + real=${file#$CRT_DIR/} + if ! diff -q ${file} ${WORKING_DIR}/${real} &>/dev/null; then + echo "${file} does not match ${WORKING_DIR}/${real}"; mismatches=$((mismatches+1)) fi; done -if [ $mismatches -gt "0" ]; then +if [ ${mismatches} -gt "0" ]; then echo "" - echo $break + echo ${break} noun="mismatch was" - if [ $mismatches -gt "0" ]; then + if [ ${mismatches} -gt "0" ]; then noun="mismatches were" fi - echo "$mismatches $noun detected." + echo "${mismatches} ${noun} detected." echo "Do not manually edit sig-list.md or README.md files inside the sig folders." echo "Instead make your changes to sigs.yaml and then run \`make\`."; - echo $break + echo ${break} exit 1; fi -rm -rf $TMPDIR exit 0 diff --git a/incubator.md b/incubator.md index 96279b05..506d11cd 100644 --- a/incubator.md +++ b/incubator.md @@ -22,25 +22,25 @@ New Kubernetes Community Projects can be created by pulling out code from the Ku ### Applications on top of Kubernetes -If you are building an Open Source application on top of Kubernetes that is unrelated to the development, operation, monitoring, APIs, lifecycle, integration, or other primary concerns of the Kubernetes project there is no reason to become a Kubernetes project. If you are unsure if your project is a good fit try running it by a few Kubernetes Special Interest Groups or kubernetes-dev@googlegroups.com. +If you are building an Open Source application on top of Kubernetes that is unrelated to the development, operation, monitoring, APIs, lifecycle, integration, or other primary concerns of the Kubernetes project there is no reason to become a Kubernetes project. If you are unsure that your project is a good fit try running it by a few Kubernetes Special Interest Groups or kubernetes-dev@googlegroups.com. ## Entering Incubation To create a new project for incubation you must follow these steps: write a proposal, find a champion, gain acceptance into a SIG, and finally get approval of a Sponsor. The Sponsor and Champion cannot be the same person. -Your proposal should include two items. First, a README which outlines the problem to be solved, an example use case as-if the project existed, and a rough roadmap with timelines. Second, an OWNERS file that outlines the makeup of the initial team developing the project. Initially this can be one person but ideally has a 3 or more initial developers representing a few different companies or groups. You can use whatever tool you want to host and revise the proposal until the project is accepted to the Incubator: copy/paste from your text editor, Google Docs, GitHub gist, etc. +Your proposal should include two items. First, a README which outlines the problem to be solved, an example use case as-if the project existed, and a rough roadmap with timelines. Second, an OWNERS file that outlines the makeup of the initial team developing the project. Initially this can be one person but ideally has 3 or more initial developers representing a few different companies or groups. You can use whatever tool you want to host and revise the proposal until the project is accepted to the Incubator: copy/paste from your text editor, Google Docs, GitHub gist, etc. -Once the proposal is written you should identify a champion; this person must be listed as either a reviewer or approver in an OWNER file in the [Kubernetes project](https://github.com/kubernetes/kubernetes). Next, reach out to your potential champion via email to ask if they are interested in helping you through the Incubation process. Ideally some significant follow-up discussion happens via email, calls, or chat to improve the proposal before announcing it to the wider community. +Once the proposal is written you should identify a champion; this person must be listed as either a reviewer or approver in an [OWNERS file](https://git.k8s.io/kubernetes/OWNERS) in the Kubernetes project. Next, reach out to your potential champion via email to ask if they are interested in helping you through the Incubation process. Ideally some significant follow-up discussion happens via email, calls, or chat to improve the proposal before announcing it to the wider community. The next discussion should be on a relevant Special Interest Group mailing list. You should post the proposal to the SIG mailing list and wait for discussion for a few days. Iterate on the proposal as needed and if there is rough consensus that the project belongs in the chosen SIG then list that SIG in the proposal README. If consensus isn't reached then identify another SIG and try again; repeat until a SIG is identified. The final process is to email kubernetes-dev@googlegroups.com to announce your intention to form a new Incubator project. Include your entire proposal in the body of the email and prefix the Subject with [Incubator]. Include links to your discussion on the accepted SIG mailing list to guide the discussion. -Acceptance of the project into the Kubernetes Incubator happens once a Sponsor approves. Anyone listed as an approver in the top-level pkg OWNERS file https://github.com/kubernetes/kubernetes/blob/master/pkg/OWNERS can sponsor a project by replying to the kubernetes-dev discussion with LGTM. +Acceptance of the project into the Kubernetes Incubator happens once a Sponsor approves. Anyone listed as an approver in the top-level pkg [OWNERS file](https://git.k8s.io/kubernetes/pkg/OWNERS) can sponsor a project by replying to the kubernetes-dev discussion with LGTM. ## Creation of the Incubator Project -Once your project is accepted someone will create a new GitHub repo for your project to use in the github.com/kubernetes-incubator organization. The first order of business is to add the following files to the repo: +Once your project is accepted someone will create a new GitHub repo for your project to use in the [kubernetes-incubator](https://github.com/kubernetes-incubator) organization. The first order of business is to add the following files to the repo: - a README from the accepted proposal - an OWNERS from the accepted proposal @@ -49,7 +49,7 @@ Once your project is accepted someone will create a new GitHub repo for your pro - a LICENSE file with the Apache 2.0 license - a RELEASE.md file that talks about the process for releases -A template project to start your project with these files can be found here: https://github.com/kubernetes/kubernetes-template-project +To start your project with these files can be found here at [template project](https://github.com/kubernetes/kubernetes-template-project) ## Exiting Incubation @@ -110,7 +110,7 @@ These are grandfathered in as full projects: - github.com/kubernetes/community - github.com/kubernetes/release - github.com/kubernetes/features -- github.com/kubernetes/md-check +- github.com/kubernetes/kube-state-metrics - github.com/kubernetes/pr-bot - move from mungebot, etc from contrib, currently running in "prod" on github.com/kubernetes - github.com/kubernetes/dashboard - github.com/kubernetes/helm (Graduated from incubator on Feb 2017) @@ -141,7 +141,7 @@ These projects are young but have significant user facing docs pointing at their - github.com/kubernetes/contrib - new projects could be created from the code in this repo, issue to move - github.com/kubernetes/kubedash - github.com/kubernetes/kubernetes-docs-cn -- github.com/kubernetes/kube-state-metrics +- github.com/kubernetes/md-check ## Thank You diff --git a/keps/0000-kep-template.md b/keps/0000-kep-template.md new file mode 100644 index 00000000..4494b7b3 --- /dev/null +++ b/keps/0000-kep-template.md @@ -0,0 +1,158 @@ +--- +kep-number: draft-YYYYMMDD +title: My First KEP +authors: + - "@janedoe" +owning-sig: sig-xxx +participating-sigs: + - sig-aaa + - sig-bbb +reviewers: + - TBD + - "@alicedoe" +approvers: + - TBD + - "@oscardoe" +editor: TBD +creation-date: yyyy-mm-dd +last-updated: yyyy-mm-dd +status: draft +see-also: + - KEP-1 + - KEP-2 +replaces: + - KEP-3 +superseded-by: + - KEP-100 +--- + +# Title + +This is the title of the KEP. +Keep it simple and descriptive. +A good title can help communicate what the KEP is and should be considered as part of any review. + +The *filename* for the KEP should include the KEP number along with the title. +The title should be lowercased and spaces/punctuation should be replaced with `-`. +As the KEP is approved and an official KEP number is allocated, the file should be renamed. + +To get started with this template: +* Make a copy in the appropriate directory. + Name it `draft-YYYYMMDD-my-title.md`. +* Create a PR in the [`kubernetes/community`](https://github.com/kubernetes/community) repo. +* Check in early. + Do this once the document holds together and general direction is understood by many in the sponsoring SIG. + View anything marked as a draft as a working document. + Aim for single topic PRs to keep discussions focused. + If you disagree with what is already in a document, open a new PR with suggested changes. +* As a KEP is approved, rename the file yet again with the final KEP number. + +The canonical place for the latest set of instructions (and the likely source of this file) is [here](/keps/0000-kep-template.md). + +The `Metadata` section above is intended to support the creation of tooling around the KEP process. +This will be a YAML section that is fenced as a code block. +See the KEP process for details on each of these items. + +## Table of Contents + +A table of contents is helpful for quickly jumping to sections of a KEP and for highlighting any additional information provided beyond the standard KEP template. +[Tools for generating][] a table of contents from markdown are available. + +* [Table of Contents](#table-of-contents) +* [Summary](#summary) +* [Motivation](#motivation) + * [Goals](#goals) + * [Non-Goals](#non-goals) +* [Proposal](#proposal) + * [User Stories [optional]](#user-stories-optional) + * [Story 1](#story-1) + * [Story 2](#story-2) + * [Implementation Details/Notes/Constraints [optional]](#implementation-detailsnotesconstraints-optional) + * [Risks and Mitigations](#risks-and-mitigations) +* [Graduation Criteria](#graduation-criteria) +* [Implementation History](#implementation-history) +* [Drawbacks [optional]](#drawbacks-optional) +* [Alternatives [optional]](#alternatives-optional) + +[Tools for generating]: https://github.com/ekalinin/github-markdown-toc + +## Summary + +The `Summary` section is incredibly important for producing high quality user focused documentation such as release notes or a development road map. +It should be possible to collect this information before implementation begins in order to avoid requiring implementors to split their attention between writing release notes and implementing the feature itself. +KEP editors, SIG Docs, and SIG PM should help to ensure that the tone and content of the `Summary` section is useful for a wide audience. + +A good summary is probably at least a paragraph in length. + +## Motivation + +This section is for explicitly listing the motivation, goals and non-goals of this KEP. +Describe why the change is important and the benefits to users. +The motivation section can optionally provide links to [experience reports][] to demonstrate the interest in a KEP within the wider Kubernetes community. + +[experience reports]: https://github.com/golang/go/wiki/ExperienceReports + +### Goals + +List the specific goals of the KEP. +How will we know that this has succeeded? + +### Non-Goals + +What is out of scope for his KEP? +Listing non-goals helps to focus discussion and make progress. + +## Proposal + +This is where we get down to the nitty gritty of what the proposal actually is. + +### User Stories [optional] + +Detail the things that people will be able to do if this KEP is implemented. +Include as much detail as possible so that people can understand the "how" of the system. +The goal here is to make this feel real for users without getting bogged down. + +#### Story 1 + +#### Story 2 + +### Implementation Details/Notes/Constraints [optional] + +What are the caveats to the implementation? +What are some important details that didn't come across above. +Go in to as much detail as necessary here. +This might be a good place to talk about core concepts and how they releate. + +### Risks and Mitigations + +What are the risks of this proposal and how do we mitigate. +Think broadly. +For example, consider both security security and how this will impact the larger kubernetes ecosystem. + +## Graduation Criteria + +How will we know that this has succeeded? +Gathering user feedback is crucial for building high quality experiences and SIGs have the important responsibility of setting milestones for stability and completeness. +Hopefully the content previously contained in [umbrella issues][] will be tracked in the `Graduation Criteria` section. + +[umbrella issues]: https://github.com/kubernetes/kubernetes/issues/42752 + +## Implementation History + +Major milestones in the life cycle of a KEP should be tracked in `Implementation History`. +Major milestones might include + +- the `Summary` and `Motivation` sections being merged signaling SIG acceptance +- the `Proposal` section being merged signaling agreement on a proposed design +- the date implementation started +- the first Kubernetes release where an initial version of the KEP was available +- the version of Kubernetes where the KEP graduated to general availability +- when the KEP was retired or superseded + +## Drawbacks [optional] + +Why should this KEP _not_ be implemented. + +## Alternatives [optional] + +Similar to the `Drawbacks` section the `Alternatives` section is used to highlight and record other possible approaches to delivering the value proposed by a KEP. diff --git a/keps/1-kubernetes-enhancement-proposal-process.md b/keps/1-kubernetes-enhancement-proposal-process.md new file mode 100644 index 00000000..6f808402 --- /dev/null +++ b/keps/1-kubernetes-enhancement-proposal-process.md @@ -0,0 +1,458 @@ +# Kubernetes Enhancement Proposal Process + +## Metadata +``` +--- +kep-number: 1 +title: Kubernetes Enhancement Proposal Process +authors: + - name: Caleb Miles + github: calebamiles + slack: calebamiles + - name: Joe Beda + github: jbeda + email: joe@heptio.com + slack: jbeda +owning-sig: sig-architecture +participating-sigs: + - `kubernetes-wide` +reviewers: + - name: TBD +approvers: + - name: TBD +editor: + name: TBD +creation-date: 2017-08-22 +status: draft +``` + +## Table of Contents + +* [Kubernetes Enhancement Proposal Process](#kubernetes-enhancement-proposal-process) + * [Metadata](#metadata) + * [Table of Contents](#table-of-contents) + * [Summary](#summary) + * [Motivation](#motivation) + * [Reference-level explanation](#reference-level-explanation) + * [What type of work should be tracked by a KEP](#what-type-of-work-should-be-tracked-by-a-kep) + * [KEP Template](#kep-template) + * [KEP Metadata](#kep-metadata) + * [KEP Workflow](#kep-workflow) + * [Git and GitHub Implementation](#git-and-github-implementation) + * [KEP Editor Role](#kep-editor-role) + * [Important Metrics](#important-metrics) + * [Prior Art](#prior-art) + * [Graduation Criteria](#graduation-criteria) + * [Drawbacks](#drawbacks) + * [Alternatives](#alternatives) + * [Unresolved Questions](#unresolved-questions) + * [Mentors](#mentors) + +## Summary + +A standardized development process for Kubernetes is proposed in order to + +- provide a common structure for proposing changes to Kubernetes +- ensure that the motivation for a change is clear +- allow for the enumeration stability milestones and stability graduation + criteria +- persist project information in a Version Control System (VCS) for future + Kubernauts +- support the creation of _high value user facing_ information such as: + - an overall project development roadmap + - motivation for impactful user facing changes +- support development across multiple repositories beyond `kubernetes/kubernetes` +- reserve GitHub issues for tracking work in flight rather than creating "umbrella" + issues +- ensure community participants are successfully able to drive changes to + completion across one or more releases while stakeholders are adequately + represented throughout the process + +This process is supported by a unit of work called a Kubernetes Enhancement +Proposal or KEP. A KEP attempts to combine aspects of a + +- feature, and effort tracking document +- a product requirements document +- design document + +into one file which is created incrementally in collaboration with one or more +Special Interest Groups (SIGs). + +## Motivation + +For cross project SIGs such as SIG PM and SIG Release an abstraction beyond a +single GitHub Issue or Pull request seems to be required in order to understand +and communicate upcoming changes to Kubernetes. In a blog post describing the +[road to Go 2][], Russ Cox explains + +> that it is difficult but essential to describe the significance of a problem +> in a way that someone working in a different environment can understand + +as a project it is vital to be able to track the chain of custody for a proposed +enhancement from conception through implementation. This proposal does not +attempt to mandate how SIGs track their work internally, however, it is +suggested that SIGs which do not adhere to a process which allows for their hard +work to be explained to others in the wider Kubernetes community will see their +work wallow in the shadows of obscurity. At the very least [survey data][] +suggest that high quality documentation is crucial to project adoption. +Documentation can take many forms and it is imperative to ensure that it is easy +to produce high quality user or developer focused documentation for a complex +project like Kubernetes. + +Without a standardized mechanism for describing important enhancements our +talented technical writers and product managers struggle to weave a coherent +narrative explaining why a particular release is important. Additionally for +critical infrastructure such as Kubernetes adopters need a forward looking road +map in order to plan their adoption strategy. + +The purpose of the KEP process is to reduce the amount of "tribal knowledge" in +our community. By moving decisions from a smattering of mailing lists, video +calls and hallway conversations into a well tracked artifact this process aims +to enhance communication and discoverability. + +A KEP is broken into sections which can be merged into source control +incrementally in order to support an iterative development process. An important +goal of the KEP process is ensuring that the process for submitting the content +contained in [design proposals][] is both clear and efficient. The KEP process +is intended to create high quality uniform design and implementation documents +for SIGs to deliberate. + +[tell a story]: https://blog.rust-lang.org/2017/08/31/Rust-1.20.html +[road to Go 2]: https://blog.golang.org/toward-go2 +[survey data]: http://opensourcesurvey.org/2017/ +[design proposals]: /contributors/design-proposals + + +## Reference-level explanation + +### What type of work should be tracked by a KEP + +The definition of what constitutes an "enhancement" is a foundational concern +for the Kubernetes project. Roughly any Kubernetes user or operator facing +enhancement should follow the KEP process: if an enhancement would be described +in either written or verbal communication to anyone besides the KEP author or +developer then consider creating a KEP. One concrete example is an enhancement +which should be communicated to SIG Release or SIG PM. + +Similarly, any technical effort (refactoring, major architectural change) that +will impact a large section of the development community should also be +communicated widely. The KEP process is suited for this even if it will have +zero impact on the typical user or operator. + +As the local bodies of governance, SIGs should have broad latitude in describing +what constitutes an enhancement which should be tracked through the KEP process. +SIGs may find that helpful to enumerate what _does not_ require a KEP rather +than what does. SIGs also have the freedom to customize the KEP template +according to their SIG specific concerns. For example the KEP template used to +track API changes will likely have different subsections than the template for +proposing governance changes. However, as changes start impacting other SIGs or +the larger developer community outside of a SIG, the KEP process should be used +to coordinate and communicate. + +### KEP Template + +The template for a KEP is precisely defined in the [template proposal][] + +[template proposal]: https://github.com/kubernetes/community/pull/1124 + +### KEP Metadata + +There is a place in each KEP for a YAML document that has standard metadata. +This will be used to support tooling around filtering and display. It is also +critical to clearly communicate the status of a KEP. + +Metadata items: +* **kep-number** Required + * Each proposal has a number. This is to make all references to proposals as + clear as possible. This is especially important as we create a network + cross references between proposals. + * Before having the `Approved` status, the number for the KEP will be in the + form of `draft-YYYYMMDD`. The `YYYYMMDD` is replaced with the current date + when first creating the KEP. The goal is to enable fast parallel merges of + pre-acceptance KEPs. + * On acceptance a sequential dense number will be assigned. This will be done + by the editor and will be done in such a way as to minimize the chances of + conflicts. The final number for a KEP will have no prefix. +* **title** Required + * The title of the KEP in plain language. The title will also be used in the + KEP filename. See the template for instructions and details. +* **status** Required + * The current state of the KEP. + * Must be one of `Draft`, `Deferred`, `Approved`, `Rejected`, `Withdrawn`, + `Final`, `Replaced`. +* **authors** Required + * A list of authors for the KEP. We require a name (which can be a psuedonym) + along with a github ID. Other ways to contact the author is strongly + encouraged. This is a list of maps. Subkeys of each item: `name`, + `github`, `email` (optional), `slack` (optional). +* **owning-sig** Required + * The SIG that is most closely associated with this KEP. If there is code or + other artifacts that will result from this KEP, then it is expected that + this SIG will take responsiblity for the bulk of those artifacts. + * Sigs are listed as `sig-abc-def` where the name matches up with the + directory in the `kubernetes/community` repo. +* **participating-sigs** Optional + * A list of SIGs that are involved or impacted by this KEP. + * A special value of `kubernetes-wide` will indicate that this KEP has impact + across the entire project. +* **reviewers** Required + * Reviewer(s) chosen after triage according to proposal process + * If not yet chosen replace with `TBD` + * Same name/contact scheme as `authors` +* **approvers** Required + * Approver(s) chosen after triage according to proposal process + * If not yet chosen replace with `TBD` + * Same name/contact scheme as `authors` +* **editor** Required + * Someone to keep things moving forward. + * If not yet chosen replace with `TBD` + * Same name/contact scheme as `authors` +* **creation-date** Required + * The date that the KEP was first submitted in a PR. + * In the form `yyyy-mm-dd` + * While this info will also be in source control, it is helpful to have the set of KEP files stand on their own. +* **last-updated** Optional + * The date that the KEP was last changed significantly. + * In the form `yyyy-mm-dd` +* **see-also** Optional + * A list of other KEPs that are relevant to this KEP. + * In the form `KEP-123` +* **replaces** Optional + * A list of KEPs that this KEP replaces. Those KEPs should list this KEP in + their `superseded-by`. + * In the form `KEP-123` +* **superseded-by** + * A list of KEPs that supersede this KEP. Use of this should be paired with + this KEP moving into the `Replaced` status. + * In the form `KEP-123` + + +### KEP Workflow + +TODO(jbeda) Rationalize this with status entires in the Metadata above. + +A KEP is proposed to have the following states + +- **opened**: a new KEP has been filed but not triaged by the responsible SIG or + working group +- **accepted**: the motivation has been accepted by the SIG or working group as in + road map +- **scoped**: the design has been approved by the SIG or working group +- **started**: the implementation of the KEP has begun +- **implemented**: the implementation of the KEP is complete +- **deferred**: the KEP has been postponed by the SIG or working group despite + agreement on the motivation +- **superseded**: the KEP has been superseded by another KEP +- **retired**: the implementation of the KEP has been removed +- **rejected**: the KEP has been rejected by the SIG or working group +- **orphaned**: the author or developer of the KEP is no longer willing or able + to complete implementation + +with possible paths through the state space + +- opened -> deferred (a) +- opened -> rejected (b) +- opened -> orphaned (c) +- opened -> accepted -> orphaned (d) +- opened -> accepted -> scoped -> superseded (e) +- opened -> accepted -> scoped -> orphaned (f) +- opened -> accepted -> scoped -> started -> retired (g) +- opened -> accepted -> scoped -> started -> orphaned (h) +- opened -> accepted -> scoped -> started -> superseded (i) +- opened -> accepted -> scoped -> started -> implemented (j) +- opened -> accepted -> scoped -> started -> implemented -> retired (k) + +the happy path is denoted by (j) where an KEP is opened; accepted by a SIG as in +their roadmap; fleshed out with a design; started; and finally implemented. As +Kubernetes continues to mature, hopefully metrics on the utilization of features +will drive decisions on what features to maintain and which to deprecate and so +it is possible that a KEP would be retired if its functionality no longer provides +sufficient value to the community. + +### Git and GitHub Implementation + +Practically an KEP would be implemented as a pull request to a central repository +with the following example structure + +``` +├── 0000-kep-template.md +├── CODEOWNERS +├── index.md +├── sig-architecture +│ ├── deferred +│ ├── orphaned +│ └── retired +├── sig-network +│ ├── deferred +│ ├── kube-dns +│ ├── orphaned +│ └── retired +├── sig-node +│ ├── deferred +│ ├── kubelet +│ ├── orphaned +│ └── retired +├── sig-release +│ ├── deferred +│ ├── orphaned +│ └── retired +├── sig-storage +│ ├── deferred +│ ├── orphaned +│ └── retired +├── unsorted-to-be-used-by-newcomers-only +└── wg-resource-management + ├── deferred + ├── orphaned + └── retired +``` + +where each SIG or working group is given a top level directory with subprojects +maintained by the SIG listed in sub directories. For newcomers to the community +an `unsorted-to-be-used-by-newcomers-only` directory may be used before an KEP +can be properly routed to a SIG although hopefully if discussion for a potential +KEP begins on the mailing lists proper routing information will be provided to +the KEP author. Additionally a top level index of KEPs may be helpful for people +looking for a complete list of KEPs. There should be basic CI to ensure that an +`index.md` remains up to date. + +Ideally no work would begin within the repositories of the Kubernetes organization +before a KEP has been approved by the responsible SIG or working group. While the +details of how SIGs organize their work is beyond the scope of this proposal one +possibility would be for each charter SIG to create a top level repository within +the Kubernetes org where implementation issues managed by that SIG would be filed. + +### KEP Editor Role + +Taking a cue from the [Python PEP process][], I believe that a group of KEP editors +will be required to make this process successful; the job of an KEP editor is +likely very similar to the [PEP editor responsibilities][] and will hopefully +provide another opportunity for people who do not write code daily to contribute +to Kubernetes. + +In keeping with the PEP editors which + +> Read the PEP to check if it is ready: sound and complete. The ideas must make +> technical sense, even if they don't seem likely to be accepted. +> The title should accurately describe the content. +> Edit the PEP for language (spelling, grammar, sentence structure, etc.), markup +> (for reST PEPs), code style (examples should match PEP 8 & 7). + +KEP editors should generally not pass judgement on a KEP beyond editorial +corrections. + +[Python PEP process]: https://www.python.org/dev/peps/pep-0001/ +[PEP editor responsibilities]: https://www.python.org/dev/peps/pep-0001/#pep-editor-responsibilities-workflow + +### Important Metrics + +It is proposed that the primary metrics which would signal the success or +failure of the KEP process are + +- how many "features" are tracked with a KEP +- distribution of time a KEP spends in each state +- KEP rejection rate +- PRs referencing a KEP merged per week +- number of issued open which reference a KEP +- number of contributors who authored a KEP +- number of contributors who authored a KEP for the first time +- number of orphaned KEPs +- number of retired KEPs +- number of superseded KEPs + +### Prior Art + +The KEP process as proposed was essentially stolen from the [Rust RFC process] which +itself seems to be very similar to the [Python PEP process][] + +[Rust RFC process]: https://github.com/rust-lang/rfcs + +## Graduation Criteria + +should hit at least the following milestones + +- a release note draft can be generated by referring primarily to KEP content +- a yearly road map is expressed as a KEP + +## Drawbacks + +Any additional process has the potential to engender resentment within the +community. There is also a risk that the KEP process as designed will not +sufficiently address the scaling challenges we face today. PR review bandwidth is +already at a premium and we may find that the KEP process introduces an unreasonable +bottleneck on our development velocity. + +It certainly can be argued that the lack of a dedicated issue/defect tracker +beyond GitHub issues contributes to our challenges in managing a project as large +as Kubernetes, however, given that other large organizations, including GitHub +itself, make effective use of GitHub issues perhaps the argument is overblown. + +The centrality of Git and GitHub within the KEP process also may place too high +a barrier to potential contributors, however, given that both Git and GitHub are +required to contribute code changes to Kubernetes today perhaps it would be reasonable +to invest in providing support to those unfamiliar with this tooling. + +Expanding the proposal template beyond the single sentence description currently +required in the [features issue template][] may be a heavy burden for non native +English speakers and here the role of the KEP editor combined with kindness and +empathy will be crucial to making the process successful. + +[features issue template]: https://git.k8s.io/features/ISSUE_TEMPLATE.md + +## Alternatives + +This KEP process is related to +- the generation of a [architectural roadmap][] +- the fact that the [what constitutes a feature][] is still undefined +- [issue management][] +- the difference between an [accepted design and a proposal][] +- [the organization of design proposals][] + +this proposal attempts to place these concerns within a general framework. + +[architectural roadmap]: https://github.com/kubernetes/community/issues/952 +[what constitutes a feature]: https://github.com/kubernetes/community/issues/531 +[issue management]: https://github.com/kubernetes/community/issues/580 +[accepted design and a proposal]: https://github.com/kubernetes/community/issues/914 +[the organization of design proposals]: https://github.com/kubernetes/community/issues/918 + +### Github issues vs. KEPs + +The use of GitHub issues when proposing changes does not provide SIGs good +facilities for signaling approval or rejection of a proposed change to Kubernetes +since anyone can open a GitHub issue at any time. Additionally managing a proposed +change across multiple releases is somewhat cumbersome as labels and milestones +need to be updated for every release that a change spans. These long lived GitHub +issues lead to an ever increasing number of issues open against +`kubernetes/features` which itself has become a management problem. + +In addition to the challenge of managing issues over time, searching for text +within an issue can be challenging. The flat hierarchy of issues can also make +navigation and categorization tricky. While not all community members might +not be comfortable using Git directly, it is imperative that as a community we +work to educate people on a standard set of tools so they can take their +experience to other projects they may decide to work on in the future. While +git is a fantastic version control system (VCS), it is not a project management +tool nor a cogent way of managing an architectural catalog or backlog; this +proposal is limited to motivating the creation of a standardized definition of +work in order to facilitate project management. This primitive for describing +a unit of work may also allow contributors to create their own personalized +view of the state of the project while relying on Git and GitHub for consistency +and durable storage. + +## Unresolved Questions + +- How reviewers and approvers are assigned to a KEP +- Approval decision process for a KEP +- Example schedule, deadline, and time frame for each stage of a KEP +- Communication/notification mechanisms +- Review meetings and escalation procedure +- Decision on where development should occur + +## Mentors + +- caleb miles + - github: [calebamiles](https://github.com/calebamiles/) + - slack: [calebamiles](https://coreos.slack.com/team/caleb.miles) + - email: [caleb.miles@coreos.com](mailto:caleb.miles@coreos.com) + - pronoun: "he" diff --git a/keps/OWNERS b/keps/OWNERS new file mode 100644 index 00000000..77efac8a --- /dev/null +++ b/keps/OWNERS @@ -0,0 +1,16 @@ +reviewers: + - sig-architecture-leads + - jbeda + - bgrant0607 + - jdumars + - calebamiles + - idvoretskyi +approvers: + - sig-architecture-leads + - jbeda + - bgrant0607 + - jdumars + - calebamiles + - idvoretskyi +labels: + - sig/architecture diff --git a/keps/README.md b/keps/README.md new file mode 100644 index 00000000..72487d8b --- /dev/null +++ b/keps/README.md @@ -0,0 +1,19 @@ +# Kubernetes Enhancement Proposals (KEPs) + +This directory contains both the template and process for KEPs. + +This process is still in an _alpha_ state and is opt-in for those that want to provide feedback for the process. + +## Directory structure + +Most KEPs will be sponsored out of a specific SIG. Those can be incubated inside of a subdirectory here for that SIG. SIGs can add those directories and OWNERs files on an as needed basis. + +Any KEPs that cross many SIG boundaries can be created at the root of the KEP directory. + +## KEP numbering + +The next dense number KEP will be 2. + +To assign a number create a commit that both renames the KEP file and updates this file. + +In process KEPs can use a date based numbering scheme. See [KEP-1](1-kubernetes-enhancement-proposal-process.md) for details of this process.
\ No newline at end of file diff --git a/mentoring/OWNERS b/mentoring/OWNERS new file mode 100644 index 00000000..a6101b13 --- /dev/null +++ b/mentoring/OWNERS @@ -0,0 +1,7 @@ +reviewers: + - parispittman +approvers: + - parispittman + - sig-contributor-experience-leads +labels: + - sig/contributor-experience diff --git a/mentoring/README.md b/mentoring/README.md new file mode 100644 index 00000000..4423d6c4 --- /dev/null +++ b/mentoring/README.md @@ -0,0 +1,33 @@ +# Kubernetes Mentoring Initiatives + +This folder will be used for all mentoring initiatives for Kubernetes. + +--- +## Kubernetes Pilots + +We understand that everyone has different learning styles and we want to support as many of those as possible. Mentoring is vital to the growth of an individual and organization of every kind. For Kubernetes, the larger the project becomes, it's necessary to keep a continuous pipeline of quality contributors. + +*What's a Pilot?* +A pilot is a Kubernetes mentor helping new and current members navigate the seas of our repos. + +## Current mentoring activities: +All are currently in an incubation phase. Please reach out to Paris Pittman (parispittman@google.com or Paris on Kubernetes slack channel) for more information on how to get involved. The preliminary deck for mentoring strategies is [here](https://docs.google.com/presentation/d/1bRjDEPEn3autWzaEFirbLfHagbZV04Q9kTCalYmnnXw/edit?usp=sharing0). + +[Contributor Office Hours](/community/office-hours.md) +[Group Mentoring Cohorts](/mentoring/group-mentoring.md) +[Outreachy](/sig-cli/outreachy.md) + +#### Inspiration and Thanks +This is not an out of the box program but was largely inspired by the following: +* [Ada Developer Academy](https://adadevelopersacademy.org/) +* [Google Summer of Code](https://developers.google.com/open-source/gsoc/) +* [exercism.io](https://github.com/OperationCode/exercism-io-mentoring) +* [OpenStack Mentoring](https://wiki.openstack.org/wiki/Mentoring) +* [Apache Mentoring Programme](https://community.apache.org/mentoringprogramme.html) + +Thanks to: +* the many contributors who reviewed and participated in brainstorming, +* founding mentees for their willingness to try this out, +* founding Pilots (@chrislovecnm, @luxas, @kow3ns) + +We welcome PRs, suggestions, and help! diff --git a/mentoring/group-mentee-guide.md b/mentoring/group-mentee-guide.md new file mode 100644 index 00000000..abb41cb8 --- /dev/null +++ b/mentoring/group-mentee-guide.md @@ -0,0 +1,77 @@ +# Mentee Guide for Group Mentoring +This doc is a work in progress + +## Welcome and Summary +Welcome prospective mentee! This is an experimental group mentoring program where individuals will work self-paced in a semi-structured learning environment over the course of three months. Want to get lgtm rights? Be a decision maker in an OWNERS file? This could be a great way to get the knowledge you need, some accountability to do it, and interact with active contributors who may on the same path as you and our experienced Kubernetes developers. + +During the three month period, you will work on a path to the next level of membership which will take some work along with peers who have the same goal. This cohort will include mentors who are at least the level that you are targeting with ample experience in the project. These mentors will counsel but not provide answers to the problem or the solution itself. Think of them as true pilots of the sea - navigational experts - only to our codebase. At the end of the three month period, with open communication, the decision to accept or not proceed with the advancement should not be a surprise. + +Familiarize yourself with the [community membership requirements doc](/community-membership.md) including the requirements needed for your specific journey. You’ll receive sponsorship from the mentors and other people you've worked on issues/PRs with upon successful completion of the program. + +## Expectations +* You understand the requirements to become $member_role (ex: member->reviewer) +* Good standing throughout the program and uphold our code of conduct +* Attend two bi-weekly standups/workshops a month (1 hour each) +* The program is at your pace but these meetings will discuss your wins and blocks +* These meetings will also cover development area topics, suggested activities, and groomed issues +* Be helpful to your peers in the cohort +* Give back and be a mentor in a future cohort +* Mutual respect from mentors and peers who are in the cohort with you + +## Development Areas & Activities +These topics will be covered during bi-weekly standups/workshops. The suggested activities will be covered in the mentee's normal day to day. Know something that should be added? Start a convo/add a PR - your comments are appreciated. + +### Current Member Cohort +* How to communicate in the Kubernetes ecosystem +* Governance +* Issue grooming / Identifying issues to work on +* [need three more] + +Suggested Activity +* Scrub XS and typo fixes into branches they maintain for occasional quick bulk review/approval with mentors +* Help to review docs + +### Current Reviewer Cohort +* Code Reviews the Kubernetes Way (i.e.- best practices) +* Analyzing if the solution is good for the project as a whole and not just that codebase +* How to groom issues for new members +* Communication effectively as a leader +* [need at least two more] + +Suggested Activity +* Be a tech reviewer for docs +SIG Docs to come in and do a quick ‘what it means to be a tech reviewer’ and ‘good qualities’ + +### Current Approver Cohort +* Communication +* [need 5 more] + + + +### Other Help Resources +- slack + - #kubernetes-dev + - your respective sig or other sigs that could help + - [SIG List](/sig-list.md) + - #sig-contribex + - #kubernetes-teachers +- mailing lists + - kubernetes-dev@googlegroups.com + - your respective sig (ie - kubernetes-sig-cli@googlegroups.com) +- contributor office hours + - Kubernetes Community Office Hours (need to create page; will update with link) + - #office-hours on slack +- the docs + - k/community is your friend for upstream workflows, processes, and information around contributing + - This repo includes the community/devel folder which will be extra helpful that includes docs such as: + - [Code Review Expectations](/contributors/devel/community-expectations.md) + - [Collaboration on k8s](/contributors/devel/collab.md) + + +### Test Cohort Special Circumstances & Notes +* This is a group of current members working towards reviewership. If you are a current member and would like to participate, reach out to Paris on slack. +* Since this is a predetermined group, the program may last two months instead of three. Expected completion date: end of February + + +Thank you to both mentees and mentors for taking on this new, unchartered territory! We will be building documentation around you and appreciate your guidance as a founding member of this program! + diff --git a/mentoring/group-mentoring.md b/mentoring/group-mentoring.md new file mode 100644 index 00000000..3e68c2b4 --- /dev/null +++ b/mentoring/group-mentoring.md @@ -0,0 +1,34 @@ +# Group Mentoring for Contributor Ladder Growth +*This is a work in progress* + +## Summary +This is an experimental group mentoring program where individuals will work self-paced in a semi-structured learning environment over the course of three months. Want to get `/lgtm` rights? Be a decision maker in an OWNERS file? This could be a great way to get the knowledge you need, some accountability to do it. Through this program you will get the opportunity to interact with active contributors who may be on the same path as you as well very experienced Kubernetes contributors (Pilots). + +## How This Works +This program is built around our [community membership guidelines](/community-membership.md). This doc shows how you can grow through the project in different roles and provides the requirements to get to each level on our ladder. + +A cohort will consist of 8-10 individuals aiming to achieve the same goal - the next stage on the contributor ladder - with the help of 3 Pilots representing 3 different projects. For example, our first test cohort is Member -> Reviewer in SIG Apps, SIG Cluster Lifecycle, and SIG AWS. The semi-structured learning element will happen in bi-weekly standups/workshops over zoom with various development areas targeted. + +Each cohort will last 2-3 months based on the time requirement in the community membership guidelines per level. + +## Benefits of a Cohort +* Peer mentoring +* Clear goals, objectives, and timelines +* Each mentee is working toward the same goal, cohort has an end date/timeframe +* Exposure to multiple areas of the project making well-rounded contributors +* Pilots share time commitments and responsibilities + +## Logistics +SIGs and other parts of the project will reach out to SIG Contributor Experience with requests for additional roles and provide a Pilot that is not the lead but at the same level they are targeting. + +Mentees will fill out the [Mentoring/Contributor Info Form](https://goo.gl/forms/SHWAiZ9Ih1qwuJbs1) which will eventually be posted to kubernetes.io and included in PR template. Based on responses, mentees will be matched with future cohorts. + +Feedback is continuous but will have a half way check in point. The goal is that mentees should not be surprised by the outcome of the program. + +## First Cohort +Our founding cohort is a predetermined set of mentees and Pilots. Our main goal with this cohort is to establish that group mentoring is viable for our workflow and culture. They will begin in mid-December and end around the end of February. + +## Important Links +[Mentor Guide] - to be updated +[Mentee Guide](/mentoring/group-mentee-guide.md) + diff --git a/project-managers/README.md b/project-managers/README.md index 180df247..bdc6a1b5 100644 --- a/project-managers/README.md +++ b/project-managers/README.md @@ -3,7 +3,7 @@ kubernetes-pm is a github team that will help to manage and maintain the project in ways other than just writing code. Specifically, members of the kubernetes-pm team are expected to: -* Add/change routing labels to issues ([sig/, area/](https://github.com/kubernetes/kubernetes/wiki)) +* Add/change routing labels to issues (eg: area/foo, kind/foo, priority/foo, sig/foo) * Apply release-note labels to PRs (until that is automated or eliminated) * Set the milestone on a PR or issue * Assign issues to the correct people for immediate triage/work diff --git a/sig-api-machinery/OWNERS b/sig-api-machinery/OWNERS index c43aa85f..0df76e64 100644 --- a/sig-api-machinery/OWNERS +++ b/sig-api-machinery/OWNERS @@ -1,10 +1,6 @@ reviewers: - - lavalamp - - deads2k - - mbohlool - - bgrant0607 + - sig-api-machinery-leads approvers: - - deads2k - - lavalamp - - mbohlool - - bgrant0607 + - sig-api-machinery-leads +labels: + - sig/api-machinery diff --git a/sig-api-machinery/README.md b/sig-api-machinery/README.md index 6ec6a110..f311bc89 100644 --- a/sig-api-machinery/README.md +++ b/sig-api-machinery/README.md @@ -4,25 +4,41 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # API Machinery SIG Covers all aspects of API server, API registration and discovery, generic API CRUD semantics, admission control, encoding/decoding, conversion, defaulting, persistence layer (etcd), OpenAPI, third-party resource, garbage collection, and client libraries. ## Meetings -* [Wednesdays at 18:00 UTC](https://staging.talkgadget.google.com/hangouts/_/google.com/kubernetes-sig) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=18:00&tz=UTC). +* [Wednesdays at 11:00 PT (Pacific Time)](https://staging.talkgadget.google.com/hangouts/_/google.com/kubernetes-sig) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=11:00&tz=PT%20%28Pacific%20Time%29). Meeting notes and Agenda can be found [here](https://goo.gl/x5nWrF). Meeting recordings can be found [here](https://www.youtube.com/watch?v=Lj1ScbXpnpY&list=PL69nYSiGNLP21oW3hbLyjjj4XhrwKxH2R). ## Leads -* [Daniel Smith](https://github.com/lavalamp), Google -* [David Eads](https://github.com/deads2k), Red Hat +* Daniel Smith (**[@lavalamp](https://github.com/lavalamp)**), Google +* David Eads (**[@deads2k](https://github.com/deads2k)**), Red Hat ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-api-machinery) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-api-machinery) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fapi-machinery) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-api-machinery-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-api-machinery-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-api-machinery-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-api-machinery-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-api-machinery-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-api-machinery-feature-requests) | Feature Requests | +| @kubernetes/sig-api-machinery-misc | [link](https://github.com/orgs/kubernetes/teams/sig-api-machinery-misc) | General Discussion | +| @kubernetes/sig-api-machinery-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-api-machinery-pr-reviews) | PR Reviews | +| @kubernetes/sig-api-machinery-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-api-machinery-proposals) | Design Proposals | +| @kubernetes/sig-api-machinery-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-api-machinery-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> ## Additional links diff --git a/sig-apps/OWNERS b/sig-apps/OWNERS index 7fb4a1f0..12723930 100644 --- a/sig-apps/OWNERS +++ b/sig-apps/OWNERS @@ -1,8 +1,6 @@ reviewers: - - michelleN - - mattfarina - - prydonius + - sig-apps-leads approvers: - - michelleN - - mattfarina - - prydonius + - sig-apps-leads +labels: + - sig/apps diff --git a/sig-apps/README.md b/sig-apps/README.md index 6ed28172..8fd2d48f 100644 --- a/sig-apps/README.md +++ b/sig-apps/README.md @@ -4,39 +4,68 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Apps SIG Covers deploying and operating applications in Kubernetes. We focus on the developer and devops experience of running applications in Kubernetes. We discuss how to define and run apps in Kubernetes, demo relevant tools and projects, and discuss areas of friction that can lead to suggesting improvements or feature requests. ## Meetings -* [Mondays at 16:00 UTC](https://zoom.us/j/4526666954) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=16:00&tz=UTC). +* [Mondays at 9:00 PT (Pacific Time)](https://zoom.us/my/sig.apps) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=9:00&tz=PT%20%28Pacific%20Time%29). Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1LZLBGW2wRDwAfdBNHJjFfk9CFoyZPcIYGWU7R1PQ3ng/edit#). Meeting recordings can be found [here](https://www.youtube.com/watch?v=hn23Z-vL_cM&list=PL69nYSiGNLP2LMq7vznITnpd2Fk1YIZF3). ## Leads -* [Michelle Noorali](https://github.com/michelleN), Microsoft -* [Matt Farina](https://github.com/mattfarina), Samsung SDS -* [Adnan Abdulhussein](https://github.com/prydonius), Bitnami +* Michelle Noorali (**[@michelleN](https://github.com/michelleN)**), Microsoft +* Matt Farina (**[@mattfarina](https://github.com/mattfarina)**), Samsung SDS +* Adnan Abdulhussein (**[@prydonius](https://github.com/prydonius)**), Bitnami +* Kenneth Owens (**[@kow3ns](https://github.com/kow3ns)**), Google ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-apps) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-apps) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fapps) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-apps-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-apps-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-apps-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-apps-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-apps-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-apps-feature-requests) | Feature Requests | +| @kubernetes/sig-apps-misc | [link](https://github.com/orgs/kubernetes/teams/sig-apps-misc) | General Discussion | +| @kubernetes/sig-apps-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-apps-pr-reviews) | PR Reviews | +| @kubernetes/sig-apps-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-apps-proposals) | Design Proposals | +| @kubernetes/sig-apps-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-apps-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> + ## Goals -* Discuss running applications in k8s -* Discuss how to define and run apps in k8s (APIs, CLIs, SDKs, package management tools, etc.) -* Suggest k8s features where we see friction -* Be the voice of the people running applications into the k8s development (developers and devops) -* Help people get involved in the kubernetes community + +* Discuss running and defining applications in Kubernetes (e.g., APIs, SDKs, Controllers, package management tools, etc.) +* Work on improvements to the Workload API +* Suggest Kubernetes features where we see friction +* Be the voice of the people running applications in Kubernetes (developers and devops) +* Help people get involved in the Kubernetes community * Show early features/demos of tools that make running apps easier ## Non-goals -* Our job is not to go implement stacks. We're helping people to help themselves. We will help connect people to the right folks * but we do not want to own a set of examples (as a group) -* Do not endorse one particular tool -* Do not pick which apps to run on top of the platform -* Do not recommend one way to do things + +* Do not endorse one particular ecosystem tool +* Do not pick which apps to run on top of Kubernetes +* Do not recommend one way to do things (e.g., picking a template language) + +## [Helm](https://helm.sh) and [Charts](https://github.com/kubernetes/charts) + +Helm and Charts each have their own regular meetings. + +* The Helm Developer call is [Thursday at 9:30 PT (Pacific Time)](https://zoom.us/j/4526666954). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=9:30&tz=PT%20%28Pacific%20Time%29). Meeting notes and agenda can be found [here](https://docs.google.com/document/d/1elWRfvH3AkHdr8pOaqyPbqSZ6ONR-l1Sb9_gapqh8ZA/edit). +* Charts Chat is every other [Tuesday at 09:00 PT](https://zoom.us/j/166909412). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=9:00&tz=PT%20(Pacific%20Time)). Meeting notes and agenda can be found [here](https://docs.google.com/document/d/1h6UTTuNRbFI81higrN3JUV2XxyzqqVjZET4Xz4WTR-8/edit#heading=h.57pbxthvt6k3). + +[Meetings can be found on the SIG Apps calendar.](https://calendar.google.com/calendar/embed?src=phfni1v25vnmi4q06m851230so%40group.calendar.google.com&ctz=America%2FNew_York) + <!-- END CUSTOM CONTENT --> diff --git a/sig-apps/agenda.md b/sig-apps/agenda.md index f132183c..aaca6480 100644 --- a/sig-apps/agenda.md +++ b/sig-apps/agenda.md @@ -73,11 +73,11 @@ _Note, the [minutes and agenda have moved to Google Docs](https://docs.google.co ## May 31, 2016 * Canceled in honor of a short week -## May 25, 2016 [[notes & video](https://github.com/kubernetes/community/blob/master/sig-apps/minutes/2016-05-25.md)] +## May 25, 2016 [[notes & video](/sig-apps/minutes/2016-05-25.md)] * Intro * Mike Metral of Rackspace will demo how to recursively process configuration files with the -R flag -## May 18, 2016 [[notes](https://github.com/kubernetes/community/blob/master/sig-apps/minutes/2016-05-18.md)] +## May 18, 2016 [[notes](/sig-apps/minutes/2016-05-18.md)] * Intro * Discussion on the future of SIG-Apps * Pranshanth B. of Google will demo PetSet diff --git a/sig-apps/minutes/2016-06-08.md b/sig-apps/minutes/2016-06-08.md index 3c6674cf..a597b9f4 100644 --- a/sig-apps/minutes/2016-06-08.md +++ b/sig-apps/minutes/2016-06-08.md @@ -11,7 +11,7 @@ - Michelle Noorali gave an update on where you can find information and examples on PetSets. - Here are some links provided by Pranshanth from Google. - [github issue](https://github.com/kubernetes/kubernetes/issues/260#issuecomment-220395798) - - [example pets](https://github.com/kubernetes/contrib/tree/master/pets) + - [example pets](https://git.k8s.io/contrib/pets) - Feel free to get your hands dirty. We will be discussing the provided examples in the upcoming weeks. Watch the [recording](https://youtu.be/wXZAXemhGb0). diff --git a/sig-apps/minutes/2016-08-03.md b/sig-apps/minutes/2016-08-03.md index 26379cfa..9e76cd3e 100644 --- a/sig-apps/minutes/2016-08-03.md +++ b/sig-apps/minutes/2016-08-03.md @@ -10,7 +10,7 @@ A: _(Clayton)_ Yes. Handling deployment failures at a high level, a generic idea for a trigger controller which watches another system for changes and makes updates to a Deployment, and hooks. * Ryan showed off OC which is a command line tool which is a wrapper for kubectl * Comment: One of the challenges Kubernetes faces today is that there is not a great way to extensibly pull in new chunks of APIs. - * This is something that is actively being worked on today. This work is being discussed and worked on in [SIG-API-Machinery](https://github.com/kubernetes/community/tree/master/sig-api-machinery) + * This is something that is actively being worked on today. This work is being discussed and worked on in [SIG-API-Machinery](/sig-api-machinery) * Free O'Reilly EBooks can be found [here](http://gist-reveal.it/4ca683dff6cdb9601c495e27d4bb5289#/oreilly-ebooks) courtesy of Red Hat. diff --git a/sig-architecture/OWNERS b/sig-architecture/OWNERS index 1e02f05b..2df39a59 100644 --- a/sig-architecture/OWNERS +++ b/sig-architecture/OWNERS @@ -1,6 +1,6 @@ reviewers: - - jdumars - - bgrant0607 + - sig-architecture-leads approvers: - - jdumars - - bgrant0607 + - sig-architecture-leads +labels: + - sig/architecture diff --git a/sig-architecture/README.md b/sig-architecture/README.md index 157be933..45c3abf5 100644 --- a/sig-architecture/README.md +++ b/sig-architecture/README.md @@ -4,25 +4,41 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Architecture SIG The Architecture SIG maintains and evolves the design principles of Kubernetes, and provides a consistent body of expertise necessary to ensure architectural consistency over time. ## Meetings -* [Mondays at 17:00 UTC](https://zoom.us/j/2018742972) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=17:00&tz=UTC). +* [Thursdays at 15:30 UTC](https://zoom.us/j/2018742972) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=15:30&tz=UTC). Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1BlmHq5uPyBUDlppYqAAzslVbAO8hilgjqZUTaNXUhKM/edit). Meeting recordings can be found [here](https://www.youtube.com/watch?v=d5ERqm3oHN0&list=PL69nYSiGNLP2m6198LaLN6YahX7EEac5g). ## Leads -* [Brian Grant](https://github.com/bgrant0607), Google -* [Jaice Singer DuMars](https://github.com/jdumars), Microsoft +* Brian Grant (**[@bgrant0607](https://github.com/bgrant0607)**), Google +* Jaice Singer DuMars (**[@jdumars](https://github.com/jdumars)**), Microsoft ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-architecture) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-architecture) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Farchitecture) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-architecture-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-architecture-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-architecture-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-architecture-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-architecture-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-architecture-feature-requests) | Feature Requests | +| @kubernetes/sig-architecture-misc-use-only-as-a-last-resort | [link](https://github.com/orgs/kubernetes/teams/sig-architecture-misc-use-only-as-a-last-resort) | General Discussion | +| @kubernetes/sig-architecture-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-architecture-pr-reviews) | PR Reviews | +| @kubernetes/sig-architecture-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-architecture-proposals) | Design Proposals | +| @kubernetes/sig-architecture-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-architecture-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> diff --git a/sig-architecture/backlog.md b/sig-architecture/backlog.md index 64ef8664..ecba1b63 100644 --- a/sig-architecture/backlog.md +++ b/sig-architecture/backlog.md @@ -1,39 +1,40 @@ # Backlog -## SIG-related +## Priority short list -* Change meeting day? -* Improve charter to meet expectations of Steering Committee -* Clarify criteria for areas out of scope for the SIG -* Formalize decision processes +TODO + +## Inbound requests + +* Component config field deprecation +* [Well known labels](https://groups.google.com/forum/#!topic/kubernetes-sig-architecture/PEp7NqWuFpw) ## Processes -* Proposal process * [API review process](https://github.com/kubernetes/community/pull/419/files) * Determine initial reviewers and approvers * [Branching policy](https://github.com/kubernetes/community/issues/566) -## Technical issues and conventions +## Conformance -* Events -* Component config field deprecation -* [Well known labels](https://groups.google.com/forum/#!topic/kubernetes-sig-architecture/PEp7NqWuFpw) -* [Conditions and phase](https://github.com/kubernetes/kubernetes/issues/7856) +* [Umbrella](https://github.com/kubernetes/community/issues/432) + +## Major technical issues and conventions + +* [Multi-repo plan and roadmap](https://github.com/kubernetes/kubernetes/issues/24343) * ["v2 API" plan](https://github.com/kubernetes/kubernetes/issues/8190) -* Multi-repo plan and roadmap -* Architectural layers +* [Architectural layers](https://github.com/kubernetes/community/issues/952) + +## SIG-related + +* Improve charter to meet expectations of Steering Committee +* Clarify criteria for areas out of scope for the SIG +* Formalize decision processes ## Documentation * [Architectural diagram](https://github.com/kubernetes/community/issues/767) * [Clean up design-proposals directory](https://github.com/kubernetes/community/issues/651) -* [Organize design-proposals by area](https://github.com/kubernetes/community/issues/918) * Document who owns client library, build, and release artifacts * Document who owns conformance definition, profiles, etc. * Extension strategy - -## Conformance - -* [Umbrella](https://github.com/kubernetes/community/issues/432) -* At some point, we need to approve of scope, profiles, required APIs/features/behaviors, etc. diff --git a/sig-architecture/charter.md b/sig-architecture/charter.md index 04fd2ecf..55214bab 100644 --- a/sig-architecture/charter.md +++ b/sig-architecture/charter.md @@ -16,12 +16,12 @@ Specific areas of focus include: * Maintaining, evolving, and enforcing the deprecation policy * [Deprecation policy](https://kubernetes.io/docs/reference/deprecation-policy/) * Documenting and evolving the system architecture - * [Kubernetes Design and Architecture](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture.md) + * [Kubernetes Design and Architecture](../contributors/design-proposals/architecture/architecture.md) * Defining and driving necessary extensibility points * Establishing and documenting design principles - * [Design principles](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/principles.md) + * [Design principles](../contributors/design-proposals/architecture/principles.md) * Establishing and documenting conventions for system and user-facing APIs - * [API conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md) + * [API conventions](/contributors/devel/api-conventions.md) * Developing necessary technical review processes, such as the proposal and API review processes * Driving improvement of overall code organization, including github orgs and repositories * Educating approvers/owners of other SIGs (e.g., by holding office hours) @@ -29,7 +29,7 @@ Specific areas of focus include: Out of scope: * Issues specific to a particular component or functional area, which would be the purview of some other SIG, except where they deviate from project-wide principles and conventions. -* [Release support policy](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/versioning.md) +* [Release support policy](/contributors/design-proposals/release/versioning.md) is owned by SIG Release TODO: diff --git a/sig-auth/OWNERS b/sig-auth/OWNERS index ef9c5847..3100c753 100644 --- a/sig-auth/OWNERS +++ b/sig-auth/OWNERS @@ -1,8 +1,6 @@ reviewers: - - ericchiang - - liggitt - - deads2k + - sig-auth-leads approvers: - - ericchiang - - liggitt - - deads2k + - sig-auth-leads +labels: + - sig/auth diff --git a/sig-auth/README.md b/sig-auth/README.md index 2ac2d5a3..af9a484c 100644 --- a/sig-auth/README.md +++ b/sig-auth/README.md @@ -4,7 +4,7 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Auth SIG @@ -17,13 +17,29 @@ Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/ Meeting recordings can be found [here](https://www.youtube.com/watch?v=DJDuDNALcMo&list=PL69nYSiGNLP0VMOZ-V7-5AchXTHAQFzJw). ## Leads -* [Eric Chiang](https://github.com/ericchiang), CoreOS -* [Jordan Liggitt](https://github.com/liggitt), Red Hat -* [David Eads](https://github.com/deads2k), Red Hat +* Eric Chiang (**[@ericchiang](https://github.com/ericchiang)**), CoreOS +* Jordan Liggitt (**[@liggitt](https://github.com/liggitt)**), Red Hat +* David Eads (**[@deads2k](https://github.com/deads2k)**), Red Hat ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-auth) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-auth) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fauth) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-auth-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-auth-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-auth-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-auth-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-auth-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-auth-feature-requests) | Feature Requests | +| @kubernetes/sig-auth-misc | [link](https://github.com/orgs/kubernetes/teams/sig-auth-misc) | General Discussion | +| @kubernetes/sig-auth-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-auth-pr-reviews) | PR Reviews | +| @kubernetes/sig-auth-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-auth-proposals) | Design Proposals | +| @kubernetes/sig-auth-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-auth-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> ## Goals diff --git a/sig-autoscaling/OWNERS b/sig-autoscaling/OWNERS index 480f8936..17089492 100644 --- a/sig-autoscaling/OWNERS +++ b/sig-autoscaling/OWNERS @@ -1,6 +1,6 @@ reviewers: - - mwielgus - - directxman12 + - sig-autoscaling-leads approvers: - - mwielgus - - directxman12 + - sig-autoscaling-leads +labels: + - sig/autoscaling diff --git a/sig-autoscaling/README.md b/sig-autoscaling/README.md index 05eb625e..f377c61e 100644 --- a/sig-autoscaling/README.md +++ b/sig-autoscaling/README.md @@ -4,25 +4,41 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Autoscaling SIG Covers autoscaling of clusters, horizontal and vertical autoscaling of pods, setting initial resources for pods, topics related to monitoring pods and gathering their metrics (e.g.: Heapster) ## Meetings -* [Thursdays at 15:30 UTC](https://zoom.us/j/176352399) (biweekly/triweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=15:30&tz=UTC). +* [Mondays at 14:00 UTC](https://zoom.us/my/k8s.sig.autoscaling) (biweekly/triweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=14:00&tz=UTC). Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1RvhQAEIrVLHbyNnuaT99-6u9ZUMp7BfkPupT2LAZK7w/edit). -Meeting recordings can be found [here](). + ## Leads -* [Marcin Wielgus](https://github.com/mwielgus), Google -* [Solly Ross](https://github.com/directxman12), Red Hat +* Marcin Wielgus (**[@mwielgus](https://github.com/mwielgus)**), Google +* Solly Ross (**[@directxman12](https://github.com/directxman12)**), Red Hat ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-autoscaling) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-autoscaling) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fautoscaling) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-autoscaling-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-autoscaling-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-autoscaling-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-autoscaling-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-autoscaling-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-autoscaling-feature-requests) | Feature Requests | +| @kubernetes/sig-autoscaling-misc | [link](https://github.com/orgs/kubernetes/teams/sig-autoscaling-misc) | General Discussion | +| @kubernetes/sig-autoscaling-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-autoscaling-pr-reviews) | PR Reviews | +| @kubernetes/sig-autoscaling-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-autoscaling-proposals) | Design Proposals | +| @kubernetes/sig-autoscaling-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-autoscaling-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> ## Concerns diff --git a/sig-aws/CONTRIBUTING.md b/sig-aws/CONTRIBUTING.md new file mode 100644 index 00000000..124c0c4e --- /dev/null +++ b/sig-aws/CONTRIBUTING.md @@ -0,0 +1,86 @@ +# Contributing + +The process for contributing code to Kubernetes via SIG-aws [community][community page]. + +**Note**: This page is focused on helping new contributors become active +members of the community through sustained contributions. + +## Introduction + +Welcome to the Kubernetes sig-aws contributing guide. We are excited +about the prospect of you joining our [community][community page]! +Mentoring and on-boarding new contributors is critical to the success +of the project. + +Please be aware that all contributions to Kubernetes require time +and commitment from project maintainers to direct and review work. +This is done in additional to many other maintainer responsibilities, and +direct engagement from maintainers is a finite resource. + +## Before You Begin + +**Note**: Complete the following steps before reaching out to aws community members. + +### Agree to contribution rules + +Follow the [CLA signup instructions](../CLA.md). + +### Understand the big picture + +This is important. + +- Play around with Kubernetes [Kubernetes Basics Tutorial]. +- Get to know possibilities to set up Kubernetes on AWS https://kubernetes.io/docs/getting-started-guides/aws/ +- Understand how Kubernetes on aws differs from other installations of Kubernetes [/contributors/design-proposals/aws/aws_under_the_hood.md] + + +## Adopt an issue + +New contributors can try the following to work on an existing bug or approved design: + +- In [slack][slack-messages] (signup [here][slack-signup]), + @mention a [lead][leads] and ask if there are any issues you could pick up. + We also maintain a list of [sig aws issues where help is wanted][aws_good_starter_issues]. + Most of them are not very complex, so that's probably a good starting point. + Leads can recommend issues that have enough priority to receive PR review bandwidth. +- Send an email to the _kubernetes-sig-aws@googlegroups.com_ [group] + + > Subject: New sig-aws contributor _${yourName}_ + > + > Body: Hello, my name is _${yourName}_. I would like to get involved in + > contributing to the Kubernetes project. I have read all of the + > user documentation listed on the community contributing page. + > What should I do next to get started? + +- Attend a sig-aws [meeting] and introduce yourself as looking to get started. + +## Meet the community + +Engage with the SIG aws community! Let us know who you are and how things are going! + +- In [slack][slack-messages] (signup [here][slack-signup]), + @mention a [lead][leads] and ask if there are any issues you could pick up, or + let them know what you are working on. + +- Attend a sig-aws [meeting] and introduce yourself and what you are working on. + +- The sig-aws [community page] lists sig-aws [leads], channels of [communication], +and group [meeting] times. + +[@mentions]: https://help.github.com/articles/basic-writing-and-formatting-syntax/#mentioning-users-and-teams +[Kubernetes Basics Tutorial]: https://kubernetes.io/docs/tutorials/kubernetes-basics +[PR]: https://help.github.com/articles/creating-a-pull-request +[agenda]: https://docs.google.com/document/d/1-i0xQidlXnFEP9fXHWkBxqySkXwJnrGJP9OGyP2_P14/edit +[communication]: /sig-aws#contact +[community page]: /sig-aws +[design repo]: /contributors/design-proposals/aws +[development guide]: /contributors/devel/development.md +[group]: https://groups.google.com/forum/#!forum/kubernetes-sig-aws +[kops]: https://git.k8s.io/kops/ +[leads]: /sig-aws#leads +[management overview]: https://kubernetes.io/docs/concepts/tools/kubectl/object-management-overview +[meeting]: /sig-aws#meetings +[slack-messages]: https://kubernetes.slack.com/messages/sig-aws +[slack-signup]: http://slack.k8s.io/ +[kube-aws-tools]: kubernetes-on-aws.md +[aws_good_starter_issues]: https://github.com/kubernetes/kops/issues?utf8=%E2%9C%93&q=is%3Aopen+is%3Aissue+label%3Agood-starter-issue diff --git a/sig-aws/OWNERS b/sig-aws/OWNERS index c189d1bc..83317bbe 100644 --- a/sig-aws/OWNERS +++ b/sig-aws/OWNERS @@ -1,8 +1,6 @@ reviewers: - - kris-nova - - justinsb - - chrislovecnm + - sig-aws-leads approvers: - - kris-nova - - justinsb - - chrislovecnm + - sig-aws-leads +labels: + - sig/aws diff --git a/sig-aws/README.md b/sig-aws/README.md index 89f25a28..bd9cdaa6 100644 --- a/sig-aws/README.md +++ b/sig-aws/README.md @@ -4,28 +4,39 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # AWS SIG Covers maintaining, supporting, and using Kubernetes hosted on AWS Cloud. ## Meetings -* [Fridays at 17:00 UTC](https://zoom.us/my/k8ssigaws) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=17:00&tz=UTC). +* [Fridays at 9:00 PT (Pacific Time)](https://zoom.us/my/k8ssigaws) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=9:00&tz=PT%20%28Pacific%20Time%29). Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1-i0xQidlXnFEP9fXHWkBxqySkXwJnrGJP9OGyP2_P14/edit). -Meeting recordings can be found [here](). + ## Leads -* [Justin Santa Barbara](https://github.com/justinsb) -* [Kris Nova](https://github.com/kris-nova), Microsoft -* [Chris Love](https://github.com/chrislovecnm) -* [Mackenzie Burnett](https://github.com/mfburnett), Redspread +* Justin Santa Barbara (**[@justinsb](https://github.com/justinsb)**) +* Kris Nova (**[@kris-nova](https://github.com/kris-nova)**), Microsoft +* Chris Love (**[@chrislovecnm](https://github.com/chrislovecnm)**) +* Mackenzie Burnett (**[@mfburnett](https://github.com/mfburnett)**), Redspread ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-aws) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-aws) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Faws) -<!-- BEGIN CUSTOM CONTENT --> +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-aws-misc | [link](https://github.com/orgs/kubernetes/teams/sig-aws-misc) | General Discussion | + +<!-- BEGIN CUSTOM CONTENT --> +## Participate +For information on how to participate in this community, read our [contribution guidelines](CONTRIBUTING.md). <!-- END CUSTOM CONTENT --> diff --git a/sig-aws/kubernetes-on-aws.md b/sig-aws/kubernetes-on-aws.md index 9f5376d9..2d5d72aa 100644 --- a/sig-aws/kubernetes-on-aws.md +++ b/sig-aws/kubernetes-on-aws.md @@ -6,15 +6,16 @@ This page lists different options, in alphabetic order, of starting a Kubernetes * Clocker: http://www.clocker.io/tutorials/kubernetes-cluster.html * Heptio: https://github.com/aws-quickstart/quickstart-heptio * Juju Charms: https://jujucharms.com/canonical-kubernetes/ +* KAT - Kubernetes cluster on AWS with Terraform: https://github.com/xuwang/kube-aws-terraform * Kargo: https://github.com/kubernetes-incubator/kargo * Kismatic Enterprise Toolkit: https://github.com/apprenda/kismatic * Kraken 2: https://github.com/samsung-cnct/k2 * kube-aws: https://github.com/kubernetes-incubator/kube-aws * Kubeadm Quickstart: https://github.com/upmc-enterprises/kubeadm-aws +* Kubernetes on AWS: https://github.com/zalando-incubator/kubernetes-on-aws/ * Kubernetes Operations (kops): https://github.com/kubernetes/kops * OpenShift: https://access.redhat.com/articles/2623521 * Stackpoint.io: https://stackpoint.io * Tack: https://github.com/kz8s/tack * Tectonic: http://github.com/coreos/tectonic-installer * Weaveworks AMI: https://github.com/weaveworks/kubernetes-ami - diff --git a/sig-azure/OWNERS b/sig-azure/OWNERS index 4bb1a8e8..bf52f70e 100644 --- a/sig-azure/OWNERS +++ b/sig-azure/OWNERS @@ -1,8 +1,6 @@ reviewers: - - colemickens - - slack - - jdumars + - sig-azure-leads approvers: - - colemickens - - slack - - jdumars + - sig-azure-leads +labels: + - sig/azure diff --git a/sig-azure/README.md b/sig-azure/README.md index d48dd5a7..27fdb76c 100644 --- a/sig-azure/README.md +++ b/sig-azure/README.md @@ -4,7 +4,7 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Azure SIG @@ -17,13 +17,23 @@ Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/ Meeting recordings can be found [here](https://www.youtube.com/watch?v=yQLeUKi_dwg&list=PL69nYSiGNLP2JNdHwB8GxRs2mikK7zyc4). ## Leads -* [Jason Hansen](https://github.com/slack), Microsoft -* [Cole Mickens](https://github.com/colemickens), Microsoft -* [Jaice Singer DuMars](https://github.com/jdumars), Microsoft +* Jason Hansen (**[@slack](https://github.com/slack)**), Microsoft +* Cole Mickens (**[@colemickens](https://github.com/colemickens)**), Microsoft +* Jaice Singer DuMars (**[@jdumars](https://github.com/jdumars)**), Microsoft ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-azure) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-azure) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fazure) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-azure-misc | [link](https://github.com/orgs/kubernetes/teams/sig-azure-misc) | General Discussion | <!-- BEGIN CUSTOM CONTENT --> diff --git a/sig-big-data/OWNERS b/sig-big-data/OWNERS index be30a58b..7e8ab706 100644 --- a/sig-big-data/OWNERS +++ b/sig-big-data/OWNERS @@ -1,4 +1,6 @@ reviewers: - - foxish + - sig-big-data-leads approvers: - - foxish + - sig-big-data-leads +labels: + - sig/big-data diff --git a/sig-big-data/README.md b/sig-big-data/README.md index 355a51fb..92e741bd 100644 --- a/sig-big-data/README.md +++ b/sig-big-data/README.md @@ -4,7 +4,7 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Big Data SIG @@ -14,14 +14,31 @@ Covers deploying and operating big data applications (Spark, Kafka, Hadoop, Flin * [Wednesdays at 17:00 UTC](https://zoom.us/my/sig.big.data) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=17:00&tz=UTC). Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit). -Meeting recordings can be found [here](). +Meeting recordings can be found [here](https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit). ## Leads -* [Anirudh Ramanathan](https://github.com/foxish), Google +* Anirudh Ramanathan (**[@foxish](https://github.com/foxish)**), Google +* Erik Erlandson (**[@erikerlandson](https://github.com/erikerlandson)**), Red Hat ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-big-data) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-big-data) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fbig-data) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-big-data-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-big-data-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-big-data-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-big-data-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-big-data-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-big-data-feature-requests) | Feature Requests | +| @kubernetes/sig-big-data-misc | [link](https://github.com/orgs/kubernetes/teams/sig-big-data-misc) | General Discussion | +| @kubernetes/sig-big-data-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-big-data-pr-reviews) | PR Reviews | +| @kubernetes/sig-big-data-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-big-data-proposals) | Design Proposals | +| @kubernetes/sig-big-data-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-big-data-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> ## Goals diff --git a/sig-cli/CONTRIBUTING.md b/sig-cli/CONTRIBUTING.md index 046e6cc7..d650665f 100644 --- a/sig-cli/CONTRIBUTING.md +++ b/sig-cli/CONTRIBUTING.md @@ -1,83 +1,213 @@ # Contributing -The process for contributing code to Kubernetes via the sig-cli [community][community page]. +The process for contributing code to Kubernetes via SIG-cli [community][community page]. -## TL;DR +**Note**: This page is focused on helping new contributors become active +members of the community through sustained contributions. -- The sig-cli [community page] lists sig-cli [leads], - channels of [communication], and group [meeting] times. -- New contributors: please start by adopting an [existing issue]. -- Request a feature by making an [issue] and mentioning - `@kubernetes/sig-cli-feature-requests`. -- Write a [design proposal] before starting work on a new feature. -- Write [tests]! - -## Before You Begin +## Introduction Welcome to the Kubernetes sig-cli contributing guide. We are excited about the prospect of you joining our [community][community page]! +Mentoring and on-boarding new contributors is critical to the success +of the project. + +Please be aware that all contributions to Kubernetes require time +and commitment from project maintainers to direct and review work. +This is done in additional to many other maintainer responsibilities, and +direct engagement from maintainers is a finite resource. + +## Pick your track + +Determine in what capacity you are looking to contribute: + +### Guided + +**Who is this for?** + +Contributors looking to engage with the SIG cli community for +a sustained period of time and looking to build working relationships +with existing members. Route to becoming a SIG cli member as +a reviewer or approver. + +**How does it work?** + +Work items come from a backlog of groomed items provided by SIG cli community members. +Each items has a stake holder willing to provide limited direction to contributors +working on it. Contributors typically need to put in 10x the time per-issue as the +maintainers providing direction. Contributors are expected to learn and do research +to complete the task independently with only periodic direction (~weekly). + +**What is expected of contributors?** + +Contributors are expected to make progress on items weekly and +provide periodic updates to any issue they are working on. +Contributors are expected exercise ownership of their code by fixing bugs +that are discovered. + +### Self service + +**Who is this for?** -Please understand that all contributions to Kubernetes require time -and commitment from the project maintainers to review the ux, software -design, and code. Mentoring and on-boarding new contributors is done -in addition to many other responsibilities. +Contributors that are looking to contribute only 1 or 2 items, or +have a specific issue they would like to like resolve and are willing +to contribute the solution. -### Understand the big picture +**How does it work?** -- Complete the [Kubernetes Basics Tutorial]. -- Be familiar with [kubectl user facing documentation ][kubectl docs]. -- Read the concept guides starting with the [management overview]. +Contributors are free to pick up any work items that they like. Maintainers +will be focused on directing contributors working on Guided items, so contributors +picking up non-Guided items will have almost no direction or support from maintainers. + +**What is expected of contributors?** + +Contributions must be relatively small, simple, well documented and well tested. +Since maintainers will need to own any code for these contributions, these should +be very limited in scope and contain minimal risk +(e.g. simple regression fixes, improved documentation, improved testing). + +## Before You Begin + +**Note**: Complete the following steps before reaching out to cli community members. + +### Agree to contribution rules + +Follow the [CLA signup instructions](../CLA.md). + +### Learn a bit about the kubectl cli + +This is important. + +- Learn about using kubectl with Kubernetes in the [Kubernetes Basics Tutorial]. +- Learn about managing configuration in the [kubectl docs]. ### Modify your own `kubectl` fork -Make sure you are ready to immediately get started once you have been -assigned a piece of work. Do this right away. +Make sure you are ready to immediately get started before you claim any piece of +work. - Setup your [development environment][development guide]. -- Look at code: + - This is hard. Sorry. We want to make this easier. +- Familiarize yourself with the code: - [kubernetes/cmd/kubectl] is the entry point - [kubernetes/pkg/kubectl] is the implementation - Look at how some of the other commands are implemented -- Add a new command to do something simple: +- Try adding a new command to do something simple: - Add `kubectl hello-world`: print "Hello World" - Add `kubectl hello-kubernetes -f file`: Print "Hello \<kind of resource\> \<name of resource\>" - Add `kubectl hello-kubernetes type/name`: Print "Hello \<kind of resource\> \<name of resource\> \<creation time\>" -### Agree to contribution rules +**Note:** Consider publishing your command to a fork so a maintainer can look at it. -Follow the [CLA signup instructions](../CLA.md). +## Your first contribution ### Adopt an issue -New contributors can try the following to work on an existing [bug] or [approved design][design repo]: +Pick up an [issue] from the backlog by commenting on the issue that you would like to work on it. +Be sure to mention the author of the issue as well as the SIG cli members `@seans3` and `@mengqiy`. + +Using the following comment will make it easier for us to search for issues folks want to have +assigned to them: + +`cc @seans3 @mengqiy I would like to take this` + +**Note:** Don't do this unless you will start work on the issue within a few days of being assigned. + +**Note:** GitHub only allows issues to be assigned to GitHub accounts that are part +of the organization or listed as outside collaborators. In order to become an outside +collaborator, contributors have to have shown commitment to the issue by performing some +work. Once you have begun, reach out to the issue author and show them your progress +(e.g. in a fork). + +**Picking your first issue** + +For your first issue, we recommend picking an issue labeled with "good first issue" from the [issue] backlog. + +**Picking the right size of issue** + +Be sure to pick up an issue that is appropriate to the time you are able to commit. +We recommend first time contributors start with small or medium issues. + +Following are very rough estimates, but are best effort only. They assume you have a +development environment already set up and are able to build a kubectl binary and +use it against a cluster. These estimates assume some knowledge of Go. + +- `size/S` + - 4-10 hours +- `size/M` + - 10-20 hours +- `size/L` + - 20+ hours +- `size/XL` + - 40-80 hours + +Meta/Umbrella issues may have multiple components. By signing up for a Meta/Umbrella issue, +you are only committing to one piece of it. Let the issue author know when you have completed +some piece of it, and if you would like to continue working on it, or have it unassigned. + +**Picking the right kind of issue** + +Guided issues have a *type* defining the type of work to be done. Pick up an +issue that fits your experience level and interest. Documentation and +test-coverage issues typically are smaller in scope and easier to complete than +features and cleanup issues. + +- `type/code-cleanup` + - Usually some refactoring or small rewrites of code. +- `type/code-documentation` + - Write `doc.go` with package overview and examples or add code comments to document + existing types and functions. +- `type/code-feature` + - Usually a new go package / library for some functionality that is requested. + Should be encapsulated in its own interfaces with thorough unit tests for the new library. +- `type/code-test-coverage` + - Audit tests for a package. Run coverage tools and also manually look at what functions + are missing unit or integration tests. Write tests for these functions. + +**Provide periodic status updates** + +Once you have requested an issue and it has been accepted, you will be expected +to provide periodic updates to it. Do update the issue with your status at least every +week, and publish your work to a fork so the community can see your progress and +provide early feedback. + +If you find the issue is too challenging, time consuming, or you are no longer able to work on it, +this is perfectly acceptable and please let the issue author know. +If you like, you may pick up a different issue immediately or sometime in the future. + +**Summary**: + +- Don't pick up an issue until you are ready to start working on it +- When you want to pick up an issue, be sure to comment `@seans3` and `@mengqiy`. + Expect a response within 2 days. +- Update the issue every week with your progress so we know it is being actively worked on. +- There is an expectation that some time will be committed to working on the issue each + week until it is completed, or you are blocked on a maintainer. + +### Meet the community + +Engage with the SIG cli community! Let us know who you are and how things are going! + +- Fill out the [about me form] so we know a bit about you and can direct your work accordingly. + - **Note:** After filling out the form, reach out via slack or the googlegroup and let us know. + +- Message the [cli mentors] googlegroup and let them know you filled out the form + and are looking to get started. - In [slack][slack-messages] (signup [here][slack-signup]), - @mention a [lead][leads] and ask if there are any issues you could pick up. - Leads can recommend issues that have enough priority to receive PR review bandwidth. - We also maintain a list of [CLI issues where help is wanted][cli_help_wanted_issues]. - Most of them are not very complex, so that's probably a good starting point. -- Send an email to the _kubernetes-sig-cli@googlegroups.com_ [group] - - > Subject: New sig-cli contributor _${yourName}_ - > - > Body: Hello, my name is _${yourName}_. I would like to get involved in - > contributing to the Kubernetes project. I have read all of the - > user documentation listed on the community contributing page. - > What should I do next to get started? - -- Attend a sig-cli [meeting] and introduce yourself as looking to get started. - -### Bug lifecycle - -1. An [issue] is filed that - - includes steps to reproduce the issue including client / server version, - - mentions `@kubernetes/sig-cli-bugs`. -2. A [PR] fixing the issue is implemented that - - __includes unit and test-cmd tests__, - - incorporates review feedback, - - description includes `Closes #<Issue Number>` or `Fixes #<Issue Number>`, - - description or comment @mentions `@kubernetes/sig-cli-pr-reviews`. -3. Fix appears in the next Kubernetes release! + @mention a [lead][leads] and ask if there are any issues you could pick up, or + let them know what you are working on. + +- Attend a sig-cli [meeting] and introduce yourself and what you are working on. + +- The sig-cli [community page] lists sig-cli [leads], channels of [communication], +and group [meeting] times. + +## Information about how Features are developed + +Once you have made several contributions, you may want to start developing +features that you come up with. This section is about how to propose new +features and get them accepted. ## Feature requests @@ -92,7 +222,7 @@ the problem that the feature addresses. Working on a feature without getting approval for the user experience and software design often results in wasted time and effort due to -decisions around flag-names, command names, and specific command +decisions around flag names, command names, and specific command behavior. To minimize wasted work and improve communication across efforts, @@ -112,7 +242,7 @@ any PRs are sent for code review. - docs are completed, - feature is designated _alpha_, _beta_ or _GA_. 6. Implement the code per discussion in [bug lifecycle][bug]. -7. Update [kubectl concept docs]. +7. Update [kubectl docs]. 8. Wait for your feature to appear in the next Kubernetes release! @@ -151,7 +281,7 @@ constrained. Community members are free to say - This is desirable but we need help on these other existing issues before tackling this. - No, this problem should be solved in another way. -The proposal can be merged into the [design repo] after [lead][leads] +The proposal can be merged into the [design repo] after [leads][leads] approval and discussion as a meeting [agenda] item. Then coding can begin. @@ -283,27 +413,27 @@ See the sig-cli [community page] for points of contact and meeting times: [`PTAL`]: https://en.wiktionary.org/wiki/PTAL [agenda]: https://docs.google.com/document/d/1r0YElcXt6G5mOWxwZiXgGu_X6he3F--wKwg-9UBc29I/edit [bug]: #bug-lifecycle -[communication]: https://github.com/kubernetes/community/tree/master/sig-cli#communication -[community page]: https://github.com/kubernetes/community/tree/master/sig-cli +[communication]: /sig-cli#contact +[community page]: /sig-cli [design proposal]: #design-proposals -[design repo]: https://github.com/kubernetes/community/tree/master/contributors/design-proposals/sig-cli -[design template]: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/sig-cli/template.md -[development guide]: https://github.com/kubernetes/community/blob/master/contributors/devel/development.md +[design repo]: /contributors/design-proposals/cli +[design template]: /contributors/design-proposals/Design_Proposal_TEMPLATE.md +[development guide]: /contributors/devel/development.md [existing issue]: #adopt-an-issue [feature repo]: https://github.com/kubernetes/features [feature request]: #feature-requests [feature]: https://github.com/kubernetes/features [group]: https://groups.google.com/forum/#!forum/kubernetes-sig-cli -[issue]: https://github.com/kubernetes/kubernetes/issues -[cli_help_wanted_issues]: https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Fcli+label%3Ahelp-wanted -[kubectl concept docs]: https://github.com/kubernetes/kubernetes.github.io/tree/master/docs/concepts/tools/kubectl -[kubectl docs]: https://kubernetes.io/docs/user-guide/kubectl-overview -[kubernetes/cmd/kubectl]: https://github.com/kubernetes/kubernetes/tree/master/cmd/kubectl -[kubernetes/pkg/kubectl]: https://github.com/kubernetes/kubernetes/tree/master/pkg/kubectl -[leads]: https://github.com/kubernetes/community/tree/master/sig-cli#leads +[issue]: https://github.com/kubernetes/kubectl/projects/3 +[kubectl docs]: https://kubernetes.io/docs/tutorials/object-management-kubectl/object-management/ +[kubernetes/cmd/kubectl]: https://git.k8s.io/kubernetes/cmd/kubectl +[kubernetes/pkg/kubectl]: https://git.k8s.io/kubernetes/pkg/kubectl +[leads]: /sig-cli#leads [management overview]: https://kubernetes.io/docs/concepts/tools/kubectl/object-management-overview -[meeting]: https://github.com/kubernetes/community/tree/master/sig-cli#meetings +[meeting]: /sig-cli#meetings [release]: #release [slack-messages]: https://kubernetes.slack.com/messages/sig-cli [slack-signup]: http://slack.k8s.io/ -[tests]: https://github.com/kubernetes/community/blob/master/contributors/devel/testing.md +[tests]: /contributors/devel/testing.md +[cli mentors]: https://groups.google.com/a/google.com/forum/#!forum/kubernetes-sig-cli-mentors +[about me form]: https://docs.google.com/forms/d/1ID6DX1abiDr9Z9_sXXC0DsMwuyHb_NeFdB3xeRa4Vf0 diff --git a/sig-cli/OWNERS b/sig-cli/OWNERS index 9e3894d0..248d3e7c 100644 --- a/sig-cli/OWNERS +++ b/sig-cli/OWNERS @@ -1,8 +1,6 @@ reviewers: - - pwittrock - - AdoHe - - fabianofranz + - sig-cli-leads approvers: - - pwittrock - - AdoHe - - fabianofranz + - sig-cli-leads +labels: + - sig/cli diff --git a/sig-cli/README.md b/sig-cli/README.md index c2196d41..62982c85 100644 --- a/sig-cli/README.md +++ b/sig-cli/README.md @@ -4,26 +4,43 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # CLI SIG Covers kubectl and related tools. We focus on the development and standardization of the CLI framework and its dependencies, the establishment of conventions for writing CLI commands, POSIX compliance, and improving the command line tools from a developer and devops user experience and usability perspective. ## Meetings -* [Wednesdays at 16:00 UTC](https://zoom.us/my/sigcli) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=16:00&tz=UTC). +* [Wednesdays at 09:00 PT (Pacific Time)](https://zoom.us/my/sigcli) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=09:00&tz=PT%20%28Pacific%20Time%29). Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1r0YElcXt6G5mOWxwZiXgGu_X6he3F--wKwg-9UBc29I/edit?usp=sharing). -Meeting recordings can be found [here](https://www.youtube.com/watch?v=X29sffrQJU4&list=PL69nYSiGNLP28HaTzSlFe6RJVxpFmbUvF). +Meeting recordings can be found [here](https://www.youtube.com/playlist?list=PL69nYSiGNLP28HaTzSlFe6RJVxpFmbUvF). ## Leads -* [Fabiano Franz](https://github.com/fabianofranz), Red Hat -* [Phillip Wittrock](https://github.com/pwittrock), Google -* [Tony Ado](https://github.com/AdoHe), Alibaba +* Fabiano Franz (**[@fabianofranz](https://github.com/fabianofranz)**), Red Hat +* Phillip Wittrock (**[@pwittrock](https://github.com/pwittrock)**), Google +* Tony Ado (**[@AdoHe](https://github.com/AdoHe)**), Alibaba ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-cli) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-cli) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fcli) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-cli-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-cli-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-cli-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-cli-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-cli-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-cli-feature-requests) | Feature Requests | +| @kubernetes/sig-cli-maintainers | [link](https://github.com/orgs/kubernetes/teams/sig-cli-maintainers) | CLI Maintainers | +| @kubernetes/sig-cli-misc | [link](https://github.com/orgs/kubernetes/teams/sig-cli-misc) | General Discussion | +| @kubernetes/sig-cli-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-cli-pr-reviews) | PR Reviews | +| @kubernetes/sig-cli-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-cli-proposals) | Design Proposals | +| @kubernetes/sig-cli-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-cli-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> diff --git a/sig-cli/migrated-from-wiki/README.md b/sig-cli/migrated-from-wiki/README.md new file mode 100644 index 00000000..c34a79fe --- /dev/null +++ b/sig-cli/migrated-from-wiki/README.md @@ -0,0 +1 @@ +The content in here has been migrated from https://github.com/kubernetes/community/wiki and is likely severely out of date. Please contact this SIG if you have questions or ideas about where this content should go. diff --git a/sig-cli/migrated-from-wiki/contributor-guide-building-your-kubernetes-tool.md b/sig-cli/migrated-from-wiki/contributor-guide-building-your-kubernetes-tool.md new file mode 100644 index 00000000..bfcd4233 --- /dev/null +++ b/sig-cli/migrated-from-wiki/contributor-guide-building-your-kubernetes-tool.md @@ -0,0 +1,64 @@ +# Client Tool Release Publishing Guidelines + +Projects should publish releases for client side tools. + +## Go Projects + +### Static Linking + +See [Go executables are statically linked, except when they are not](http://matthewkwilliams.com/index.php/2014/09/28/go-executables-are-statically-linked-except-when-they-are-not/). + + +- How to compile a statically linked binary: `go` file must be compiled without cgo support. + +```sh +# Disable cgo +export CGO_ENABLED=0 +``` + +- How to check if a binary is statically linked + +```sh +# List dynamic dependencies (shared libraries): +# 1. if it's dynamically linked, you'll see +$ ldd <your_tool> + linux-vdso.so.1 => (0x00007ffe937ea000) + libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0a7dae5000) + libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0a7d720000) + /lib64/ld-linux-x86-64.so.2 (0x00007f0a7dd03000) +# 2. if it's statically linked, you'll see +$ ldd <your_tool> + not a dynamic executable + +# Recognize the type of data in a file +# 1. if it's dynamically linked, you'll see +$ file <your_tool> +/usr/local/your_tool: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=86c6d2ff21297a06cc7319244f35e2671612beae, not stripped +# 2. if it's statically linked, you'll see +$ file <your_tool> +/usr/local/your_tool: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped +``` + +### Targets +Build your release binary for the following targets: + +- darwin-amd64 +- linux-386 +- linux-amd64 +- linux-armv6l +- linux-ppc64le +- windows-amd64 + +### Packaging + +Package binaries into a tar.gz file and make available on GitHub releases page. + +# Service Side Release Publishing Guidelines + +### Packaging + +Server side programs should be packaged into container images. Stateless services should be run as Deployments (as opposed to Replication Controllers). + +# Documentation Guidelines + +TODO: Write this diff --git a/sig-cli/migrated-from-wiki/roadmap-kubectl.md b/sig-cli/migrated-from-wiki/roadmap-kubectl.md new file mode 100644 index 00000000..4fbf7a84 --- /dev/null +++ b/sig-cli/migrated-from-wiki/roadmap-kubectl.md @@ -0,0 +1,112 @@ +# kubectl roadmap + +`kubectl` is the Kubernetes CLI. + +If you'd like to contribute, please read the [conventions](/contributors/devel/kubectl-conventions.md) and familiarize yourself with [existing commands](http://kubernetes.io/docs/user-guide/kubectl-overview/). + +**Owner:** @kubernetes/kubectl + +**Label:** [component/kubectl](https://github.com/kubernetes/kubernetes/labels/component%2Fkubectl) + +**Motivation:** [kubectl brainstorm](https://docs.google.com/document/d/1tPrTL5Fi8BjlIK_XwNz-W260ll2ZYflrdbtnwE1PQoE/edit?pli=1#) + +### Add new commands / subcommands / flags +* [Simplify support for multiple files](https://github.com/kubernetes/kubernetes/issues/24649) + * Manifest that can specify multiple files / http(s) URLs + * [Default manifest manifest](https://github.com/kubernetes/kubernetes/issues/3268) (ala Dockerfile or Makefile) + * Unpack archive (tgz, zip) and then invoke “-f” on that directory + * URL shortening via default URL prefix +* [Imperative `set` commands](https://github.com/kubernetes/kubernetes/issues/21648) +* [`view` commands](https://github.com/kubernetes/kubernetes/issues/29679) +* [Support `run --edit` and `create --edit`](https://github.com/kubernetes/kubernetes/issues/18064) +* [More `kubectl create <sub-command>`](https://github.com/kubernetes/kubernetes/issues/25382) +* [Support `--dry-run` for every mutation](https://github.com/kubernetes/kubernetes/issues/11488) +* kubectl commands aliases + * [Allow user defined aliases for resources and commands](https://github.com/kubernetes/kubernetes/issues/18023) + * [Suggest possibly matching kubectl commands](https://github.com/kubernetes/kubernetes/issues/25180) +* Improve `kubectl run` + * Make generated objects more discoverable: suggest the user to do `kubectl get all` to see what's generated ([extend `all` to more resources](https://github.com/kubernetes/kubernetes/issues/22337)) + * [Make it optional to specify name (auto generate name from image)](https://github.com/kubernetes/kubernetes/issues/2643) + * [Make `kubectl run --restart=Never` creates Pods (instead of Jobs)](https://github.com/kubernetes/kubernetes/issues/24533) +* Create commands/flags for common get + template patterns (e.g. getting service IP address) +* [Implement `kubectl cp`](https://github.com/kubernetes/kubernetes/issues/13776) to copy files between containers and local for debugging +* `kubectl rollout` + * [Add `kubectl rollout start` to show how to start a rollout](https://github.com/kubernetes/kubernetes/issues/25142) + * [Add `kubectl rollout status`](https://github.com/kubernetes/kubernetes/issues/25235) +* Scripting support + * [wait](https://github.com/kubernetes/kubernetes/issues/1899) + * [watch / IFTTT](https://github.com/kubernetes/kubernetes/issues/5164) +* [Add `kubectl top`](https://github.com/kubernetes/kubernetes/issues/11382) which lists resource metrics. + +### Alternative interfaces + +* Create a terminal based console, ref [docker console](https://github.com/dustinlacewell/console) ([video](https://www.youtube.com/watch?v=wSzZxbDYgtY)) +* [Add `kubectl sh`, an interactive shell](https://github.com/kubernetes/kubernetes/issues/25385), or make a kubectlshell in contrib and make bash completion part of it (ref [pythonshell](https://gist.github.com/bprashanth/9a3c8dfbba443698ddd960b8087107bf)) +* Think about how/whether to invoke generation commands such as `kubectl run` or `kubectl create configmap` in bulk, declaratively, such as part of the `apply` flow. +* [ChatOps](https://www.pagerduty.com/blog/what-is-chatops/) bot -- such as [kubebot](https://github.com/harbur/kubebot) (add to tools documentation) + +### Improve help / error messages / output +* Make kubectl functionality more discoverable + * [Overhaul kubectl help](https://github.com/kubernetes/kubernetes/issues/16089) + * ~~[Print "Usage" at the bottom](https://github.com/kubernetes/kubernetes/issues/7496)~~ + * Add keywords (critical words) to help + * List valid resources for each command + * Make short description of each command more concrete; use the same language for each command + * Link to docs ([kubernetes.io/docs](http://kubernetes.io/docs)) + * [Update `kubectl help` descriptions and examples from docs](https://github.com/kubernetes/kubernetes/issues/25290) + * Embed formatting and post-process for different media (terminal, man, github, etc.) + * [Suppress/hide global flags](https://github.com/kubernetes/kubernetes/issues/23402) + * ~~[Categorize kubectl commands or list them in alphabetical order]~~(https://github.com/kubernetes/kubernetes/issues/21585) + * [Implement search in `kubectl help`](https://github.com/kubernetes/kubernetes/issues/25234) + * [Suggest next/alternative commands](https://github.com/kubernetes/kubernetes/issues/19736) + * [Add a verbosity flag that explains all the things that it's doing](https://github.com/kubernetes/kubernetes/issues/25272) + * ~~[Fix incomplete kubectl bash completion](https://github.com/kubernetes/kubernetes/issues/25287)~~ +* Improve error messages (note that not all of these problems are in kubectl itself) + * [when kubectl doesn’t know what cluster to talk to](https://github.com/kubernetes/kubernetes/issues/24420) + * ~~[non-existent namespace produces obscure error](https://github.com/kubernetes/kubernetes/issues/15542)~~ + * [line numbers with validation errors](https://github.com/kubernetes/kubernetes/issues/12231) + * [invalid lines with validation errors](https://github.com/kubernetes/kubernetes/issues/6132) + * [malformed inputs produce misleading error messages](https://github.com/kubernetes/kubernetes/issues/9012) + * [non-yaml/json produces obscure error](https://github.com/kubernetes/kubernetes/issues/8838) + * [error messages for non-existent groups/types](https://github.com/kubernetes/kubernetes/issues/19530) + * Suggest resource type when not provided (e.g. `kubectl get my-pod-name` should suggest running `kubectl get pod/my-pod-name`) + * [errors for other non-existent resources](https://github.com/kubernetes/kubernetes/issues/6703) + * [lack of apiVersion/kind produces confusing error messages](https://github.com/kubernetes/kubernetes/issues/6439) + * [update validation errors could be more informative](https://github.com/kubernetes/kubernetes/issues/8668) + * [field validation errors could be more helpful](https://github.com/kubernetes/kubernetes/issues/10534) + * [field errors should use json field names](https://github.com/kubernetes/kubernetes/issues/3084) + * [clearer error for bad image/registry](https://github.com/kubernetes/kubernetes/issues/7960) + * [no error for illegal scale](https://github.com/kubernetes/kubernetes/issues/11148) + * [deletion timeout doesn't provide any details](https://github.com/kubernetes/kubernetes/issues/19427) + * [service creation timeout doesn't provide any details](https://github.com/kubernetes/kubernetes/issues/4860) + * [create secret with invalid data has obscure error message](https://github.com/kubernetes/kubernetes/issues/10309) + * [--all-namespaces error is unclear](https://github.com/kubernetes/kubernetes/issues/15834) + * [exec has unclear errors](https://github.com/kubernetes/kubernetes/issues/9944) + * [logs has misleading errors](https://github.com/kubernetes/kubernetes/issues/6376) + * [improve error reporting by adding URLs](https://github.com/kubernetes/kubernetes/issues/5551) + * Improve jsonpath / gotemplate error messages (it's tricky to get the path just right) + * [error message for user with no permissions is extremely cryptic](https://github.com/kubernetes/kubernetes/issues/26909) +* [Cleanup `kubectl get/describe` output](https://github.com/kubernetes/kubernetes/issues/20941) + * [Clarify kubectl get/describe service output](https://github.com/kubernetes/kubernetes/issues/22702) +* [Define and document command conventions for users](https://github.com/kubernetes/kubernetes/issues/25388) + +### Bug fix +* Fix [apply](https://github.com/kubernetes/kubernetes/issues?utf8=%E2%9C%93&q=is%3Aopen+label%3Acomponent%2Fkubectl+label%3Akind%2Fbug+apply), [edit](https://github.com/kubernetes/kubernetes/issues?utf8=%E2%9C%93&q=is%3Aopen+label%3Acomponent%2Fkubectl+label%3Akind%2Fbug+edit), and [validate](https://github.com/kubernetes/kubernetes/issues?utf8=%E2%9C%93&q=is%3Aopen+label%3Acomponent%2Fkubectl+label%3Akind%2Fbug+validate) bugs + +### Installation / Release +* `gcloud` should enable kubectl bash completion when installing `kubectl` +* [Pipe-to-sh to install kubectl](https://github.com/kubernetes/kubernetes/issues/25386) +* [Static build of kubectl for containers](https://github.com/kubernetes/kubernetes/issues/23708) ([we have it](https://git.k8s.io/kubernetes/examples/kubectl-container), but it's not part of the release) + +### Others +* [Move functionality to server](https://github.com/kubernetes/kubernetes/issues/12143) +* [Eliminate round-trip conversion of API objects in kubectl](https://github.com/kubernetes/kubernetes/issues/3955) +* [Move preferences out of kubeconfig](https://github.com/kubernetes/kubernetes/issues/10693) +* And then add more preferences + * Enable/disable explanatory mode (see [kploy output](http://kubernetes.sh/kploy/)) + * Permanently disable warnings once displayed + * Default labels as columns + * Default `--record`, `--save-config`, etc. +* [Overhaul cluster-related commands](https://github.com/kubernetes/kubernetes/issues/20605) + * [Delete cluster from `kubectl config`](https://github.com/kubernetes/kubernetes/issues/25601) +* ~~["kubectl-only Ubernetes": enabe kubectl to target any one of many clusters](https://github.com/kubernetes/kubernetes/issues/23492)~~ diff --git a/sig-cli/outreachy.md b/sig-cli/outreachy.md new file mode 100644 index 00000000..89f8cc40 --- /dev/null +++ b/sig-cli/outreachy.md @@ -0,0 +1,63 @@ +# Outreachy + +Kubernetes, specifically the SIG-CLI, is happy to announce our participation in the Outreachy program, running from December 2017 to March 2018. +Please see the [main program page](https://www.outreachy.org/) for the general information about the program, +such as its purpose, timeline, eligibility requirements, and how to apply. + +**Schedule** + +* October 23: application deadline for other Outreachy communities +* October 30: application deadline for Kubernetes Outreachy applications +* November 9: selection decisions are made +* December 5 - March 5: internship + +## What is Kubernetes? +Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. +It groups containers that make up an application into logical units for easy management and discovery. Kubernetes builds upon 15 years of experience of running production workloads at Google, combined with best-of-breed ideas and practices from the community. + +[kubernetes.io](https://kubernetes.io/) + +## What are SIGs / What is SIG-CLI? +Kubernetes is a set of projects, each shepherded by a special interest group (SIG). To get a grasp of the projects that we work on, check out the complete [list of SIGs](/sig-list.md). + +SIG-CLI Covers kubectl and related tools. We focus on the development and standardization of the CLI framework and its dependencies, the establishment of conventions for writing CLI commands, POSIX compliance, and improving the command line tools from a developer and devops user experience and usability perspective. + +**Communication:** + +SIG-CLI Mailing List - kubernetes-sig-cli@googlegroups.com +Slack - http://slack.k8s.io/ for invite; #sig-cli channel + +**Mentors** + +Philip Wittrock - pwittrock@google.com +Similar to contacting the coordinators, the mentors can be contacted at any time either by sending messages to the mailing lists or slack channels. + +## Contribute + +As part of the application process, the Outreachy program recommends that candidates make small contributions to the project they intend to apply for. +To start working on the project, make sure to fill out the CLA and check if you have the right environment with this guide. The README in the [community repo](https://github.com/kubernetes/community) details these things and more. + +Check out these specific resources for how to contribute to CLI: +* SIG-CLI - [How to Contribute](/sig-cli/CONTRIBUTING.md) +* Filter issue search for: `is:open is:issue label:sig/cli label:"help wanted"` +* Hand picked issues for outreachy applications: https://github.com/kubernetes/kubectl/projects/3 + +## Available tasks + +Develop `kubectl create` commands to make it easy to create Kubernetes resources +Develop `kubectl set` commands to modify Kubernetes resources +Required Skills: Go +Mentor: Phillip Wittrock + +**Coordination:** + +* Paris Pittman - parispittman@google.com +* Josh Berkus +* Elsie Phillips + +The coordinators can be contacted at any time. The easiest way is to send a slack message. + +Do you have an idea for a task that is suitable for this program? Contact the mentors or coordinators! Or even better, volunteer for mentoring an intern during the work on your idea! + +## Code of Conduct +Kubernetes abides by the CNCF [Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md). diff --git a/sig-cluster-lifecycle/OWNERS b/sig-cluster-lifecycle/OWNERS index 4fb4b999..d69f24ee 100644 --- a/sig-cluster-lifecycle/OWNERS +++ b/sig-cluster-lifecycle/OWNERS @@ -1,8 +1,6 @@ reviewers: - - roberthbailey - - lukemarsden - - jbeda + - sig-cluster-lifecycle-leads approvers: - - roberthbailey - - lukemarsden - - jbeda + - sig-cluster-lifecycle-leads +labels: + - sig/cluster-lifecycle diff --git a/sig-cluster-lifecycle/README.md b/sig-cluster-lifecycle/README.md index 459446ae..3bcbb9a4 100644 --- a/sig-cluster-lifecycle/README.md +++ b/sig-cluster-lifecycle/README.md @@ -4,36 +4,43 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Cluster Lifecycle SIG The Cluster Lifecycle SIG is responsible for building the user experience for deploying and upgrading Kubernetes clusters. ## Meetings -* [Tuesdays at 16:00 UTC](https://zoom.us/j/166836%E2%80%8B624) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=16:00&tz=UTC). +* [Tuesdays at 09:00 PT (Pacific Time)](https://zoom.us/j/166836%E2%80%8B624) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=09:00&tz=PT%20%28Pacific%20Time%29). Meeting notes and Agenda can be found [here](https://docs.google.com/a/weave.works/document/d/1deJYPIF4LmhGjDVaqrswErIrV7mtwJgovtLnPCDxP7U/edit). Meeting recordings can be found [here](https://www.youtube.com/watch?v=ljK5dgSA7vc&list=PL69nYSiGNLP29D0nYgAGWt1ZFqS9Z7lw4). ## Leads -* [Luke Marsden](https://github.com/lukemarsden), Weave -* [Joe Beda](https://github.com/jbeda), Heptio -* [Robert Bailey](https://github.com/roberthbailey), Google +* Luke Marsden (**[@lukemarsden](https://github.com/lukemarsden)**), Weave +* Joe Beda (**[@jbeda](https://github.com/jbeda)**), Heptio +* Robert Bailey (**[@roberthbailey](https://github.com/roberthbailey)**), Google +* Lucas Käldström (**[@luxas](https://github.com/luxas)**), Luxas Labs (occasionally contracting for Weaveworks) ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-cluster-lifecycle) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-lifecycle) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fcluster-lifecycle) ## GitHub Teams -* [@sig-cluster-lifecycle-misc](https://github.com/kubernetes/teams/sig-cluster-lifecycle-misc) -* [@sig-cluster-lifecycle-test-failures](https://github.com/kubernetes/teams/sig-cluster-lifecycle-test-failures) -* [@sig-cluster-lifecycle-bugs](https://github.com/kubernetes/teams/sig-cluster-lifecycle-bugs) -* [@sig-cluster-lifecycle-feature-requests](https://github.com/kubernetes/teams/sig-cluster-lifecycle-feature-requests) -* [@sig-cluster-lifecycle-proposals](https://github.com/kubernetes/teams/sig-cluster-lifecycle-proposals) -* [@sig-cluster-lifecycle-pr-reviews](https://github.com/kubernetes/teams/sig-cluster-lifecycle-pr-reviews) -* [@sig-cluster-lifecycle-api-reviews](https://github.com/kubernetes/teams/sig-cluster-lifecycle-api-reviews) +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-cluster-lifecycle-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-cluster-lifecycle-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-cluster-lifecycle-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-cluster-lifecycle-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-cluster-lifecycle-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-cluster-lifecycle-feature-requests) | Feature Requests | +| @kubernetes/sig-cluster-lifecycle-misc | [link](https://github.com/orgs/kubernetes/teams/sig-cluster-lifecycle-misc) | General Discussion | +| @kubernetes/sig-cluster-lifecycle-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-cluster-lifecycle-pr-reviews) | PR Reviews | +| @kubernetes/sig-cluster-lifecycle-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-cluster-lifecycle-proposals) | Design Proposals | +| @kubernetes/sig-cluster-lifecycle-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-cluster-lifecycle-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> diff --git a/sig-cluster-lifecycle/migrated-from-wiki/README.md b/sig-cluster-lifecycle/migrated-from-wiki/README.md new file mode 100644 index 00000000..c34a79fe --- /dev/null +++ b/sig-cluster-lifecycle/migrated-from-wiki/README.md @@ -0,0 +1 @@ +The content in here has been migrated from https://github.com/kubernetes/community/wiki and is likely severely out of date. Please contact this SIG if you have questions or ideas about where this content should go. diff --git a/sig-cluster-lifecycle/migrated-from-wiki/roadmap-cluster-deployment.md b/sig-cluster-lifecycle/migrated-from-wiki/roadmap-cluster-deployment.md new file mode 100644 index 00000000..d80875e3 --- /dev/null +++ b/sig-cluster-lifecycle/migrated-from-wiki/roadmap-cluster-deployment.md @@ -0,0 +1,166 @@ +OBSOLETE + +Cluster lifecycle includes deployment (infrastructure provisioning and bootstrapping Kubernetes), scaling, upgrades, and turndown. + +Owner: @kubernetes/sig-cluster-lifecycle (kubernetes-sig-cluster-lifecycle at googlegroups.com, sig-cluster-lifecycle on slack) + +There is no one-size-fits-all solution for cluster deployment and management (e.g., upgrades). There's a spectrum of possible solutions, each with different tradeoffs: +* opinionated solution (easier to use for a narrower solution space) vs. toolkit (easier to adapt and extend) +* understandability (easier to modify) vs. configurability (addresses a broader solution space without coding) + +Some useful points in the spectrum are described below. + +There are a number of tasks/features/changes that would be useful to multiple points in the spectrum. We should prioritize them, since they would enable multiple solutions. + +See also: +* [Feature-tracking issue](https://github.com/kubernetes/features/issues/11) +* [Cluster deployment v2 issue](https://github.com/kubernetes/kubernetes/issues/23174) +* [Shared infrastructure issue](https://github.com/kubernetes/kube-deploy/issues/123) +* [Kube-up replacement that works for everyone](https://github.com/kubernetes/kubernetes/issues/23478) +* https://github.com/kubernetes/kubernetes/labels/area%2Fcluster-lifecycle +* [Deployment DX Notes](https://docs.google.com/document/d/1jAQ5H222pJzpv-m0OsezZgdhdkf299E7H8Q9yzNrAjg/edit#) + +We need to improve support/behavior for cluster updates/upgrades, as well. TODO: make a list of open issues. Examples include [feature gating](https://github.com/kubernetes/kubernetes/issues/4855), [node upgrades](https://github.com/kubernetes/kubernetes/issues/6079), and [node draining](https://github.com/kubernetes/kubernetes/issues/6080). + +## Single-node laptop/development cluster + +Should be sufficient to kick the tires for most examples and for local development. Should be dead simple to use and highly opinionated rather than configurable. + +Owner: @dlorenc + +See also: +* https://github.com/kubernetes/minikube + +To do: +* Replace single-node getting-started guides + * [Docker single-node](http://kubernetes.io/docs/getting-started-guides/docker/) + +## Portable multi-node cluster understandable reference implementation + +For people who want to get Kubernetes running painlessly on an arbitrary set of machines -- any cloud provider (or bare metal), any OS distro, any networking infrastructure. Porting work should be minimized via separation of concerns (composition) and ease of modification rather than automated configuration transformation. Not intended to be highly optimized by default, but the cluster should be reliable. + +Also a reference implementation for people who want to understand how to build Kubernetes clusters from scratch. + +Ideally cluster scaling and upgrades would be supported by this implementation. + +Replace [Docker multi-node guide](http://kubernetes.io/docs/getting-started-guides/docker-multinode/). + +To facilitate this, we aim to provide an understandable, declarative, decoupled infrastructure provisioning implementation and a portable cluster bootstrapping implementation. Networking setup needs to be decoupled, so it can be swapped out with alternative implementations. + +For portability, all components need to be containerized (though Kubelet may use an alternative to Docker, so long as it is portable and meets other requirements) and we need a default network overlay solution. + +Eventually, we'd like to entirely eliminate the need for Chef/Puppet/Ansible/Salt. We shouldn't need to copy files around to host filesystems. + +For simplicity, users shouldn't need to install/launch more than one component or execute more than one command per node. This could be achieved a variety of ways: monolithic binaries, monolithic containers, a launcher/controller container that spawns other containers, etc. + +Once we have this, we should delete out-of-date, untested "getting-started guides" ([example broken cluster debugging thread](https://github.com/kubernetes/dashboard/issues/971)). + +See also: +* [Summary proposal](https://git.k8s.io/kubernetes-anywhere/PROPOSAL.md) +* [kubernetes-anywhere umbrella issue](https://github.com/kubernetes/kubernetes-anywhere/issues/127) +* https://git.k8s.io/kubernetes/docs/proposals/cluster-deployment.md +* [Bootstrap API](https://github.com/kubernetes/kubernetes/issues/5754) +* [jbeda's simple setup sketch](https://gist.github.com/jbeda/7e66965a23c40a91521cf6bbc3ebf007) + +To do: +* [Containerize](https://github.com/kubernetes/kubernetes/issues/246) all the components in order to achieve OS-distro independence +* [Self-host](https://github.com/kubernetes/kubernetes/issues/18735) as much as possible, such as using Deployment, DaemonSet, [ConfigMap](https://github.com/kubernetes/kubernetes/issues/1627) + * Eliminate the need to read configurations from local disk + * [Dynamic Kubelet configuration](https://github.com/kubernetes/kubernetes/issues/27980) + * [Kubelet checkpointing](https://github.com/kubernetes/kubernetes/issues/489) in order to reliably run control-plane components without static pods + * [Run Kubelet in a container](https://github.com/kubernetes/kubernetes/issues/4869) + * [DaemonSet updates](https://github.com/kubernetes/community/wiki/Roadmap:-DaemonSet) + * [DaemonSet for bootstrapping](https://github.com/kubernetes/kubernetes/issues/15324) + * [Bootkube](https://docs.google.com/document/d/1VNp4CMjPPHevh2_JQGMl-hpz9JSLq3s7HlI87CTjl-8/edit) +* Adopt a default network overlay, but enable others to be swapped in via composition + * Need to make it clear that this is for simplicity and portability, but isn't the only option +* [Link etcd and the master components](https://github.com/kubernetes/kubernetes/issues/5755) into a monolithic [monokube-like](https://github.com/polvi/monokube) binary and/or [finish hyperkube](https://github.com/kubernetes/kubernetes/issues/16508) +* [Replace multi-node Docker getting-started guide](https://github.com/kubernetes/kubernetes/issues/24114) +* Make it easy to get the right version of Docker on major Linux distros (e.g., apt-get install...) + * It's easy to get the wrong version: docker, docker.io, docker-engine, ... + +Starting points: +* https://github.com/kubernetes/kubernetes-anywhere +* https://github.com/Capgemini/kubeform + +## Building a cluster from scratch + +For people starting from scratch: +* http://kubernetes.io/docs/getting-started-guides/scratch/ +* https://github.com/kelseyhightower/kubernetes-the-hard-way +* https://news.ycombinator.com/item?id=12022215 + +We should simplify this as much as possible, and clearly document it. + +This is probably the only viable way to support people who want to do significant customization: +* cloud provider (including bare metal) +* OS distro +* cluster size +* master and worker node configurations +* networking solution and parameters (e.g., CIDR) +* container runtime (Docker or rkt) and its configuration +* monitoring solutions +* logging solutions +* ingress controller +* image registry +* IAM +* HA +* multi-zone +* K8s component configuration + +To do: +* Simplify release packaging and installation + * Finding and installing the right version of Docker itself can be hard (`apt-get install docker/docker.io/docker-engine` isn't the right thing) + * Build rpms, debs? +* Verify that system requirements have been satisfied (docker version, kernel configuration, etc.) + * And ideally degrade gracefully and warn if they are not +* Documentation + * What is the latest release, how can I find it, how do I install it, what version of Docker/rkt/etc. is required? + * An architectural diagram (like the one we use in our presentations) would help, too. + * Explain the architecture + * Link to instructions about how to manage etcd + * Link to [Chubby paper](http://static.googleusercontent.com/media/research.google.com/en//archive/chubby-osdi06.pdf) + * Document system requirements ("Node Spec") + * OS distro versions + * kernel configuration + * resources + * IP forwarding + * [Document how to set up a cluster](https://docs.google.com/document/d/1c4DMomZgS1i6AlKbb_8CiTcuimotkcJZAqJtQTN1iqc/edit#heading=h.58fpuhrw9g2o) + * Adequately document how to configure our components. + * Improve/simplify/organize command help + * Hide/remove/deemphasize test-only options + * Document how to integrate IAM + * [Create guides to help with key decisions for production clusters](https://github.com/kubernetes/kubernetes/issues/10100) + * Selecting a networking model + * Managing a CA + * Managing user authentication and authorization + * Initial deployment requirements (memory, cpu, networking, storage) + * Upgrading best practices +* Code changes + * Finish [converting components to use configuration files rather than command-line flags](https://github.com/kubernetes/kubernetes/issues/12245) + * Facilitate [managing that configuration using ConfigMap](https://github.com/kubernetes/kubernetes/issues/1627) + * [Cluster config](https://github.com/kubernetes/kubernetes/issues/19831) + * Reduce external dependencies + * APIs for reusable building blocks, such as [TLS bootstrap](https://github.com/kubernetes/kubernetes/issues/18112), [certificate signing for addons](https://github.com/kubernetes/kubernetes/issues/11725), [teardown](https://github.com/kubernetes/kubernetes/issues/4630) + * Need key/cert rotation ([master](https://github.com/kubernetes/kubernetes/issues/4672), [service accounts](https://github.com/kubernetes/kubernetes/issues/20165)) + * Finish generalization of [component registration](https://github.com/kubernetes/kubernetes/pull/13216) + * [Improve addon management](https://github.com/kubernetes/features/issues/18) + +## Production-grade, easy-to-use cluster management tools/services + +Easy to use and opinionated. Potentially highly optimized. Acceptable for production use. Not necessarily easily portable nor easy to extend/adapt/change. + +Examples: +* [Kube-AWS](https://github.com/coreos/coreos-kubernetes/tree/master/multi-node/aws) +* [kops](https://github.com/kubernetes/kops) +* [Kargo](https://github.com/kubespray/kargo) + * (is https://git.k8s.io/contrib/ansible still needed?) +* [kompose8](https://github.com/digitalrebar/kompos8) +* [Tectonic](https://tectonic.com/) +* [Kraken](https://github.com/samsung-cnct/kraken) +* [NavOps Launch](https://www.navops.io/launch.html) +* [Photon Cluster Manager](https://github.com/vmware/photon-controller/tree/master/java/cluster-manager) +* [Platform 9](https://platform9.com/blog/containers-as-a-service-kubernetes-docker/) +* [GKE](https://cloud.google.com/container-engine/) +* [Stackpoint.io](https://stackpoint.io/) +* [Juju](https://jujucharms.com/canonical-kubernetes) diff --git a/sig-cluster-ops/OWNERS b/sig-cluster-ops/OWNERS index 87ed6163..4e76d690 100644 --- a/sig-cluster-ops/OWNERS +++ b/sig-cluster-ops/OWNERS @@ -1,6 +1,6 @@ reviewers: - - jdumars - - zehicle + - sig-cluster-ops-leads approvers: - - jdumars - - zehicle + - sig-cluster-ops-leads +labels: + - sig/cluster-ops diff --git a/sig-cluster-ops/README.md b/sig-cluster-ops/README.md index 80d05c7a..2768e892 100644 --- a/sig-cluster-ops/README.md +++ b/sig-cluster-ops/README.md @@ -4,7 +4,7 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Cluster Ops SIG @@ -17,12 +17,13 @@ Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/ Meeting recordings can be found [here](https://www.youtube.com/watch?v=7uyy37pCk4U&list=PL69nYSiGNLP3b38liicqy6fm2-jWT4FQR). ## Leads -* [Rob Hirschfeld](https://github.com/zehicle), RackN -* [Jaice Singer DuMars](https://github.com/jdumars), Microsoft +* Rob Hirschfeld (**[@zehicle](https://github.com/zehicle)**), RackN +* Jaice Singer DuMars (**[@jdumars](https://github.com/jdumars)**), Microsoft ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-cluster-ops) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-ops) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fcluster-ops) <!-- BEGIN CUSTOM CONTENT --> diff --git a/sig-contributor-experience/OWNERS b/sig-contributor-experience/OWNERS index 23b252bb..e9c41ef7 100644 --- a/sig-contributor-experience/OWNERS +++ b/sig-contributor-experience/OWNERS @@ -1,6 +1,6 @@ reviewers: - - grodrigues3 - - Phillels + - sig-contributor-experience-leads approvers: - - grodrigues3 - - Phillels + - sig-contributor-experience-leads +labels: + - sig/contributor-experience diff --git a/sig-contributor-experience/README.md b/sig-contributor-experience/README.md index 66bad493..e7ff32a7 100644 --- a/sig-contributor-experience/README.md +++ b/sig-contributor-experience/README.md @@ -4,25 +4,40 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Contributor Experience SIG Developing and sustaining a healthy community of contributors is critical to scaling the project and growing the ecosystem. We need to ensure our contributors are happy and productive, we need to break the commit-rate ceiling we hit in July 2015, and we need to get the monotonically growing PR merge latency and numbers of open PRs and issues under control. ## Meetings -* [Wednesdays at 16:30 UTC](https://zoom.us/j/7658488911) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=16:30&tz=UTC). +* [Wednesdays at 9:30 PT (Pacific Time)](https://zoom.us/j/7658488911) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=9:30&tz=PT%20%28Pacific%20Time%29). Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1qf-02B7EOrItQgwXFxgqZ5qjW0mtfu5qkYIF1Hl4ZLI/). Meeting recordings can be found [here](https://www.youtube.com/watch?v=EMGUdOKwSns&list=PL69nYSiGNLP2x_48wbOPO0vXQgNTm_xxr). ## Leads -* [Garrett Rodrigues](https://github.com/grodrigues3), Google -* [Elsie Phillips](https://github.com/Phillels), CoreOS +* Garrett Rodrigues (**[@grodrigues3](https://github.com/grodrigues3)**), Google +* Elsie Phillips (**[@Phillels](https://github.com/Phillels)**), CoreOS ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-contribex) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-wg-contribex) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fcontributor-experience) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-contributor-experience-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-contributor-experience-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-contributor-experience-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-contributor-experience-feature-requests) | Feature Requests | +| @kubernetes/sig-contributor-experience-misc-use-only-as-a-last-resort | [link](https://github.com/orgs/kubernetes/teams/sig-contributor-experience-misc-use-only-as-a-last-resort) | General Discussion | +| @kubernetes/sig-contributor-experience-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-contributor-experience-pr-reviews) | PR Reviews | +| @kubernetes/sig-contributor-experience-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-contributor-experience-proposals) | Design Proposals | +| @kubernetes/sig-contributor-experience-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-contributor-experience-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> diff --git a/sig-contributor-experience/migrated-from-wiki/README.md b/sig-contributor-experience/migrated-from-wiki/README.md new file mode 100644 index 00000000..c34a79fe --- /dev/null +++ b/sig-contributor-experience/migrated-from-wiki/README.md @@ -0,0 +1 @@ +The content in here has been migrated from https://github.com/kubernetes/community/wiki and is likely severely out of date. Please contact this SIG if you have questions or ideas about where this content should go. diff --git a/sig-contributor-experience/migrated-from-wiki/effective-reviewable.md b/sig-contributor-experience/migrated-from-wiki/effective-reviewable.md new file mode 100644 index 00000000..1df0337c --- /dev/null +++ b/sig-contributor-experience/migrated-from-wiki/effective-reviewable.md @@ -0,0 +1,12 @@ +*Or, one weird trick to make Reviewable awesome* + +The Kubernetes team is still new to _Reviewable_. As you discover new cool features and workflows, add them here. Once we have built up a good number of tricks we can reorganize this list. + +- Hold off on publishing comments (using the "Publish" button) until you have completed your review. [(source)](@pwittrock) +- When leaving comments, select a "disposition" (the button with your profile picture) to indicate whether the comment requires resolution, or is just advisory and hence requires no response. [(source)](@pwittrock) +- Change a comment's "disposition" to "close" those to which the author didn't respond explicitly but did address with satisfactory changes to the code. Otherwise, the comment hangs out there awaiting a response; in contrast to GitHub's review system, _Reviewable_ doesn't consider a change to the target line to be a sufficient indicator of resolution or obsolescence, which is a safer design. Use the <kbd>y</kbd> to acknowledge the current comment, which indicates that no further response is necessary. +- To "collapse" a whole file in the multi-file view, click the rightmost value in the revision range control. This is effectively saying, "Show no diffs." +- Use the red/green "eye" icon to indicate completion of and to keep track of which files you have reviewed. The <kbd>x</kbd> keyboard shortcut toggles the completion status of the file currently focused in the status bar across the top. +- Use the <kbd>p</kbd> and <kbd>n</kbd> keys to navigate to the previous and next unreviewed file—that is, those whose status is a red circle with a crossed white eye icon, meaning incomplete, as opposed to those with a green circle with a white eye, meaning complete. +- Use the <kbd>j</kbd> and <kbd>k</kbd> keys to navigate to the previous and next comment. Use <kbd>S-j</kbd> and <kbd>S-k</kbd> to navigate between the previous and next _unaddressed_ comment. Usually as the reviewer, you use the latter to go back and check on whether your previous suggestions were addressed. +- Reply with `+lgtm` to apply the "LGTM" label directly from _Reviewable_.
\ No newline at end of file diff --git a/sig-creation-procedure.md b/sig-creation-procedure.md index 4e3540d7..ad2a9942 100644 --- a/sig-creation-procedure.md +++ b/sig-creation-procedure.md @@ -1,3 +1,3 @@ ## SIG creation procedure -Moved to [sig governance](/sig-governance.md#sig-creation-procecure)
\ No newline at end of file +Moved to [sig governance](/sig-governance.md#sig-creation-procedure) diff --git a/sig-docs/OWNERS b/sig-docs/OWNERS index 618a050c..5e9e6627 100644 --- a/sig-docs/OWNERS +++ b/sig-docs/OWNERS @@ -1,6 +1,6 @@ reviewers: - - jaredbhatti - - devin-donnelly + - sig-docs-leads approvers: - - jaredbhatti - - devin-donnelly + - sig-docs-leads +labels: + - sig/docs diff --git a/sig-docs/README.md b/sig-docs/README.md index 77b6f561..1de27516 100644 --- a/sig-docs/README.md +++ b/sig-docs/README.md @@ -4,25 +4,36 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Docs SIG Covers documentation, doc processes, and doc publishing for Kubernetes. ## Meetings -* [Tuesdays at 17:30 UTC](https://zoom.us/j/678394311) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=17:30&tz=UTC). +* [Tuesdays at 17:30 UTC](https://zoom.us/j/678394311) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=17:30&tz=UTC). Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1Ds87eRiNZeXwRBEbFr6Z7ukjbTow5RQcNZLaSvWWQsE/edit). -Meeting recordings can be found [here](). +Meeting recordings can be found [here](https://www.youtube.com/playlist?list=PL69nYSiGNLP3b5hlx0YV7Lo7DtckM84y8). ## Leads -* [Devin Donnelly](https://github.com/devin-donnelly), Google -* [Jared Bhatti](https://github.com/jaredbhatti), Google +* Devin Donnelly (**[@devin-donnelly](https://github.com/devin-donnelly)**), Google +* Jared Bhatti (**[@jaredbhatti](https://github.com/jaredbhatti)**), Google ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-docs) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-docs) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fdocs) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-docs-maintainers | [link](https://github.com/orgs/kubernetes/teams/sig-docs-maintainers) | Documentation Maintainers | +| @kubernetes/sig-docs-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-docs-pr-reviews) | Documentation PR Reviewers | <!-- BEGIN CUSTOM CONTENT --> ## Goals diff --git a/sig-docs/migrated-from-wiki/README.md b/sig-docs/migrated-from-wiki/README.md new file mode 100644 index 00000000..c34a79fe --- /dev/null +++ b/sig-docs/migrated-from-wiki/README.md @@ -0,0 +1 @@ +The content in here has been migrated from https://github.com/kubernetes/community/wiki and is likely severely out of date. Please contact this SIG if you have questions or ideas about where this content should go. diff --git a/sig-docs/migrated-from-wiki/roadmap-docs.md b/sig-docs/migrated-from-wiki/roadmap-docs.md new file mode 100644 index 00000000..63f453e4 --- /dev/null +++ b/sig-docs/migrated-from-wiki/roadmap-docs.md @@ -0,0 +1,15 @@ +# Docs and examples roadmap + +If you'd like to help with documentation, please read the [kubernetes.io site instructions](https://git.k8s.io/kubernetes.github.io/README.md). + +If you'd like to contribute an example, please read the [guidelines](https://git.k8s.io/kubernetes/examples/guidelines.md). + +Owners: @kubernetes/docs, @kubernetes/examples + +Labels: [kind/documentation](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Akind%2Fdocumentation), [kind/example](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Akind%2Fexample) + +We're currently planning a documentation site overhaul. Join kubernetes-dev@googlegroups.com to gain access to the proposals. + +* [Overhaul proposal](https://docs.google.com/document/d/12Nfj2E8tZL-CMnnpkSH9m5k_oqd4qJh4qDwYDxPGJJI/edit?ts=5705b1fc#) +* [Mocks](https://docs.google.com/presentation/d/14mJwJGlwBJHIpMRGR-nYMSyyNk-kRpabyz74-YYXeTo/edit#slide=id.g12d21f53ed_2_11) +* [API documentation improvements](https://github.com/kubernetes/kubernetes/issues/19680)
\ No newline at end of file diff --git a/sig-federation/OWNERS b/sig-federation/OWNERS deleted file mode 100644 index da6b0996..00000000 --- a/sig-federation/OWNERS +++ /dev/null @@ -1,6 +0,0 @@ -reviewers: - - csbell - - quinton-hoole -approvers: - - csbell - - quinton-hoole diff --git a/sig-federation/README.md b/sig-federation/README.md deleted file mode 100644 index bf3e6f00..00000000 --- a/sig-federation/README.md +++ /dev/null @@ -1,29 +0,0 @@ -<!--- -This is an autogenerated file! - -Please do not edit this file directly, but instead make changes to the -sigs.yaml file in the project root. - -To understand how this file is generated, see generator/README.md. ---> -# Federation SIG - -Covers the Federation of Kubernetes Clusters and related topics. This includes: application resiliency against availability zone outages; hybrid clouds; spanning of multiple could providers; application migration from private to public clouds (and vice versa); and other similar subjects. - -## Meetings -* [Tuesdays at 16:30 UTC](https://plus.google.com/hangouts/_/google.com/k8s-federation) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=16:30&tz=UTC). - -Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/18mk62nOXE_MCSSnb4yJD_8UadtzJrYyJxFwbrgabHe8/edit). -Meeting recordings can be found [here](https://www.youtube.com/watch?v=iWKC3FsNHWg&list=PL69nYSiGNLP0HqgyqTby6HlDEz7i1mb0-). - -## Leads -* [Christian Bell](https://github.com/csbell), Google -* [Quinton Hoole](https://github.com/quinton-hoole), Huawei - -## Contact -* [Slack](https://kubernetes.slack.com/messages/sig-federation) -* [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-federation) - -<!-- BEGIN CUSTOM CONTENT --> - -<!-- END CUSTOM CONTENT --> diff --git a/sig-gcp/OWNERS b/sig-gcp/OWNERS new file mode 100644 index 00000000..cd2232f4 --- /dev/null +++ b/sig-gcp/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-gcp-leads +approvers: + - sig-gcp-leads +labels: + - sig/gcp diff --git a/sig-gcp/README.md b/sig-gcp/README.md new file mode 100644 index 00000000..2ffac48b --- /dev/null +++ b/sig-gcp/README.md @@ -0,0 +1,43 @@ +<!--- +This is an autogenerated file! + +Please do not edit this file directly, but instead make changes to the +sigs.yaml file in the project root. + +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md +--> +# GCP SIG + +A Special Interest Group for building, deploying, maintaining, supporting, and using Kubernetes on the Google Cloud Platform. + +## Meetings +* [Thursdays at 16:00 UTC](https://zoom.us/j/761149873) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=16:00&tz=UTC). + +Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1mtmwZ4oVSSWhbEw8Lfzvc7ig84qxUpdK6uHyJp8rSGU/edit). + + +## Leads +* Adam Worrall (**[@abgworrall](https://github.com/abgworrall)**), Google + +## Contact +* [Slack](https://kubernetes.slack.com/messages/sig-gcp) +* [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-gcp) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-gcp-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-gcp-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-gcp-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-gcp-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-gcp-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-gcp-feature-requests) | Feature Requests | +| @kubernetes/sig-gcp-misc | [link](https://github.com/orgs/kubernetes/teams/sig-gcp-misc) | General Discussion | +| @kubernetes/sig-gcp-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-gcp-pr-reviews) | PR Reviews | +| @kubernetes/sig-gcp-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-gcp-proposals) | Design Proposals | +| @kubernetes/sig-gcp-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-gcp-test-failures) | Test Failures and Triage | + +<!-- BEGIN CUSTOM CONTENT --> + +<!-- END CUSTOM CONTENT --> diff --git a/sig-governance.md b/sig-governance.md index 5b4708b9..4d3660d4 100644 --- a/sig-governance.md +++ b/sig-governance.md @@ -11,24 +11,30 @@ In order to standardize Special Interest Group efforts, create maximum transpare * Participate in release planning meetings and retrospectives, and burndown meetings, as needed * Ensure related work happens in a project-owned github org and repository, with code and tests explicitly owned and supported by the SIG, including issue triage, PR reviews, test-failure response, bug fixes, etc. * Use the above forums as the primary means of working, communicating, and collaborating, as opposed to private emails and meetings -* Represent the SIG for the PM group - see [PM SIG representatives](https://github.com/kubernetes/community/blob/master/sig-product-management/SIG%20PM%20representatives.md). +* Represent the SIG for the PM group - see [PM SIG representatives](/sig-product-management/SIG%20PM%20representatives.md). ## SIG roles - **SIG Participant**: active in one or more areas of the project; wide variety of roles are represented - **SIG Lead**: SIG organizer -## SIG creation procedure +## SIG creation and maintenance procedure ### Prerequisites * Propose the new SIG publicly, including a brief mission statement, by emailing kubernetes-dev@googlegroups.com and kubernetes-users@googlegroups.com, then wait a couple of days for feedback * Ask a repo maintainer to create a github label, if one doesn't already exist: sig/foo -* Request a new [kubernetes.slack.com](http://kubernetes.slack.com) channel (#sig-foo) from **@sarahnovotny**. New users can join at [slack.kubernetes.io](http://slack.kubernetes.io). +* Request a new [kubernetes.slack.com](http://kubernetes.slack.com) channel (#sig-foo) from [@parispittman](https://github.com/parispittman) or [@castrojo](https://github.com/castrojo). New users can join at [slack.kubernetes.io](http://slack.kubernetes.io). * Slack activity is archived at [kubernetes.slackarchive.io](http://kubernetes.slackarchive.io). To start archiving a new channel invite the slackarchive bot to the channel via `/invite @slackarchive` * Organize video meetings as needed. No need to wait for the [Weekly Community Video Conference](community/README.md) to discuss. Please report summary of SIG activities there. - * Request a Zoom account from pyao@linuxfoundation.org cc'ing sarahnovotny@google.com. - * Add the meeting to the community meeting calendar by inviting cgnt364vd8s86hr2phapfjc6uk@group.calendar.google.com. + * Request a Zoom account by emailing Paris Pittman(`parispittman@google.com`) and Jorge Castro(`jorge@heptio.com`). You must set up a google group (see below) for the SIG leads so that all the SIG leads have the ability to reset the password if necessary. + * Read [how to use YouTube](/community/K8sYoutubeCollaboration.md) for publishing your videos to the Kubernetes channel. + * Calendars + 1. Create a calendar on your own account. Make it public. + 2. Share it with all SIG leads with full ownership of the calendar - they can edit, rename, or even delete it. + 3. Share it with `sc1@kubernetes.io`, `sc2@kubernetes.io`, `sc3@kubernetes.io`, with full ownership. This is just in case SIG leads ever disappear. + 4. Share it with the SIG mailing list, lowest priviledges. + 5. Share individual events with `cgnt364vd8s86hr2phapfjc6uk@group.calendar.google.com` to publish on the universal calendar. * Use existing proposal and PR process (to be documented) * Announce new SIG on kubernetes-dev@googlegroups.com * Submit a PR to add a row for the SIG to the table in the kubernetes/community README.md file, to create a kubernetes/community directory, and to add any SIG-related docs, schedules, roadmaps, etc. to your new kubernetes/community/SIG-foo directory. @@ -52,6 +58,7 @@ Create Google Groups at [https://groups.google.com/forum/#!creategroup](https:// Name convention: * kubernetes-sig-foo (the discussion group) +* kubernetes-sig-foo-leads (list for the leads, to be used with Zoom and Calendars) * kubernetes-sig-foo-misc * kubernetes-sig-foo-test-failures * kubernetes-sig-foo-bugs diff --git a/sig-instrumentation/OWNERS b/sig-instrumentation/OWNERS index d3eaa8b5..8e29eafa 100644 --- a/sig-instrumentation/OWNERS +++ b/sig-instrumentation/OWNERS @@ -1,6 +1,6 @@ reviewers: - - fabxc - - piosz + - sig-instrumentation-leads approvers: - - fabxc - - piosz + - sig-instrumentation-leads +labels: + - sig/instrumentation diff --git a/sig-instrumentation/README.md b/sig-instrumentation/README.md index d76a2c91..1f8cbc18 100644 --- a/sig-instrumentation/README.md +++ b/sig-instrumentation/README.md @@ -4,7 +4,7 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Instrumentation SIG @@ -14,15 +14,31 @@ Covers best practices for cluster observability through metrics, logging, and ev * [Thursdays at 16:30 UTC](https://zoom.us/j/5342565819) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=16:30&tz=UTC). Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1gWuAATtlmI7XJILXd31nA4kMq6U9u63L70382Y3xcbM/edit). -Meeting recordings can be found [here](). + ## Leads -* [Piotr Szczesniak](https://github.com/piosz), Google -* [Fabian Reinartz](https://github.com/fabxc), CoreOS +* Piotr Szczesniak (**[@piosz](https://github.com/piosz)**), Google +* Fabian Reinartz (**[@fabxc](https://github.com/fabxc)**), CoreOS ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-instrumentation) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-instrumentation) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Finstrumentation) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-instrumentation-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-instrumentation-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-instrumentation-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-instrumentation-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-instrumentation-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-instrumentation-feature-requests) | Feature Requests | +| @kubernetes/sig-instrumentation-misc | [link](https://github.com/orgs/kubernetes/teams/sig-instrumentation-misc) | General Discussion | +| @kubernetes/sig-instrumentation-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-instrumentation-pr-reviews) | PR Reviews | +| @kubernetes/sig-instrumentation-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-instrumentation-proposals) | Design Proposals | +| @kubernetes/sig-instrumentation-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-instrumentation-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> diff --git a/sig-list.md b/sig-list.md index 1c136841..a98efee7 100644 --- a/sig-list.md +++ b/sig-list.md @@ -4,12 +4,12 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # SIGs and Working Groups Most community activity is organized into Special Interest Groups (SIGs), -time bounded Working Groups, and the [community meeting](communication.md#Meeting). +time bounded Working Groups, and the [community meeting](communication.md#weekly-meeting). SIGs follow these [guidelines](governance.md) although each of these groups may operate a little differently depending on their needs and workflow. @@ -20,43 +20,49 @@ When the need arises, a [new SIG can be created](sig-creation-procedure.md) ### Master SIG List -| Name | Leads | Contact | Meetings | -|------|-------|---------|----------| -|[API Machinery](sig-api-machinery/README.md)|* [Daniel Smith](https://github.com/lavalamp), Google<br>* [David Eads](https://github.com/deads2k), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/sig-api-machinery)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-api-machinery)|* [Wednesdays at 18:00 UTC (biweekly)](https://staging.talkgadget.google.com/hangouts/_/google.com/kubernetes-sig)<br> -|[AWS](sig-aws/README.md)|* [Justin Santa Barbara](https://github.com/justinsb)<br>* [Kris Nova](https://github.com/kris-nova), Microsoft<br>* [Chris Love](https://github.com/chrislovecnm)<br>* [Mackenzie Burnett](https://github.com/mfburnett), Redspread<br>|* [Slack](https://kubernetes.slack.com/messages/sig-aws)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-aws)|* [Fridays at 17:00 UTC (biweekly)](https://zoom.us/my/k8ssigaws)<br> -|[Apps](sig-apps/README.md)|* [Michelle Noorali](https://github.com/michelleN), Microsoft<br>* [Matt Farina](https://github.com/mattfarina), Samsung SDS<br>* [Adnan Abdulhussein](https://github.com/prydonius), Bitnami<br>|* [Slack](https://kubernetes.slack.com/messages/sig-apps)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-apps)|* [Mondays at 16:00 UTC (weekly)](https://zoom.us/j/4526666954)<br> -|[Architecture](sig-architecture/README.md)|* [Brian Grant](https://github.com/bgrant0607), Google<br>* [Jaice Singer DuMars](https://github.com/jdumars), Microsoft<br>|* [Slack](https://kubernetes.slack.com/messages/sig-architecture)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-architecture)|* [Mondays at 17:00 UTC (biweekly)](https://zoom.us/j/2018742972)<br> -|[Auth](sig-auth/README.md)|* [Eric Chiang](https://github.com/ericchiang), CoreOS<br>* [Jordan Liggitt](https://github.com/liggitt), Red Hat<br>* [David Eads](https://github.com/deads2k), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/sig-auth)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-auth)|* [Wednesdays at 18:00 UTC (biweekly)](https://zoom.us/my/k8s.sig.auth)<br> -|[Autoscaling](sig-autoscaling/README.md)|* [Marcin Wielgus](https://github.com/mwielgus), Google<br>* [Solly Ross](https://github.com/directxman12), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/sig-autoscaling)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-autoscaling)|* [Thursdays at 15:30 UTC (biweekly/triweekly)](https://zoom.us/j/176352399)<br> -|[Azure](sig-azure/README.md)|* [Jason Hansen](https://github.com/slack), Microsoft<br>* [Cole Mickens](https://github.com/colemickens), Microsoft<br>* [Jaice Singer DuMars](https://github.com/jdumars), Microsoft<br>|* [Slack](https://kubernetes.slack.com/messages/sig-azure)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-azure)|* [Wednesdays at 16:00 UTC (biweekly)](https://deis.zoom.us/j/2018742972)<br> -|[Big Data](sig-big-data/README.md)|* [Anirudh Ramanathan](https://github.com/foxish), Google<br>|* [Slack](https://kubernetes.slack.com/messages/sig-big-data)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-big-data)|* [Wednesdays at 17:00 UTC (weekly)](https://zoom.us/my/sig.big.data)<br> -|[CLI](sig-cli/README.md)|* [Fabiano Franz](https://github.com/fabianofranz), Red Hat<br>* [Phillip Wittrock](https://github.com/pwittrock), Google<br>* [Tony Ado](https://github.com/AdoHe), Alibaba<br>|* [Slack](https://kubernetes.slack.com/messages/sig-cli)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-cli)|* [Wednesdays at 16:00 UTC (biweekly)](https://zoom.us/my/sigcli)<br> -|[Cluster Lifecycle](sig-cluster-lifecycle/README.md)|* [Luke Marsden](https://github.com/lukemarsden), Weave<br>* [Joe Beda](https://github.com/jbeda), Heptio<br>* [Robert Bailey](https://github.com/roberthbailey), Google<br>|* [Slack](https://kubernetes.slack.com/messages/sig-cluster-lifecycle)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-lifecycle)|* [Tuesdays at 16:00 UTC (weekly)](https://zoom.us/j/166836%E2%80%8B624)<br> -|[Cluster Ops](sig-cluster-ops/README.md)|* [Rob Hirschfeld](https://github.com/zehicle), RackN<br>* [Jaice Singer DuMars](https://github.com/jdumars), Microsoft<br>|* [Slack](https://kubernetes.slack.com/messages/sig-cluster-ops)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-ops)|* [Thursdays at 20:00 UTC (biweekly)](https://zoom.us/j/297937771)<br> -|[Contributor Experience](sig-contributor-experience/README.md)|* [Garrett Rodrigues](https://github.com/grodrigues3), Google<br>* [Elsie Phillips](https://github.com/Phillels), CoreOS<br>|* [Slack](https://kubernetes.slack.com/messages/sig-contribex)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-wg-contribex)|* [Wednesdays at 16:30 UTC (biweekly)](https://zoom.us/j/7658488911)<br> -|[Docs](sig-docs/README.md)|* [Devin Donnelly](https://github.com/devin-donnelly), Google<br>* [Jared Bhatti](https://github.com/jaredbhatti), Google<br>|* [Slack](https://kubernetes.slack.com/messages/sig-docs)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-docs)|* [Tuesdays at 17:30 UTC (biweekly)](https://zoom.us/j/678394311)<br> -|[Federation](sig-federation/README.md)|* [Christian Bell](https://github.com/csbell), Google<br>* [Quinton Hoole](https://github.com/quinton-hoole), Huawei<br>|* [Slack](https://kubernetes.slack.com/messages/sig-federation)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-federation)|* [Tuesdays at 16:30 UTC (biweekly)](https://plus.google.com/hangouts/_/google.com/k8s-federation)<br> -|[Instrumentation](sig-instrumentation/README.md)|* [Piotr Szczesniak](https://github.com/piosz), Google<br>* [Fabian Reinartz](https://github.com/fabxc), CoreOS<br>|* [Slack](https://kubernetes.slack.com/messages/sig-instrumentation)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-instrumentation)|* [Thursdays at 16:30 UTC (weekly)](https://zoom.us/j/5342565819)<br> -|[Network](sig-network/README.md)|* [Tim Hockin](https://github.com/thockin), Google<br>* [Dan Williams](https://github.com/dcbw), Red Hat<br>* [Casey Davenport](https://github.com/caseydavenport), Tigera<br>|* [Slack](https://kubernetes.slack.com/messages/sig-network)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-network)|* [Thursdays at 21:00 UTC (biweekly)](https://zoom.us/j/5806599998)<br> -|[Node](sig-node/README.md)|* [Dawn Chen](https://github.com/dchen1107), Google<br>* [Derek Carr](https://github.com/derekwaynecarr), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/sig-node)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-node)|* [Tuesdays at 17:00 UTC (weekly)](https://plus.google.com/hangouts/_/google.com/sig-node-meetup?authuser=0)<br> -|[On Premise](sig-on-premise/README.md)|* [Tomasz Napierala](https://github.com/zen), Mirantis<br>* [Marco Ceppi](https://github.com/marcoceppi), Canonical<br>* [Dalton Hubble](https://github.com/dghubble), CoreOS<br>|* [Slack](https://kubernetes.slack.com/messages/sig-onprem)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-on-prem)|* [Wednesdays at 16:00 UTC (weekly)](https://zoom.us/my/k8s.sig.onprem)<br> -|[OpenStack](sig-openstack/README.md)|* [Ihor Dvoretskyi](https://github.com/idvoretskyi), Mirantis<br>* [Steve Gordon](https://github.com/xsgordon), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/sig-openstack)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-openstack)|* [Wednesdays at 15:00 UTC (biweekly)](https://zoom.us/j/417251241)<br> -|[Product Management](sig-product-management/README.md)|* [Aparna Sinha](https://github.com/apsinha), Google<br>* [Ihor Dvoretskyi](https://github.com/idvoretskyi), Mirantis<br>* [Caleb Miles](https://github.com/calebamiles), CoreOS<br>|* [Slack](https://kubernetes.slack.com/messages/kubernetes-pm)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-pm)|* [Tuesdays at 15:00 UTC (biweekly)](https://zoom.us/j/845373595)<br> -|[Release](sig-release/README.md)|* [Phillip Wittrock](https://github.com/pwittrock), Google<br>* [Caleb Miles](https://github.com/calebamiles), CoreOS<br>|* [Slack](https://kubernetes.slack.com/messages/sig-release)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-release)|* [Tuesdays at 21:00 UTC (biweekly)](https://zoom.us/j/664772523)<br> -|[Scalability](sig-scalability/README.md)|* [Wojciech Tyczynski](https://github.com/wojtek-t), Google<br>* [Bob Wise](https://github.com/countspongebob), Samsung SDS<br>* [Joe Beda](https://github.com/jbeda), Heptio<br>|* [Slack](https://kubernetes.slack.com/messages/sig-scalability)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-scale)|* [Thursdays at 16:00 UTC (weekly)](https://zoom.us/j/989573207)<br> -|[Scheduling](sig-scheduling/README.md)|* [David Oppenheimer](https://github.com/davidopp), Google<br>* [Timothy St. Clair](https://github.com/timothysc), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/sig-scheduling)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-scheduling)|* [Mondays at 20:00 UTC (biweekly)](https://zoom.us/zoomconference?m=rN2RrBUYxXgXY4EMiWWgQP6Vslgcsn86)<br>* [Wednesdays at 07:30 UTC (biweekly)](https://zoom.us/zoomconference?m=rN2RrBUYxXgXY4EMiWWgQP6Vslgcsn86)<br> -|[Service Catalog](sig-service-catalog/README.md)|* [Paul Morie](https://github.com/pmorie), Red Hat<br>* [Aaron Schlesinger](https://github.com/arschles), Microsoft<br>* [Ville Aikas](https://github.com/vaikas-google), Google<br>* [Doug Davis](https://github.com/duglin), IBM<br>|* [Slack](https://kubernetes.slack.com/messages/sig-service-catalog)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-service-catalog)|* [Mondays at 20:00 UTC (weekly)](https://zoom.us/j/7201225346)<br> -|[Storage](sig-storage/README.md)|* [Saad Ali](https://github.com/saad-ali), Google<br>* [Bradley Childs](https://github.com/childsb), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/sig-storage)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-storage)|* [Thursdays at 16:00 UTC (biweekly)](https://zoom.us/j/614261834)<br> -|[Testing](sig-testing/README.md)|* [Aaron Crickenberger](https://github.com/spiffxp), Samsung SDS<br>* [Erick Feja](https://github.com/fejta), Google<br>* [Timothy St. Clair](https://github.com/timothysc), Heptio<br>|* [Slack](https://kubernetes.slack.com/messages/sig-testing)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-testing)|* [Tuesdays at 20:00 UTC (weekly)](https://zoom.us/j/2419653117)<br> -|[UI](sig-ui/README.md)|* [Dan Romlein](https://github.com/danielromlein), Apprenda<br>* [Sebastian Florek](https://github.com/floreks), Fujitsu<br>|* [Slack](https://kubernetes.slack.com/messages/sig-ui)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-ui)|* [Thursdays at 16:00 UTC (weekly)](https://groups.google.com/forum/#!forum/kubernetes-sig-ui)<br> -|[Windows](sig-windows/README.md)|* [Michael Michael](https://github.com/michmike), Apprenda<br>|* [Slack](https://kubernetes.slack.com/messages/sig-windows)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-windows)|* [Tuesdays at 16:30 UTC (weekly)](https://zoom.us/my/sigwindows)<br> +| Name | Label | Leads | Contact | Meetings | +|------|--------|-------|---------|----------| +|[API Machinery](sig-api-machinery/README.md)|api-machinery|* [Daniel Smith](https://github.com/lavalamp), Google<br>* [David Eads](https://github.com/deads2k), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/sig-api-machinery)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-api-machinery)|* [Wednesdays at 11:00 PT (Pacific Time) (biweekly)](https://staging.talkgadget.google.com/hangouts/_/google.com/kubernetes-sig)<br> +|[AWS](sig-aws/README.md)|aws|* [Justin Santa Barbara](https://github.com/justinsb)<br>* [Kris Nova](https://github.com/kris-nova), Microsoft<br>* [Chris Love](https://github.com/chrislovecnm)<br>* [Mackenzie Burnett](https://github.com/mfburnett), Redspread<br>|* [Slack](https://kubernetes.slack.com/messages/sig-aws)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-aws)|* [Fridays at 9:00 PT (Pacific Time) (biweekly)](https://zoom.us/my/k8ssigaws)<br> +|[Apps](sig-apps/README.md)|apps|* [Michelle Noorali](https://github.com/michelleN), Microsoft<br>* [Matt Farina](https://github.com/mattfarina), Samsung SDS<br>* [Adnan Abdulhussein](https://github.com/prydonius), Bitnami<br>* [Kenneth Owens](https://github.com/kow3ns), Google<br>|* [Slack](https://kubernetes.slack.com/messages/sig-apps)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-apps)|* [Mondays at 9:00 PT (Pacific Time) (weekly)](https://zoom.us/my/sig.apps)<br> +|[Architecture](sig-architecture/README.md)|architecture|* [Brian Grant](https://github.com/bgrant0607), Google<br>* [Jaice Singer DuMars](https://github.com/jdumars), Microsoft<br>|* [Slack](https://kubernetes.slack.com/messages/sig-architecture)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-architecture)|* [Thursdays at 15:30 UTC (weekly)](https://zoom.us/j/2018742972)<br> +|[Auth](sig-auth/README.md)|auth|* [Eric Chiang](https://github.com/ericchiang), CoreOS<br>* [Jordan Liggitt](https://github.com/liggitt), Red Hat<br>* [David Eads](https://github.com/deads2k), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/sig-auth)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-auth)|* [Wednesdays at 18:00 UTC (biweekly)](https://zoom.us/my/k8s.sig.auth)<br> +|[Autoscaling](sig-autoscaling/README.md)|autoscaling|* [Marcin Wielgus](https://github.com/mwielgus), Google<br>* [Solly Ross](https://github.com/directxman12), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/sig-autoscaling)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-autoscaling)|* [Mondays at 14:00 UTC (biweekly/triweekly)](https://zoom.us/my/k8s.sig.autoscaling)<br> +|[Azure](sig-azure/README.md)|azure|* [Jason Hansen](https://github.com/slack), Microsoft<br>* [Cole Mickens](https://github.com/colemickens), Microsoft<br>* [Jaice Singer DuMars](https://github.com/jdumars), Microsoft<br>|* [Slack](https://kubernetes.slack.com/messages/sig-azure)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-azure)|* [Wednesdays at 16:00 UTC (biweekly)](https://deis.zoom.us/j/2018742972)<br> +|[Big Data](sig-big-data/README.md)|big-data|* [Anirudh Ramanathan](https://github.com/foxish), Google<br>* [Erik Erlandson](https://github.com/erikerlandson), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/sig-big-data)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-big-data)|* [Wednesdays at 17:00 UTC (weekly)](https://zoom.us/my/sig.big.data)<br> +|[CLI](sig-cli/README.md)|cli|* [Fabiano Franz](https://github.com/fabianofranz), Red Hat<br>* [Phillip Wittrock](https://github.com/pwittrock), Google<br>* [Tony Ado](https://github.com/AdoHe), Alibaba<br>|* [Slack](https://kubernetes.slack.com/messages/sig-cli)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-cli)|* [Wednesdays at 09:00 PT (Pacific Time) (biweekly)](https://zoom.us/my/sigcli)<br> +|[Cluster Lifecycle](sig-cluster-lifecycle/README.md)|cluster-lifecycle|* [Luke Marsden](https://github.com/lukemarsden), Weave<br>* [Joe Beda](https://github.com/jbeda), Heptio<br>* [Robert Bailey](https://github.com/roberthbailey), Google<br>* [Lucas Käldström](https://github.com/luxas), Luxas Labs (occasionally contracting for Weaveworks)<br>|* [Slack](https://kubernetes.slack.com/messages/sig-cluster-lifecycle)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-lifecycle)|* [Tuesdays at 09:00 PT (Pacific Time) (weekly)](https://zoom.us/j/166836%E2%80%8B624)<br> +|[Cluster Ops](sig-cluster-ops/README.md)|cluster-ops|* [Rob Hirschfeld](https://github.com/zehicle), RackN<br>* [Jaice Singer DuMars](https://github.com/jdumars), Microsoft<br>|* [Slack](https://kubernetes.slack.com/messages/sig-cluster-ops)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-ops)|* [Thursdays at 20:00 UTC (biweekly)](https://zoom.us/j/297937771)<br> +|[Contributor Experience](sig-contributor-experience/README.md)|contributor-experience|* [Garrett Rodrigues](https://github.com/grodrigues3), Google<br>* [Elsie Phillips](https://github.com/Phillels), CoreOS<br>|* [Slack](https://kubernetes.slack.com/messages/sig-contribex)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-wg-contribex)|* [Wednesdays at 9:30 PT (Pacific Time) (biweekly)](https://zoom.us/j/7658488911)<br> +|[Docs](sig-docs/README.md)|docs|* [Devin Donnelly](https://github.com/devin-donnelly), Google<br>* [Jared Bhatti](https://github.com/jaredbhatti), Google<br>|* [Slack](https://kubernetes.slack.com/messages/sig-docs)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-docs)|* [Tuesdays at 17:30 UTC (weekly)](https://zoom.us/j/678394311)<br> +|[GCP](sig-gcp/README.md)||* [Adam Worrall](https://github.com/abgworrall), Google<br>|* [Slack](https://kubernetes.slack.com/messages/sig-gcp)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-gcp)|* [Thursdays at 16:00 UTC (biweekly)](https://zoom.us/j/761149873)<br> +|[Instrumentation](sig-instrumentation/README.md)|instrumentation|* [Piotr Szczesniak](https://github.com/piosz), Google<br>* [Fabian Reinartz](https://github.com/fabxc), CoreOS<br>|* [Slack](https://kubernetes.slack.com/messages/sig-instrumentation)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-instrumentation)|* [Thursdays at 16:30 UTC (weekly)](https://zoom.us/j/5342565819)<br> +|[Multicluster](sig-multicluster/README.md)|multicluster|* [Christian Bell](https://github.com/csbell), Google<br>* [Quinton Hoole](https://github.com/quinton-hoole), Huawei<br>|* [Slack](https://kubernetes.slack.com/messages/sig-multicluster)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-multicluster)|* [Tuesdays at 9:30 PT (Pacific Time) (biweekly)](https://zoom.us/my/k8s.mc)<br> +|[Network](sig-network/README.md)|network|* [Tim Hockin](https://github.com/thockin), Google<br>* [Dan Williams](https://github.com/dcbw), Red Hat<br>* [Casey Davenport](https://github.com/caseydavenport), Tigera<br>|* [Slack](https://kubernetes.slack.com/messages/sig-network)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-network)|* [Thursdays at 14:00 PT (Pacific Time) (biweekly)](https://zoom.us/j/5806599998)<br> +|[Node](sig-node/README.md)|node|* [Dawn Chen](https://github.com/dchen1107), Google<br>* [Derek Carr](https://github.com/derekwaynecarr), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/sig-node)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-node)|* [Tuesdays at 10:00 PT (Pacific Time) (weekly)](https://plus.google.com/hangouts/_/google.com/sig-node-meetup?authuser=0)<br> +|[On Premise](sig-on-premise/README.md)|onprem|* [Marco Ceppi](https://github.com/marcoceppi), Canonical<br>* [Dalton Hubble](https://github.com/dghubble), CoreOS<br>|* [Slack](https://kubernetes.slack.com/messages/sig-onprem)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-on-prem)|* [Wednesdays at 16:00 UTC (weekly)](https://zoom.us/my/k8s.sig.onprem)<br> +|[OpenStack](sig-openstack/README.md)|openstack|* [Ihor Dvoretskyi](https://github.com/idvoretskyi), CNCF<br>* [Steve Gordon](https://github.com/xsgordon), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/sig-openstack)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-openstack)|* [Thursdays at 00:00 UTC (biweekly)](https://zoom.us/j/417251241)<br> +|[Product Management](sig-product-management/README.md)|none|* [Aparna Sinha](https://github.com/apsinha), Google<br>* [Ihor Dvoretskyi](https://github.com/idvoretskyi), CNCF<br>* [Caleb Miles](https://github.com/calebamiles), CoreOS<br>|* [Slack](https://kubernetes.slack.com/messages/kubernetes-pm)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-pm)|* [Tuesdays at 15:00 UTC (biweekly)](https://zoom.us/j/845373595)<br> +|[Release](sig-release/README.md)|release|* [Phillip Wittrock](https://github.com/pwittrock), Google<br>* [Caleb Miles](https://github.com/calebamiles), CoreOS<br>|* [Slack](https://kubernetes.slack.com/messages/sig-release)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-release)|* [Tuesdays at 21:00 UTC (biweekly)](https://zoom.us/j/664772523)<br> +|[Scalability](sig-scalability/README.md)|scalability|* [Wojciech Tyczynski](https://github.com/wojtek-t), Google<br>* [Bob Wise](https://github.com/countspongebob), Samsung SDS<br>|* [Slack](https://kubernetes.slack.com/messages/sig-scalability)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-scale)|* [Thursdays at 16:00 UTC (bi-weekly)](https://zoom.us/j/989573207)<br> +|[Scheduling](sig-scheduling/README.md)|scheduling|* [David Oppenheimer](https://github.com/davidopp), Google<br>* [Timothy St. Clair](https://github.com/timothysc), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/sig-scheduling)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-scheduling)|* [Mondays at 20:00 UTC (biweekly)](https://zoom.us/zoomconference?m=rN2RrBUYxXgXY4EMiWWgQP6Vslgcsn86)<br>* [Wednesdays at 07:30 UTC (biweekly)](https://zoom.us/zoomconference?m=rN2RrBUYxXgXY4EMiWWgQP6Vslgcsn86)<br> +|[Service Catalog](sig-service-catalog/README.md)|service-catalog|* [Paul Morie](https://github.com/pmorie), Red Hat<br>* [Aaron Schlesinger](https://github.com/arschles), Microsoft<br>* [Ville Aikas](https://github.com/vaikas-google), Google<br>* [Doug Davis](https://github.com/duglin), IBM<br>|* [Slack](https://kubernetes.slack.com/messages/sig-service-catalog)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-service-catalog)|* [Mondays at 20:00 UTC (weekly)](https://zoom.us/j/7201225346)<br> +|[Storage](sig-storage/README.md)|storage|* [Saad Ali](https://github.com/saad-ali), Google<br>* [Bradley Childs](https://github.com/childsb), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/sig-storage)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-storage)|* [Thursdays at 9:00 PT (Pacific Time) (biweekly)](https://zoom.us/j/614261834)<br> +|[Testing](sig-testing/README.md)|testing|* [Aaron Crickenberger](https://github.com/spiffxp), Samsung SDS<br>* [Erick Feja](https://github.com/fejta), Google<br>* [Steve Kuznetsov](https://github.com/stevekuznetsov), Red Hat<br>* [Timothy St. Clair](https://github.com/timothysc), Heptio<br>|* [Slack](https://kubernetes.slack.com/messages/sig-testing)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-testing)|* [Tuesdays at 13:00 PT (Pacific Time) (weekly)](https://zoom.us/my/k8s.sig.testing)<br> +|[UI](sig-ui/README.md)|ui|* [Dan Romlein](https://github.com/danielromlein), Google<br>* [Sebastian Florek](https://github.com/floreks), Fujitsu<br>|* [Slack](https://kubernetes.slack.com/messages/sig-ui)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-ui)|* [Thursdays at 18:00 CET (Central European Time) (weekly)](https://groups.google.com/forum/#!forum/kubernetes-sig-ui)<br> +|[Windows](sig-windows/README.md)|windows|* [Michael Michael](https://github.com/michmike), Apprenda<br>|* [Slack](https://kubernetes.slack.com/messages/sig-windows)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-windows)|* [Tuesdays at 12:30 Eastern Standard Time (EST) (bi-weekly)](https://zoom.us/my/sigwindows)<br> ### Master Working Group List | Name | Organizers | Contact | Meetings | |------|------------|---------|----------| +|[App Def](wg-app-def/README.md)|* [Antoine Legrand](https://github.com/ant31), CoreOS<br>* [Sebastien Goasguen](https://github.com/sebgoa), Bitnami<br>|* [Slack](https://kubernetes.slack.com/messages/wg-app-def)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-wg-app-def)|* [Wednesdays at 16:00 UTC (bi-weekly)](https://zoom.us/j/748123863)<br> +|[Cloud Provider](wg-cloud-provider/README.md)|* [Sidhartha Mani](https://github.com/wlan0), Caascade Labs<br>* [Jago Macleod](https://github.com/jagosan), Google<br>|* [Slack](https://kubernetes.slack.com/messages/wg-cloud-provider)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-wg-cloud-provider)|* [Wednesdays at 10:00 PT (Pacific Time) (weekly)](https://zoom.us/my/cloudprovider)<br> +|[Cluster API](wg-cluster-api/README.md)|* [Kris Nova](https://github.com/kris-nova), Microsoft<br>* [Jacob Beacham](https://github.com/pipejakob), Google<br>* [Robert Bailey](https://github.com/roberthbailey), Google<br>|* [Slack](https://kubernetes.slack.com/messages/sig-cluster-lifecycle)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-lifecycle)| |[Container Identity](wg-container-identity/README.md)|* [Clayton Coleman](https://github.com/smarterclayton), Red Hat<br>* [Greg Gastle](https://github.com/destijl), Google<br>|* [Slack](https://kubernetes.slack.com/messages/wg-container-identity)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-wg-container-identity)|* [Tuesdays at 15:00 UTC (bi-weekly (On demand))](TBD)<br> -|[Resource Management](wg-resource-management/README.md)|* [Vishnu Kannan](https://github.com/vishh), Google<br>* [Derek Carr](https://github.com/derekwaynecarr), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/wg-resource-mgmt)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-wg-resource-management)|* [Tuesdays at 18:00 UTC (weekly (On demand))](https://zoom.us/j/4799874685)<br> +|[Kubeadm Adoption](wg-kubeadm-adoption/README.md)|* [Lucas Käldström](https://github.com/luxas), Luxas Labs (occasionally contracting for Weaveworks)<br>* [Justin Santa Barbara](https://github.com/justinsb)<br>|* [Slack](https://kubernetes.slack.com/messages/sig-cluster-lifecycle)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-lifecycle)|* [Tuesdays at 18:00 UTC (bi-weekly)](https://zoom.us/j/166836%E2%80%8B624)<br> +|[Multitenancy](wg-multitenancy/README.md)|* [David Oppenheimer](https://github.com/davidopp), Google<br>* [Jessie Frazelle](https://github.com/jessfraz), Microsoft<br>|* [Slack](https://kubernetes.slack.com/messages/wg-multitenancy)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-wg-multitenancy)| +|[Resource Management](wg-resource-management/README.md)|* [Vishnu Kannan](https://github.com/vishh), Google<br>* [Derek Carr](https://github.com/derekwaynecarr), Red Hat<br>|* [Slack](https://kubernetes.slack.com/messages/wg-resource-mgmt)<br>* [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-wg-resource-management)|* [Wednesdays at 11:00 PT (Pacific Time) (weekly (On demand))](https://zoom.us/j/4799874685)<br> <!-- BEGIN CUSTOM CONTENT --> <!-- END CUSTOM CONTENT --> diff --git a/sig-multicluster/CONTRIBUTING.md b/sig-multicluster/CONTRIBUTING.md new file mode 100644 index 00000000..d0600a77 --- /dev/null +++ b/sig-multicluster/CONTRIBUTING.md @@ -0,0 +1,294 @@ +# Contributing + +The process for contributing code to Kubernetes via the sig-multicluster [community][community page]. + +## TL;DR + +- The sig-multicluster [community page] lists sig-multicluster [leads] + and group [meeting] times. +- Request a feature by making an [issue] and mentioning + `@kubernetes/sig-multicluster-feature-requests`. +- Write a design proposal before starting work on a new feature. +- Write [tests]! + +## Before You Begin + +Welcome to the Kubernetes sig-multicluster contributing guide. We are excited +about the prospect of you joining our [community][community page]! + +Please understand that all contributions to Kubernetes require time +and commitment from the project maintainers to review the ux, software +design, and code. Mentoring and on-boarding new contributors is done +in addition to many other responsibilities. + +### Understand the big picture + +- Complete the [Kubernetes Basics Tutorial]. + +### Agree to contribution rules + +Follow the [CLA signup instructions](../CLA.md). + +### Adopt an issue + +New contributors can try the following to work on an existing bug or approved design: + +- In [slack][slack-messages] (signup [here][slack-signup]), + @mention a [lead][leads] and ask if there are any issues you could pick up. + We also maintain a list of [multi cluster issues where help is wanted][multicluster_help_wanted_issues]. + Most of them are not very complex, so that's probably a good starting point. + Leads can recommend issues that have enough priority to receive PR review bandwidth. +- Send an email to the _kubernetes-sig-multicluster@googlegroups.com_ [group] + + > Subject: New sig-multicluster contributor _${yourName}_ + > + > Body: Hello, my name is _${yourName}_. I would like to get involved in + > contributing to the Kubernetes project. I have read all of the + > user documentation listed on the community contributing page. + > What should I do next to get started? + +- Attend a sig-multicluster [meeting] and introduce yourself as looking to get started. + +### Bug lifecycle + +#### Filing a bug + +1. An [issue] is filed that + - includes steps to reproduce the issue including client / server version, + - mentions `@kubernetes/sig-multicluster-bugs`. + +#### Sending a fix + +2. A [PR] fixing the issue is implemented that + - __includes unit and test-cmd tests__, + - incorporates review feedback, + - description includes `Closes #<Issue Number>` or `Fixes #<Issue Number>`, + - description or comment @mentions `@kubernetes/sig-multicluster-pr-reviews`. +3. Fix appears in the next Kubernetes release! + +## Feature requests + +__New contributors:__ Please start by adopting an [existing issue]. + +A feature request is an [issue] mentioning `@kubernetes/sig-multicluster-feature-requests`. + +To encourage readership, the issue description should _concisely_ (2-4 sentence) describe +the problem that the feature addresses. + +### Feature lifecycle + +Working on a feature without getting approval for the user experience +and software design often results in wasted time and effort due to +decisions around names and user experience. + +To minimize wasted work and improve communication across efforts, +the user experience and software design must be agreed upon before +any PRs are sent for code review. + +1. Identify a problem by filing an [issue] (mention `@kubernetes/sig-multicluster-feature-requests`). +2. Submit a [design proposal] and get it approved by a lead. +3. Announce the proposal as an [agenda] item for the sig-multicluster [meeting]. + - Ensures awareness and feedback. + - Should be included in meeting notes sent to the sig-multicluster [group]. +4. _Merge_ the proposal PR after approval and announcement. +5. A [lead][leads] adds the associated feature to the [feature repo], ensuring that + - release-related decisions are properly made and communicated, + - API changes are vetted, + - testing is completed, + - docs are completed, + - feature is designated _alpha_, _beta_ or _GA_. +6. Implement the code per discussion in [bug lifecycle][bug]. +7. Update docs. +8. Wait for your feature to appear in the next Kubernetes release! + + +## Design Proposals + +__New contributors:__ Please start by adopting an [existing issue]. + +A design proposal is a single markdown document in the [design repo] +that follows the [design template]. + +To make one, +- Prepare the markdown document as a PR to that repo. + - Avoid _Work In Progress_ (WIP) PRs (send it only after + you consider it complete). + - For early feedback, use the email discussion [group]. +- Mention `@kubernetes/sig-multicluster-proposals` in the description. +- Mention the related [feature request]. + +Expect feedback from 2-3 different sig-multicluster community members. + +Incorporate feedback and comment [`PTAL`]. + +Once a [lead][leads] has agreed (via review commentary) that design +and code review resources can be allocated to tackle the proposal, the +details of the user experience and design should be discussed in the +community. + +This step is _important_; it prevents code churn and thrashing around +issues like flag names, command names, etc. + +It is normal for sig-multicluster community members to push back on feature +proposals. sig-multicluster development and review resources are extremely +constrained. Community members are free to say + +- No, not this release (or year). +- This is desirable but we need help on these other existing issues before tackling this. +- No, this problem should be solved in another way. + +The proposal can be merged into the [design repo] after [lead][leads] +approval and discussion as a meeting [agenda] item. + +Then coding can begin. + +## Implementation + +Contributors can begin implementing a feature before any of the above +steps have been completed, but _should not send a PR until +the [design proposal] has been merged_. + +See the [development guide] for instructions on setting up the +Kubernetes development environment. + +Implementation PRs should +- mention the issue of the associated design proposal, +- mention `@kubernetes/sig-multicluster-pr-reviews`, +- __include tests__. + +Small features and flag changes require only unit/integration tests, +while larger changes require both unit/integration tests and e2e tests. + +### Report progress + +_Leads need your help to ensure that progress is made to +get the feature into a [release]._ + +While working on the issue, leave a weekly update on the issue +including: + +1. What's finished? +2. What's part is being worked on now? +3. Anything blocking? + + +## Documentation + +_Let users know about cool new features by updating user facing documentation._ + +Depending on the contributor and size of the feature, this +may be done either by the same contributor that implemented the feature, +or another contributor who is more familiar with the existing docs +templates. + +## Release + +Several weeks before a Kubernetes release, development enters a stabilization +period where no new features are merged. For a feature to be accepted +into a release, it must be fully merged and tested by this time. If +your feature is not fully complete, _including tests_, it will have +to wait until the next release. + +## Merge state meanings + +- Merged: + - Ready to be implemented. +- Unmerged: + - Experience and design still being worked out. + - Not a high priority issue but may implement in the future: revisit + in 6 months. + - Unintentionally dropped. +- Closed: + - Not something we plan to implement in the proposed manner. + - Not something we plan to revisit in the next 12 months. + +## Escalation + +### If your bug issue is stuck + +If an issue isn't getting any attention and is unresolved, mention +`@kubernetes/sig-multicluster-bugs`. + +Highlight the severity and urgency of the issue. For severe issues +escalate by contacting sig [leads] and attending the [meeting]. + +### If your feature request issue is stuck + +If an issue isn't getting any attention and is unresolved, mention +`@kubernetes/sig-multicluster-feature-requests`. + +If a particular issue has a high impact for you or your business, +make sure this is clear on the bug, and reach out to the sig leads +directly. Consider attending the sig meeting to discuss over video +conference. + +### If your PR is stuck + +It may happen that your PR seems to be stuck without clear actionable +feedback for a week or longer. A PR _associated with a bug or design +proposal_ is much less likely to be stuck than a dangling PR. + +However, if it happens do the following: + +- If your PR is stuck for a week or more because it has never gotten any + comments, mention `@kubernetes/sig-multicluster-pr-reviews` and ask for attention. +- If your PR is stuck for a week or more _after_ it got comments, but + the attention has died down. Mention the reviewer and comment with + [`PTAL`]. + +If you are still not able to get any attention after a couple days, +escalate to sig [leads] by mentioning them. + +### If your design proposal issue is stuck + +It may happen that your design doc gets stuck without getting merged +or additional feedback. If you believe that your design is important +and has been dropped, or it is not moving forward, please add it to +the sig multicluster bi-weekly meeting [agenda] and mail the [group] saying +you'd like to discuss it. + +### General escalation instructions + +See the sig-multicluster [community page] for points of contact and meeting times: + +- attend the sig-multicluster [meeting] +- message one of the sig leads on [slack][slack-messages] (signup [here][slack-signup]) +- send an email to the _kubernetes-sig-multicluster@googlegroups.com_ [group]. + +## Use of [@mentions] + +- `@{any lead}` solicit opinion or advice from [leads]. +- `@kubernetes/sig-multicluster-bugs` sig-multicluster centric bugs. +- `@kubernetes/sig-multicluster-pr-reviews` triggers review of code fix PR. +- `@kubernetes/sig-multicluster-feature-requests` flags a feature request. +- `@kubernetes/sig-multicluster-proposals` flags a design proposal. + +[@mentions]: https://help.github.com/articles/basic-writing-and-formatting-syntax/#mentioning-users-and-teams +[Kubernetes Basics Tutorial]: https://kubernetes.io/docs/tutorials/kubernetes-basics +[PR]: https://help.github.com/articles/creating-a-pull-request +[`PTAL`]: https://en.wiktionary.org/wiki/PTAL +[agenda]: https://docs.google.com/document/d/18mk62nOXE_MCSSnb4yJD_8UadtzJrYyJxFwbrgabHe8/edit +[bug]: #bug-lifecycle +[community page]: /sig-multicluster +[design proposal]: #design-proposals +[design repo]: /contributors/design-proposals/sig-multicluster +[design template]: /contributors/design-proposals/sig-multicluster/template.md +[development guide]: /contributors/devel/development.md +[existing issue]: #adopt-an-issue +[feature repo]: https://github.com/kubernetes/features +[feature request]: #feature-requests +[feature]: https://github.com/kubernetes/features +[group]: https://groups.google.com/forum/#!forum/kubernetes-sig-multicluster +[issue]: https://github.com/kubernetes/kubernetes/issues +[multicluster_help_wanted_issues]: https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3A"help+wanted"+label%3Asig%2Fmulticluster +[kubectl concept docs]: https://git.k8s.io/kubernetes.github.io/docs/concepts/tools/kubectl +[kubectl docs]: https://kubernetes.io/docs/user-guide/kubectl-overview +[kubernetes/cmd/kubectl]: https://git.k8s.io/kubernetes/cmd/kubectl +[kubernetes/pkg/kubectl]: https://git.k8s.io/kubernetes/pkg/kubectl +[leads]: /sig-multicluster#leads +[management overview]: https://kubernetes.io/docs/concepts/tools/kubectl/object-management-overview +[meeting]: /sig-multicluster#meetings +[release]: #release +[slack-messages]: https://kubernetes.slack.com/messages/sig-multicluster +[slack-signup]: http://slack.k8s.io/ +[tests]: /contributors/devel/testing.md diff --git a/sig-multicluster/ONCALL.md b/sig-multicluster/ONCALL.md new file mode 100644 index 00000000..903845e7 --- /dev/null +++ b/sig-multicluster/ONCALL.md @@ -0,0 +1,76 @@ +# Overview + +We have an oncall rotation for Federation in the SIG. The role description is as +follows: + +* Ensure that the testgrid (https://k8s-testgrid.appspot.com/sig-multicluster) + is green. This person will be the point of contact if testgrid turns red. + Will identify the problem and fix it (most common scenarios: find culprit PR + and revert it or free quota by deleting leaked resources). Will also report + most common failure scenarios and suggest improvements. Its up to the sig or + individuals to prioritize and take up those tasks. + +The on-call playbook is available +[here](/contributors/devel/on-call-federation-build-cop.md) + +# Joining the rotation + +Add your name at the end of the current rotation schedule if you want to join +the rotation. Anyone is free to join as long as they can perform the expected +work described above. No special permissions are required but familiarity with +existing codebase is recommended. + +# Swapping the rotation + +If anyone is away on their oncall week (vacation, illness, etc), they are +responsible for finding someone to swap with (by sending a PR, approved by that +person). Swapping one week for another is usually relatively uncontentious. + +# Extending the rotation schedule + +Anyone can extend the existing schedule by assigning upcoming weeks to people in +the same order as the existing schedule. cc the rotation members on the PR so +that they know. Please extend the schedule unless there are atlease 2 people +assigned after you. + +# Current Oncall schedule + +``` +25 September - 1 October: Madhu (https://github.com/madhusudancs) +2 October - 8 October: Shashidhara (https://github.com/shashidharatd) +9 October - 15 October: Christian (https://github.com/csbell) +16 October - 22 October: Nikhil Jindal (https://github.com/nikhiljindal) +23 October - 29 October: Irfan (https://github.com/irfanurrehman) +30 October - 5 November: Maru (https://github.com/marun) +6 November - 12 November: Jonathan (https://github.com/perotinus) +(Madhu to be removed from next rotation) +``` + +# Past 5 rotation cycles + +``` +(Adding Irfan) +7 August - 13 August: Nikhil Jindal (https://github.com/nikhiljindal) +14 August - 20 August: Shashidhara (https://github.com/shashidharatd) +21 August - 27 August: Christian (https://github.com/csbell) +28 August - 3 September: Madhu (https://github.com/madhusudancs) +4 September - 10 September: Irfan (https://github.com/irfanurrehman) +11 September - 17 September: Maru (https://github.com/marun) +18 September - 24 September: Jonathan (https://github.com/perotinus) + + +(Adding Jonathan) +26 June - 2 July: Nikhil Jindal (https://github.com/nikhiljindal) +3 July - 9 July: Shashidhara (https://github.com/shashidharatd) +10 July - 16 July: Christian (https://github.com/csbell) +17 July - 23 July: Madhu (https://github.com/madhusudancs) +24 July - 30 July: Maru (https://github.com/marun) +31 July - 6 August: Jonathan (https://github.com/perotinus) + + +22-28 May: Nikhil Jindal (https://github.com/nikhiljindal) +29 May - 4 June: Shashidhara (https://github.com/shashidharatd) +5 June - 11 June: Madhusudan (https://github.com/madhusudancs) +12 June - 18 June: Maru (https://github.com/marun) +19 June - 25 June: Christian (https://github.com/csbell) +``` diff --git a/sig-multicluster/OWNERS b/sig-multicluster/OWNERS new file mode 100644 index 00000000..fca0e564 --- /dev/null +++ b/sig-multicluster/OWNERS @@ -0,0 +1,6 @@ +reviewers: + - sig-multicluster-leads +approvers: + - sig-multicluster-leads +labels: + - sig/multicluster diff --git a/sig-multicluster/README.md b/sig-multicluster/README.md new file mode 100644 index 00000000..eff87677 --- /dev/null +++ b/sig-multicluster/README.md @@ -0,0 +1,45 @@ +<!--- +This is an autogenerated file! + +Please do not edit this file directly, but instead make changes to the +sigs.yaml file in the project root. + +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md +--> +# Multicluster SIG + +Covers multi-cluster Kubernetes use cases and tooling. This includes: application resiliency against availability zone outages; hybrid clouds; spanning of multiple could providers; application migration from private to public clouds (and vice versa); and other similar subjects. This SIG was formerly called sig-federation and focused on the Federation project, but expanded its charter to all multi-cluster concerns in August 2017. + +## Meetings +* [Tuesdays at 9:30 PT (Pacific Time)](https://zoom.us/my/k8s.mc) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=9:30&tz=PT%20%28Pacific%20Time%29). + +Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/18mk62nOXE_MCSSnb4yJD_8UadtzJrYyJxFwbrgabHe8/edit). +Meeting recordings can be found [here](https://www.youtube.com/watch?v=iWKC3FsNHWg&list=PL69nYSiGNLP0HqgyqTby6HlDEz7i1mb0-). + +## Leads +* Christian Bell (**[@csbell](https://github.com/csbell)**), Google +* Quinton Hoole (**[@quinton-hoole](https://github.com/quinton-hoole)**), Huawei + +## Contact +* [Slack](https://kubernetes.slack.com/messages/sig-multicluster) +* [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-multicluster) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fmulticluster) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-multicluster-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-multicluster-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-multicluster-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-multicluster-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-multicluster-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-multicluster-feature-requests) | Feature Requests | +| @kubernetes/sig-multicluster-misc | [link](https://github.com/orgs/kubernetes/teams/sig-multicluster-misc) | General Discussion | +| @kubernetes/sig-multicluster-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-multicluster-pr-reviews) | PR Reviews | +| @kubernetes/sig-multicluster-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-multicluster-test-failures) | Test Failures and Triage | +| @kubernetes/sig-mutlicluster-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-mutlicluster-proposals) | Design Proposals | + +<!-- BEGIN CUSTOM CONTENT --> + +<!-- END CUSTOM CONTENT --> diff --git a/sig-network/OWNERS b/sig-network/OWNERS index 67dac83a..1939ca5c 100644 --- a/sig-network/OWNERS +++ b/sig-network/OWNERS @@ -1,6 +1,6 @@ reviewers: - - thockin - - dcbw + - sig-network-leads approvers: - - thockin - - dcbw + - sig-network-leads +labels: + - sig/network diff --git a/sig-network/README.md b/sig-network/README.md index acde1d8e..722e87fd 100644 --- a/sig-network/README.md +++ b/sig-network/README.md @@ -4,26 +4,42 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Network SIG Covers networking in Kubernetes. ## Meetings -* [Thursdays at 21:00 UTC](https://zoom.us/j/5806599998) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=21:00&tz=UTC). +* [Thursdays at 14:00 PT (Pacific Time)](https://zoom.us/j/5806599998) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=14:00&tz=PT%20%28Pacific%20Time%29). Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1_w77-zG_Xj0zYvEMfQZTQ-wPP4kXkpGD8smVtW_qqWM/edit). Meeting recordings can be found [here](https://www.youtube.com/watch?v=phCA5-vWkVM&list=PL69nYSiGNLP2E8vmnqo5MwPOY25sDWIxb). ## Leads -* [Tim Hockin](https://github.com/thockin), Google -* [Dan Williams](https://github.com/dcbw), Red Hat -* [Casey Davenport](https://github.com/caseydavenport), Tigera +* Tim Hockin (**[@thockin](https://github.com/thockin)**), Google +* Dan Williams (**[@dcbw](https://github.com/dcbw)**), Red Hat +* Casey Davenport (**[@caseydavenport](https://github.com/caseydavenport)**), Tigera ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-network) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-network) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fnetwork) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-network-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-network-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-network-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-network-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-network-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-network-feature-requests) | Feature Requests | +| @kubernetes/sig-network-misc | [link](https://github.com/orgs/kubernetes/teams/sig-network-misc) | General Discussion | +| @kubernetes/sig-network-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-network-pr-reviews) | PR Reviews | +| @kubernetes/sig-network-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-network-proposals) | Design Proposals | +| @kubernetes/sig-network-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-network-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> ## Areas of Responsibility diff --git a/sig-node/OWNERS b/sig-node/OWNERS index 81023f2d..ab6d8dd5 100644 --- a/sig-node/OWNERS +++ b/sig-node/OWNERS @@ -1,6 +1,6 @@ reviewers: - - dchen1107 - - derekawaynecarr + - sig-node-leads approvers: - - dchen1107 - - derekawaynecarr + - sig-node-leads +labels: + - sig/node diff --git a/sig-node/README.md b/sig-node/README.md index 5d58ed13..514661c5 100644 --- a/sig-node/README.md +++ b/sig-node/README.md @@ -4,24 +4,39 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Node SIG ## Meetings -* [Tuesdays at 17:00 UTC](https://plus.google.com/hangouts/_/google.com/sig-node-meetup?authuser=0) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=17:00&tz=UTC). +* [Tuesdays at 10:00 PT (Pacific Time)](https://plus.google.com/hangouts/_/google.com/sig-node-meetup?authuser=0) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=10:00&tz=PT%20%28Pacific%20Time%29). Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1Ne57gvidMEWXR70OxxnRkYquAoMpt56o75oZtg-OeBg/edit?usp=sharing). Meeting recordings can be found [here](https://www.youtube.com/watch?v=FbKOI9-x9hI&list=PL69nYSiGNLP1wJPj5DYWXjiArF-MJ5fNG). ## Leads -* [Dawn Chen](https://github.com/dchen1107), Google -* [Derek Carr](https://github.com/derekwaynecarr), Red Hat +* Dawn Chen (**[@dchen1107](https://github.com/dchen1107)**), Google +* Derek Carr (**[@derekwaynecarr](https://github.com/derekwaynecarr)**), Red Hat ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-node) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-node) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fnode) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-node-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-node-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-node-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-node-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-node-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-node-feature-requests) | Feature Requests | +| @kubernetes/sig-node-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-node-pr-reviews) | PR Reviews | +| @kubernetes/sig-node-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-node-proposals) | Design Proposals | +| @kubernetes/sig-node-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-node-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> ## Goals diff --git a/sig-on-premise/OWNERS b/sig-on-premise/OWNERS index 0078ef84..5df1d027 100644 --- a/sig-on-premise/OWNERS +++ b/sig-on-premise/OWNERS @@ -1,8 +1,6 @@ reviewers: - - zen - - marcoceppi - - dghubble + - sig-on-premise-leads approvers: - - zen - - marcoceppi - - dghubble + - sig-on-premise-leads +labels: + - sig/onprem diff --git a/sig-on-premise/README.md b/sig-on-premise/README.md index e7e295da..6d0e962b 100644 --- a/sig-on-premise/README.md +++ b/sig-on-premise/README.md @@ -4,7 +4,7 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # On Premise SIG @@ -17,23 +17,28 @@ Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/ Meeting recordings can be found [here](https://www.youtube.com/watch?v=dyUWqqNYUio&list=PL69nYSiGNLP2MvqC6NeegrgtOl5s1KlYa). ## Leads -* [Tomasz Napierala](https://github.com/zen), Mirantis -* [Marco Ceppi](https://github.com/marcoceppi), Canonical -* [Dalton Hubble](https://github.com/dghubble), CoreOS +* Marco Ceppi (**[@marcoceppi](https://github.com/marcoceppi)**), Canonical +* Dalton Hubble (**[@dghubble](https://github.com/dghubble)**), CoreOS ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-onprem) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-on-prem) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fonprem) ## GitHub Teams -* [@onprem-misc](https://github.com/kubernetes/teams/onprem-misc) -* [@onprem-test-failures](https://github.com/kubernetes/teams/onprem-test-failures) -* [@onprem-bugs](https://github.com/kubernetes/teams/onprem-bugs) -* [@onprem-feature-requests](https://github.com/kubernetes/teams/onprem-feature-requests) -* [@onprem-proposals](https://github.com/kubernetes/teams/onprem-proposals) -* [@onprem-pr-reviews](https://github.com/kubernetes/teams/onprem-pr-reviews) -* [@onprem-api-reviews](https://github.com/kubernetes/teams/onprem-api-reviews) +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-onprem-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-onprem-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-onprem-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-onprem-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-onprem-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-onprem-feature-requests) | Feature Requests | +| @kubernetes/sig-onprem-misc | [link](https://github.com/orgs/kubernetes/teams/sig-onprem-misc) | General Discussion | +| @kubernetes/sig-onprem-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-onprem-pr-reviews) | PR Reviews | +| @kubernetes/sig-onprem-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-onprem-proposals) | Design Proposals | +| @kubernetes/sig-onprem-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-onprem-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> diff --git a/sig-openstack/OWNERS b/sig-openstack/OWNERS index 1f7a6b84..32f44bdd 100644 --- a/sig-openstack/OWNERS +++ b/sig-openstack/OWNERS @@ -1,6 +1,6 @@ reviewers: - - idvoretskyi - - xsgordon + - sig-openstack-leads approvers: - - idvoretskyi - - xsgordon + - sig-openstack-leads +labels: + - sig/openstack diff --git a/sig-openstack/README.md b/sig-openstack/README.md index 826f8aa1..afd85ed0 100644 --- a/sig-openstack/README.md +++ b/sig-openstack/README.md @@ -4,35 +4,41 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # OpenStack SIG Coordinates the cross-community efforts of the OpenStack and Kubernetes communities. This includes OpenStack-related contributions to Kubernetes projects with OpenStack as: a deployment platform for Kubernetes; a service provider for Kubernetes; a collection of applications to run on Kubernetes. ## Meetings -* [Wednesdays at 15:00 UTC](https://zoom.us/j/417251241) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=15:00&tz=UTC). +* [Thursdays at 00:00 UTC](https://zoom.us/j/417251241) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=00:00&tz=UTC). Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1iAQ3LSF_Ky6uZdFtEZPD_8i6HXeFxIeW4XtGcUJtPyU/edit?usp=sharing_eixpa_nl&ts=588b986f). Meeting recordings can be found [here](https://www.youtube.com/watch?v=iCfUx7ilh0E&list=PL69nYSiGNLP20iTSChQ_i2QQmTBl3M7ax). ## Leads -* [Ihor Dvoretskyi](https://github.com/idvoretskyi), Mirantis -* [Steve Gordon](https://github.com/xsgordon), Red Hat +* Ihor Dvoretskyi (**[@idvoretskyi](https://github.com/idvoretskyi)**), CNCF +* Steve Gordon (**[@xsgordon](https://github.com/xsgordon)**), Red Hat ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-openstack) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-openstack) - -## GitHub Teams -* [@sig-openstack-misc](https://github.com/kubernetes/teams/sig-openstack-misc) -* [@sig-openstack-test-failures](https://github.com/kubernetes/teams/sig-openstack-test-failures) -* [@sig-openstack-bugs](https://github.com/kubernetes/teams/sig-openstack-bugs) -* [@sig-openstack-feature-requests](https://github.com/kubernetes/teams/sig-openstack-feature-requests) -* [@sig-openstack-proposals](https://github.com/kubernetes/teams/sig-openstack-proposals) -* [@sig-openstack-pr-reviews](https://github.com/kubernetes/teams/sig-openstack-pr-reviews) -* [@sig-openstack-api-reviews](https://github.com/kubernetes/teams/sig-openstack-api-reviews) - +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fopenstack) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-openstack-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-openstack-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-openstack-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-openstack-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-openstack-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-openstack-feature-requests) | Feature Requests | +| @kubernetes/sig-openstack-misc | [link](https://github.com/orgs/kubernetes/teams/sig-openstack-misc) | General Discussion | +| @kubernetes/sig-openstack-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-openstack-pr-reviews) | PR Reviews | +| @kubernetes/sig-openstack-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-openstack-proposals) | Design Proposals | +| @kubernetes/sig-openstack-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-openstack-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> diff --git a/sig-product-management/OWNERS b/sig-product-management/OWNERS index 9a160d43..5c66db0b 100644 --- a/sig-product-management/OWNERS +++ b/sig-product-management/OWNERS @@ -1,8 +1,6 @@ reviewers: - - apsinha - - idvoretskyi - - calebamiles + - sig-product-management-leads approvers: - - apsinha - - idvoretskyi - - calebamiles + - sig-product-management-leads +labels: + - sig/pm diff --git a/sig-product-management/README.md b/sig-product-management/README.md index 864df588..9438bfcf 100644 --- a/sig-product-management/README.md +++ b/sig-product-management/README.md @@ -4,7 +4,7 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Product Management SIG @@ -20,13 +20,14 @@ Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/ Meeting recordings can be found [here](https://www.youtube.com/watch?v=VcdjaZAol2I&list=PL69nYSiGNLP3EBqpUGVsK1sMgUZVomfEQ). ## Leads -* [Aparna Sinha](https://github.com/apsinha), Google -* [Ihor Dvoretskyi](https://github.com/idvoretskyi), Mirantis -* [Caleb Miles](https://github.com/calebamiles), CoreOS +* Aparna Sinha (**[@apsinha](https://github.com/apsinha)**), Google +* Ihor Dvoretskyi (**[@idvoretskyi](https://github.com/idvoretskyi)**), CNCF +* Caleb Miles (**[@calebamiles](https://github.com/calebamiles)**), CoreOS ## Contact * [Slack](https://kubernetes.slack.com/messages/kubernetes-pm) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-pm) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fnone) <!-- BEGIN CUSTOM CONTENT --> ## Common activities diff --git a/sig-release/OWNERS b/sig-release/OWNERS index 320c2abb..9d8e7403 100644 --- a/sig-release/OWNERS +++ b/sig-release/OWNERS @@ -1,4 +1,6 @@ reviewers: - - calebamiles + - sig-release-leads approvers: - - calebamiles + - sig-release-leads +labels: + - sig/release diff --git a/sig-release/README.md b/sig-release/README.md index 71ce64bb..a7898fd8 100644 --- a/sig-release/README.md +++ b/sig-release/README.md @@ -4,7 +4,7 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Release SIG @@ -16,12 +16,30 @@ Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/ Meeting recordings can be found [here](https://www.youtube.com/watch?v=I0KbWz8MTMk&list=PL69nYSiGNLP3QKkOsDsO6A0Y1rhgP84iZ). ## Leads -* [Phillip Wittrock](https://github.com/pwittrock), Google -* [Caleb Miles](https://github.com/calebamiles), CoreOS +* Phillip Wittrock (**[@pwittrock](https://github.com/pwittrock)**), Google +* Caleb Miles (**[@calebamiles](https://github.com/calebamiles)**), CoreOS ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-release) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-release) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Frelease) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-release-admins | [link](https://github.com/orgs/kubernetes/teams/sig-release-admins) | Release Team Admins | +| @kubernetes/sig-release-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-release-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-release-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-release-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-release-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-release-feature-requests) | Feature Requests | +| @kubernetes/sig-release-members | [link](https://github.com/orgs/kubernetes/teams/sig-release-members) | Release Team Members | +| @kubernetes/sig-release-misc | [link](https://github.com/orgs/kubernetes/teams/sig-release-misc) | General Discussion | +| @kubernetes/sig-release-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-release-pr-reviews) | PR Reviews | +| @kubernetes/sig-release-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-release-proposals) | Design Proposals | +| @kubernetes/sig-release-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-release-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> [SIG Release][] has moved! diff --git a/sig-scalability/OWNERS b/sig-scalability/OWNERS index 75570a98..2b68b875 100644 --- a/sig-scalability/OWNERS +++ b/sig-scalability/OWNERS @@ -1,8 +1,6 @@ reviewers: - - wojtek-t - - countspongebob - - jbeda + - sig-scalability-leads approvers: - - wojtek-t - - countspongebob - - jbeda + - sig-scalability-leads +labels: + - sig/scalability diff --git a/sig-scalability/README.md b/sig-scalability/README.md index ad78f6a3..88ac58e8 100644 --- a/sig-scalability/README.md +++ b/sig-scalability/README.md @@ -4,30 +4,54 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Scalability SIG Responsible for answering scalability related questions such as: What size clusters do we think that we should support with Kubernetes in the short to medium term? How performant do we think that the control system should be at scale? What resource overhead should the Kubernetes control system reasonably consume? -For more details about our objectives please review our [Scaling And Performance Goals](https://github.com/kubernetes/community/blob/master/sig-scalability/goals.md) +For more details about our objectives please review our [Scaling And Performance Goals](https://git.k8s.io/community/sig-scalability/goals.md) ## Meetings -* [Thursdays at 16:00 UTC](https://zoom.us/j/989573207) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=16:00&tz=UTC). +* [Thursdays at 16:00 UTC](https://zoom.us/j/989573207) (bi-weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=16:00&tz=UTC). Meeting notes and Agenda can be found [here](https://docs.google.com/a/bobsplanet.com/document/d/1hEpf25qifVWztaeZPFmjNiJvPo-5JX1z0LSvvVY5G2g/edit?usp=drive_web). Meeting recordings can be found [here](https://www.youtube.com/watch?v=NDP1uYyom28&list=PL69nYSiGNLP2X-hzNTqyELU6jYS3p10uL). ## Leads -* [Wojciech Tyczynski](https://github.com/wojtek-t), Google -* [Bob Wise](https://github.com/countspongebob), Samsung SDS -* [Joe Beda](https://github.com/jbeda), Heptio +* Wojciech Tyczynski (**[@wojtek-t](https://github.com/wojtek-t)**), Google +* Bob Wise (**[@countspongebob](https://github.com/countspongebob)**), Samsung SDS ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-scalability) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-scale) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fscalability) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-scalability-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-scalability-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-scalability-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-scalability-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-scalability-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-scalability-feature-requests) | Feature Requests | +| @kubernetes/sig-scalability-misc | [link](https://github.com/orgs/kubernetes/teams/sig-scalability-misc) | General Discussion | +| @kubernetes/sig-scalability-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-scalability-pr-reviews) | PR Reviews | +| @kubernetes/sig-scalability-proprosals | [link](https://github.com/orgs/kubernetes/teams/sig-scalability-proprosals) | Design Proposals | +| @kubernetes/sig-scalability-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-scalability-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> +## Remaining 2017 Meeting Dates + * 10/5 + * 10/19 + * 11/2 + * 11/16 + * 11/30 + * 12/14 + * 12/28 - no meeting + ## Scalability SLOs We officially support two different SLOs: diff --git a/sig-scalability/extending_slo.md b/sig-scalability/extending_slo.md index 7cbc0f25..5cbbb87f 100644 --- a/sig-scalability/extending_slo.md +++ b/sig-scalability/extending_slo.md @@ -9,6 +9,7 @@ which are enough to guarantee that cluster doesn't feel completely dead, but not We're going to define more SLOs based on most important indicators, and standardize the format in which we speak about our objectives. Our SLOs need to have two properties: - They need to be testable, i.e. we need to have a benchmark to measure if it's met, - They need to be expressed in a way that's possible to understand by a user not intimately familiar with the system internals, i.e. formulation can't depend on some arcane knowledge. + On the other hand we do not require that: - SLOs are possible to monitor in a running cluster, i.e. not all SLOs need to be easily translatable to SLAs. Being able to benchmark is enough for us. @@ -25,10 +26,12 @@ This includes current SLOs: - API call latency - E2e Pod startup latency -By churn we understand a measure of amount changes happening in the cluster. It's formal(-ish) definition will follow, but informally it can be thought about as number of user-issued requests per second plus number of pods affected by those requests. +By churn we understand a measure of amount changes happening in the cluster. Its formal(-ish) definition will follow, but informally it can be thought about as number of user-issued requests per second plus number of pods affected by those requests. More formally churn per second is defined as: +``` #Pod creations + #PodSpec updates + #user originated requests in a given second +``` The last part is necessary only to get rid of situations when user is spamming API server with various requests. In ordinary circumstances we expect it to be in the order of 1-2. ## Burst SLOs @@ -38,8 +41,8 @@ This includes the new SLO: - Pod startup throughput ## Environment -Kubernetes cluster in which we benchmark SLOs need to meet following criteria: -- Run a single master machine appropriately sized +A Kubernetes cluster in which we benchmark SLOs needs to meet the following criteria: +- Run a single appropriately sized master machine - Main etcd runs as a single instance on the master machine - Events are stored in a separate etcd instance running on the master machine - Kubernetes version is at least 1.X.Y @@ -59,7 +62,7 @@ All our performance SLOs should be defined using the following template: _One-two sentences describing the SLO, that's possible to understand by the majority of the community_ ### User Stories -_Few user stories showing in what situations users might be interested in this SLO, and why other ones are not enough_ +_A Few user stories showing in what situations users might be interested in this SLO, and why other ones are not enough_ ## Full definition ### Test description diff --git a/sig-scalability/goals.md b/sig-scalability/goals.md index 87255c60..1f497759 100644 --- a/sig-scalability/goals.md +++ b/sig-scalability/goals.md @@ -77,6 +77,6 @@ Configuration of the control plane for cluster testing varies by provider, and t ## Open Questions 1. **What, if any, reasonable use cases exist for very large numbers of very small nodes (e.g. for isolation reasons - multitenant)? Based on comments so far, it seems that the answer is yes, and needs to be addressed.**<br> -The above scaling goals explicitly accommodate a maximum of 5,000 nodes. Do we need a special case for larger numbers of small nodes (e.g. 200,000 single-core nodes). The latter already fits within our other scalability limits (cores per cluster, pods per cluster), so it might not be more difficult to achieve than those. Two example use cases I've heard anecdotally are (a) running e.g. large numbers of customers' small, largely idle wordpress instances, one per node, and (b) giving away limited numbers of free +The above scaling goals explicitly accommodate a maximum of 5,000 nodes. Do we need a special case for larger numbers of small nodes (e.g. 200,000 single-core nodes). The latter already fits within our other scalability limits (cores per cluster, pods per cluster), so it might not be more difficult to achieve than those. Two example use cases I've heard anecdotally are (a) running e.g. large numbers of customers' small, largely idle wordpress instances, one per node, and (b) giving away limited numbers of free containers to large numbers of customers for promotional reasons (similar to AWS free tier micro instances). 2. **What, if any, reasonable use cases exist for very large numbers of very small containers per core?**<br> E.g. are perhaps hundreds of containers per core useful for specialized applications? We speculate that building large numbers of very small yet useful containers each within say 20MB of RAM, and <1% of a core is difficult (as opposed to very small supportive/sidecar pods alongside larger pods, which is totally legitimate, and supported). diff --git a/sig-scalability/provider-configs.md b/sig-scalability/provider-configs.md index d35dc452..afe183b0 100644 --- a/sig-scalability/provider-configs.md +++ b/sig-scalability/provider-configs.md @@ -120,9 +120,9 @@ proposed</td> * Leader election results are non-deterministic on on a typical cluster, and a config would be best served to be configured as worst-case. Not presently known whether there are performance impacts resulting from leader election resulting in either co-location or distribution of those components. -* Improving the cluster performance loading to match production deployment scenarios is critical on-going work, especially clusterloader: [https://github.com/kubernetes/perf-tests/tree/master/clusterloader](https://github.com/kubernetes/perf-tests/tree/master/clusterloader) +* Improving the cluster performance loading to match production deployment scenarios is critical on-going work, especially clusterloader: [https://git.k8s.io/perf-tests/clusterloader](https://git.k8s.io/perf-tests/clusterloader) -* Multi-zone / multi-az deployments are often used to manage large clusters, but for testing/scalability efforts the target is intentionally a single AZ. This keeps greater consistency between environments that do and don’t support AZ-based deployments. Failures during scalability testing are outside the SIG charter. Protecting against network partitioning and improving total cluster availability (one of the key benefits to a multi-AZ strategy) are currently out scope for the Scalability SIG efforts. +* Multi-zone / multi-az deployments are often used to manage large clusters, but for testing/scalability efforts the target is intentionally a single Availability Zone. This keeps greater consistency between environments that do and don’t support AZ-based deployments. Failures during scalability testing are outside the SIG charter. Protecting against network partitioning and improving total cluster availability (one of the key benefits to a multi-AZ strategy) are currently out scope for the Scalability SIG efforts. * Scalability issues on very large clusters of actual nodes (instead of kubemark simulations) are real. Efforts to improve large cluster networking performance e.g. IPVS are important, and will be interesting areas for cross-SIG collaboration. diff --git a/sig-scalability/thresholds.md b/sig-scalability/thresholds.md index 6d40f7cb..a17a0fa1 100644 --- a/sig-scalability/thresholds.md +++ b/sig-scalability/thresholds.md @@ -20,7 +20,7 @@ future or not. ## Kubernetes thresholds We start with explicit definition of quantities and thresholds we assume are -satisfied in the cluster. This is followed by an explanations for some of those. +satisfied in the cluster. This is followed by an explanation for some of those. Important notes about the numbers: 1. In most cases, exceeding these thresholds doesn’t mean that the cluster fails over - it just means that its overall performance degrades. @@ -42,7 +42,7 @@ Important notes about the numbers: | Total number of all objects | 250000 | | 1000000 | | Number of nodes | 5000 | | 5000 | | Number of pods | 150000 | | 500000 | -| Number of pods per node<sup>1</sup> | 100 | | 100 | +| Number of pods per node<sup>1</sup> | 110 | | 500 | | Number of pods per core<sup>1</sup> | 10 | | 10 | | Number of namespaces (ns) | 10000 | | 100000 | | Number of pods per ns | 15000 | | 50000 | diff --git a/sig-scheduling/OWNERS b/sig-scheduling/OWNERS index dd1261ee..b3248766 100644 --- a/sig-scheduling/OWNERS +++ b/sig-scheduling/OWNERS @@ -1,6 +1,6 @@ reviewers: - - davidopp - - timothysc + - sig-scheduling-leads approvers: - - davidopp - - timothysc + - sig-scheduling-leads +labels: + - sig/scheduling diff --git a/sig-scheduling/README.md b/sig-scheduling/README.md index dd9c950e..26520e7c 100644 --- a/sig-scheduling/README.md +++ b/sig-scheduling/README.md @@ -4,7 +4,7 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Scheduling SIG @@ -17,12 +17,28 @@ Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/ Meeting recordings can be found [here](https://www.youtube.com/watch?v=PweKj6SU7UA&list=PL69nYSiGNLP2vwzcCOhxrL3JVBc-eaJWI). ## Leads -* [David Oppenheimer](https://github.com/davidopp), Google -* [Timothy St. Clair](https://github.com/timothysc), Red Hat +* David Oppenheimer (**[@davidopp](https://github.com/davidopp)**), Google +* Timothy St. Clair (**[@timothysc](https://github.com/timothysc)**), Red Hat ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-scheduling) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-scheduling) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fscheduling) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-scheduling-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-scheduling-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-scheduling-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-scheduling-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-scheduling-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-scheduling-feature-requests) | Feature Requests | +| @kubernetes/sig-scheduling-misc | [link](https://github.com/orgs/kubernetes/teams/sig-scheduling-misc) | General Discussion | +| @kubernetes/sig-scheduling-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-scheduling-pr-reviews) | PR Reviews | +| @kubernetes/sig-scheduling-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-scheduling-proposals) | Design Proposals | +| @kubernetes/sig-scheduling-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-scheduling-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> diff --git a/sig-service-catalog/OWNERS b/sig-service-catalog/OWNERS index dd06eb38..5c6b18ed 100644 --- a/sig-service-catalog/OWNERS +++ b/sig-service-catalog/OWNERS @@ -1,10 +1,6 @@ reviewers: - - pmorie - - arschles - - vaikas-google - - duglin + - sig-service-catalog-leads approvers: - - pmorie - - arschles - - vaikas-google - - duglin + - sig-service-catalog-leads +labels: + - sig/service-catalog diff --git a/sig-service-catalog/README.md b/sig-service-catalog/README.md index 1cb145c4..7ef95caf 100644 --- a/sig-service-catalog/README.md +++ b/sig-service-catalog/README.md @@ -4,7 +4,7 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Service Catalog SIG @@ -13,18 +13,34 @@ To develop a Kubernetes API for the CNCF service broker and Kubernetes broker im ## Meetings * [Mondays at 20:00 UTC](https://zoom.us/j/7201225346) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=20:00&tz=UTC). -Meeting notes and Agenda can be found [here](http://goo.gl/A0m24V). +Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/17xlpkoEbPR5M6P5VDzNx17q6-IPFxKyebEekCGYiIKM/edit). Meeting recordings can be found [here](https://www.youtube.com/watch?v=ukPj1sFFkr0&list=PL69nYSiGNLP2k9ZXx9E1MvRSotFDoHUWs). ## Leads -* [Paul Morie](https://github.com/pmorie), Red Hat -* [Aaron Schlesinger](https://github.com/arschles), Microsoft -* [Ville Aikas](https://github.com/vaikas-google), Google -* [Doug Davis](https://github.com/duglin), IBM +* Paul Morie (**[@pmorie](https://github.com/pmorie)**), Red Hat +* Aaron Schlesinger (**[@arschles](https://github.com/arschles)**), Microsoft +* Ville Aikas (**[@vaikas-google](https://github.com/vaikas-google)**), Google +* Doug Davis (**[@duglin](https://github.com/duglin)**), IBM ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-service-catalog) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-service-catalog) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fservice-catalog) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-service-catalog-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-service-catalog-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-service-catalog-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-service-catalog-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-service-catalog-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-service-catalog-feature-requests) | Feature Requests | +| @kubernetes/sig-service-catalog-misc | [link](https://github.com/orgs/kubernetes/teams/sig-service-catalog-misc) | General Discussion | +| @kubernetes/sig-service-catalog-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-service-catalog-pr-reviews) | PR Reviews | +| @kubernetes/sig-service-catalog-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-service-catalog-proposals) | Design Proposals | +| @kubernetes/sig-service-catalog-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-service-catalog-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> diff --git a/sig-storage/1.3-retrospective/2016-03-28_Storage-SIG-F2F_Notes.pdf b/sig-storage/1.3-retrospective/2016-03-28_Storage-SIG-F2F_Notes.pdf Binary files differindex 95522475..7c972d69 100644 --- a/sig-storage/1.3-retrospective/2016-03-28_Storage-SIG-F2F_Notes.pdf +++ b/sig-storage/1.3-retrospective/2016-03-28_Storage-SIG-F2F_Notes.pdf diff --git a/sig-storage/1.3-retrospective/README.md b/sig-storage/1.3-retrospective/README.md index 52e28490..b9bd35cd 100644 --- a/sig-storage/1.3-retrospective/README.md +++ b/sig-storage/1.3-retrospective/README.md @@ -6,10 +6,10 @@ **Collaborators:** Saad Ali ([@saad-ali](https://github.com/saad-ali)), Paul Morie ([@pmorie](https://github.com/pmorie)), Tim Hockins ([@thockin](https://github.com/thockin)), Steve Watt ([@wattsteve](https://github.com/wattsteve)) **Links:** -* [1.3 Schedule Dates](https://github.com/kubernetes/features/blob/master/release-1.3/release-1.3.md) +* [1.3 Schedule Dates](https://git.k8s.io/sig-release/releases/release-1.3/release-1.3.md) ## Purpose -This document is intended to chronicle the decisions made by the [Storage SIG](https://github.com/kubernetes/community/blob/master/sig-storage/README.md) near the end of the Kubernetes 1.3 release with the storage stack that were not well understood by the wider community. This document should explain those decisions, why the SIG made the exception, detail the impact, and offer lessons learned for the future. +This document is intended to chronicle the decisions made by the [Storage SIG](/sig-storage/README.md) near the end of the Kubernetes 1.3 release with the storage stack that were not well understood by the wider community. This document should explain those decisions, why the SIG made the exception, detail the impact, and offer lessons learned for the future. ## What Problem Were We Trying to Solve? Kubernetes 1.2 had numerous problems and issues with the storage framework that arose from organic growth of the architecture as it tackled numerous new features it was not initially designed for. There were race conditions, maintenance and stability issues, and architectural problems with all major components of the storage stack including the Persistent Volume (PV) & Persistent Volume Claim (PVC) controller and the attach/detach and mount/unmount logic. @@ -18,11 +18,11 @@ The PV/PVC controller handles the connection of provisioned storage volumes to a A characteristic list of issues (as not all of them were well captured in GitHub issues) include: -1. Approximately a 5% rate of incidents under controlled conditions where operations related to Claims binding to Persistent Volumes would fail. +1. Approximately a 5% rate of incidents under controlled conditions where operations related to Claims binding to Persistent Volumes would fail. 2. Rapid creation and deletion of pods referencing the same volume could result in attach/detach events being triggered out of order resulting in detaching of volumes in use (resulting in data loss/corruption). The current 1.2 work around was to fail the operation. This led to surprises and failures in launching pods that referenced the same volume. 3. Item #2 created instability in use of multiple pods referencing the same Volume (a supported feature) even when only one pod uses it at a time ([#19953](https://github.com/kubernetes/kubernetes/issues/19953)) 4. Hiccups in the operation flow of binding the Claims to Volumes resulted in timeouts of tens of minutes. -5. External object bleeding. Much of the logic was centered on a state machine that lived in the kubelet. Other kube components had to be aware of the state machine and other aspects of the binding framework to use Volumes. +5. External object bleeding. Much of the logic was centered on a state machine that lived in the kubelet. Other kube components had to be aware of the state machine and other aspects of the binding framework to use Volumes. 6. Maintenance was difficult as this work was implemented in three different controllers that spread the logic for provisioning, binding, and recycling Volumes. 7. Kubelet failures on the Node could “strand” storage. Requiring users to manually unmount storage. 8. A pod's long running detach routine could impact other pods as the operations run synchronously in the kubelet sync loop. @@ -41,7 +41,7 @@ Below are the Github Issues that were filed for this area: ## How Did We Solve the Problem? Addressing these issues was the main deliverable for storage in 1.3. This required an in depth rewrite of several components. -Early in the 1.3 development cycle (March 28 to April 1, 2016) several community members in the Storage SIG met at a week long face-to-face summit at Google's office in Mountain View to address these issues. A plan was established to approach the attach/detach/mount/unmount issues as a deliberate effort with contributors already handling the design. Since that work was already in flight and a plan established, the majority of the summit was devoted to resolving the PV/PVC controller issues. Meeting notes were captured [in this document](https://github.com/kubernetes/community/blob/master/sig-storage/1.3-retrospective/2016-03-28_Storage-SIG-F2F_Notes.pdf). +Early in the 1.3 development cycle (March 28 to April 1, 2016) several community members in the Storage SIG met at a week long face-to-face summit at Google's office in Mountain View to address these issues. A plan was established to approach the attach/detach/mount/unmount issues as a deliberate effort with contributors already handling the design. Since that work was already in flight and a plan established, the majority of the summit was devoted to resolving the PV/PVC controller issues. Meeting notes were captured [in this document](/sig-storage/1.3-retrospective/2016-03-28_Storage-SIG-F2F_Notes.pdf). Three projects were planned to fix the issues outlined above: * PV/PVC Controller Redesign (a.k.a. Provisioner/Binder/Recycler controller) @@ -52,7 +52,7 @@ At the end of the design summit, the attendees of the summit agreed to pseudo co Resources were established for the PV/PVC controller rework at the conclusion of the design summit and the existing resources on the attach/detach/mount/unmount work deemed acceptable to complete the other two projects. -At this point, a group of engineers were assigned to work on the three efforts that compromised the overhaul. The plan was to not only include development work but comprehensive testing with time to have the functionality “soak” weeks before 1.3 shipped. These engineers were composed of a hybrid team of Red Hat and Google. The allocation of work made making all three sub deliverables in 1.3 aggressive but reasonable. +At this point, a group of engineers were assigned to work on the three efforts that compromised the overhaul. The plan was to not only include development work but comprehensive testing with time to have the functionality “soak” weeks before 1.3 shipped. These engineers were composed of a hybrid team of Red Hat and Google. The allocation of work made making all three sub deliverables in 1.3 aggressive but reasonable. Near the end of 1.3 development, on May 13, 2016, approximately one week prior to code freeze, a key engineer for this effort left the project. This disrupted the Kubelet Volume Redesign effort. The PV/PVC controller was complete (PR [#24331](https://github.com/kubernetes/kubernetes/pull/24331)) and committed at this point. However the Attach/Detach Controller was dependent on the Kubelet Volume Redesign and was impacted. @@ -60,10 +60,10 @@ The leads involved with the projects met and the Kubelet Volume Redesign work wa The Kubelet Volume Redesign involved changing fundamental assumptions of data flow and volume operations in kubelet. The high level change introduced a new volume manager in kubelet that handled mount/unmount logic and enabled attach/detach logic to be offloaded to the master (by default, while retaining the ability for kubelet to do attach/detach on its own). The remaining work to complete the effort was the kubelet volume redesign PR ([#26801](https://github.com/kubernetes/kubernetes/pull/26801)). This combined with the attach/detach controller (PR [#25457](https://github.com/kubernetes/kubernetes/pull/25457)) were substantial changes to the stack. -## Impact: +## Impact: 1. **Release delay** - * The large amount of churn so late in the release with little stabilization time resulted in the delay of the release by one week: The Kubernetes 1.3 release [was targeted](https://github.com/kubernetes/features/blob/master/release-1.3/release-1.3.md) for June 20 to June 24, 2016. It ended up [going out on July 1, 2016](https://github.com/kubernetes/kubernetes/releases/tag/v1.3.0). This was mostly due to the time to resolve a data corruption issue on ungracefully terminated pods caused by detaching of mounted volumes ([#27691](https://github.com/kubernetes/kubernetes/issues/27691)). A large number of the bugs introduced in the release were fixed in the 1.3.4 release which [was cut on August 1, 2016](https://github.com/kubernetes/kubernetes/releases/tag/v1.3.4). + * The large amount of churn so late in the release with little stabilization time resulted in the delay of the release by one week: The Kubernetes 1.3 release [was targeted](https://git.k8s.io/sig-release/releases/release-1.3/release-1.3.md) for June 20 to June 24, 2016. It ended up [going out on July 1, 2016](https://github.com/kubernetes/kubernetes/releases/tag/v1.3.0). This was mostly due to the time to resolve a data corruption issue on ungracefully terminated pods caused by detaching of mounted volumes ([#27691](https://github.com/kubernetes/kubernetes/issues/27691)). A large number of the bugs introduced in the release were fixed in the 1.3.4 release which [was cut on August 1, 2016](https://github.com/kubernetes/kubernetes/releases/tag/v1.3.4). 2. **Instability in 1.3's Storage stack** * The Kubelet volume redesign shipped in 1.3.0 with several bugs. These were mostly due to unexpected interactions between the new functionality and other Kubernetes components. For example, secrets were handled serially not in parallel, namespace dependencies were not well understood, etc. Most of these issues were quickly identified and addressed but waited for 1.3 patch releases. * Issues related to this include: @@ -91,6 +91,6 @@ The value of the feature freeze date is to ensure the release has time to stabil * Status: [Planned for 1.5](https://docs.google.com/document/d/1-u1UA8mBiPZiyYUi7U7Up_e-afVegKmuhmc7fpVQ9hc/edit?ts=57bcd3d4&pli=1) * Discussed at [Storage-SIG F2F meeting held August 10, 2016](https://docs.google.com/document/d/1qVL7UE7TtZ_D3P4F7BeRK4mDOvYskUjlULXmRJ4z-oE/edit). See [notes](https://docs.google.com/document/d/1vA5ul3Wy4GD98x3GZfRYEElfV4OE8dBblSK4rnmrE_M/edit#heading=h.amd7ks7tpscg). 2. Establish a formal exception process for merging large changes after feature complete dates. - * Status: [Drafted as of 1.4](https://github.com/kubernetes/features/blob/master/EXCEPTIONS.md) + * Status: [Drafted as of 1.4](https://git.k8s.io/features/EXCEPTIONS.md) Kubernetes is an incredibly fast moving project, with hundreds of active contributors creating a solution that thousands of organization rely on. Stability, trust, and openness are paramount in both the product and the community around Kubernetes. We undertook this retrospective effort to learn from the 1.3 release's shipping delay. These action items and other work in the upcoming releases are part of our commitment to continually improve our project, our community, and our ability to deliver production-grade infrastructure platform software. diff --git a/sig-storage/OWNERS b/sig-storage/OWNERS index d236db0e..fb58418f 100644 --- a/sig-storage/OWNERS +++ b/sig-storage/OWNERS @@ -1,6 +1,6 @@ reviewers: - - saad-ali - - childsb + - sig-storage-leads approvers: - - saad-ali - - childsb + - sig-storage-leads +labels: + - sig/storage diff --git a/sig-storage/README.md b/sig-storage/README.md index fac2c159..6d865b87 100644 --- a/sig-storage/README.md +++ b/sig-storage/README.md @@ -4,25 +4,41 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Storage SIG Covers storage and volume plugins. ## Meetings -* [Thursdays at 16:00 UTC](https://zoom.us/j/614261834) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=16:00&tz=UTC). +* [Thursdays at 9:00 PT (Pacific Time)](https://zoom.us/j/614261834) (biweekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=9:00&tz=PT%20%28Pacific%20Time%29). Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1-8KEG8AjAgKznS9NFm3qWqkGyCHmvU6HVl0sk5hwoAE/edit?usp=sharing). Meeting recordings can be found [here](https://www.youtube.com/watch?v=Eh7Qa7KOL8o&list=PL69nYSiGNLP02-BMqJdfFgGxYQ4Nb-2Qq). ## Leads -* [Saad Ali](https://github.com/saad-ali), Google -* [Bradley Childs](https://github.com/childsb), Red Hat +* Saad Ali (**[@saad-ali](https://github.com/saad-ali)**), Google +* Bradley Childs (**[@childsb](https://github.com/childsb)**), Red Hat ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-storage) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-storage) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fstorage) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-storage-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-storage-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-storage-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-storage-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-storage-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-storage-feature-requests) | Feature Requests | +| @kubernetes/sig-storage-misc | [link](https://github.com/orgs/kubernetes/teams/sig-storage-misc) | General Discussion | +| @kubernetes/sig-storage-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-storage-pr-reviews) | PR Reviews | +| @kubernetes/sig-storage-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-storage-proposals) | Design Proposals | +| @kubernetes/sig-storage-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-storage-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> diff --git a/sig-storage/contributing.md b/sig-storage/contributing.md index a6b5ea09..cbea6325 100644 --- a/sig-storage/contributing.md +++ b/sig-storage/contributing.md @@ -36,9 +36,9 @@ A great way to get involved is to pick an issue and help address it. We would lo ### Adding support for a new storage platform in Kubernetes For folks looking to add support for a new storage platform in Kubernetes, you have several options: - Write an in-tree volume plugin or provisioner: You can contribute a new in-tree volume plugin or provisioner, that gets built and ships with Kubernetes, for use within the Persistent Volume Framework. -[See the Ceph RBD volume plugin example](https://github.com/kubernetes/kubernetes/tree/master/pkg/volume/rbd) or [the AWS Provisioner example](https://github.com/kubernetes/kubernetes/pull/29006) +[See the Ceph RBD volume plugin example](https://git.k8s.io/kubernetes/pkg/volume/rbd) or [the AWS Provisioner example](https://github.com/kubernetes/kubernetes/pull/29006) - Write a FlexVolume plugin: This is an out-of-tree volume plugin which you develop and build separately outside of Kubernetes. -You then install the plugin on every Kubernetes host within your cluster and then [configure the plugin in Kubernetes as a FlexVolume](https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/flexvolume) +You then install the plugin on every Kubernetes host within your cluster and then [configure the plugin in Kubernetes as a FlexVolume](https://git.k8s.io/kubernetes/examples/volumes/flexvolume) - Write a Provisioner Controller: You can write a separate controller that watches for pending claims with a specific selector label on them. Once an appropriate claim is discovered, the controller then provisions the appropriate storage intended for the claim and creates a corresponding persistent volume for the claim that includes the same label used in the original claim selector. This will ensure that the PV for the new diff --git a/sig-testing/OWNERS b/sig-testing/OWNERS index b32e7cec..48c9f03c 100644 --- a/sig-testing/OWNERS +++ b/sig-testing/OWNERS @@ -1,8 +1,6 @@ reviewers: - - spiffxp - - fejta - - timothysc + - sig-testing-leads approvers: - - spiffxp - - fejta - - timothysc + - sig-testing-leads +labels: + - sig/testing diff --git a/sig-testing/README.md b/sig-testing/README.md index 552dd161..5af573d0 100644 --- a/sig-testing/README.md +++ b/sig-testing/README.md @@ -4,26 +4,43 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Testing SIG Interested in how we can most effectively test Kubernetes. We're interested specifically in making it easier for the community to run tests and contribute test results, to ensure Kubernetes is stable across a variety of cluster configurations and cloud providers. ## Meetings -* [Tuesdays at 20:00 UTC](https://zoom.us/j/2419653117) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=20:00&tz=UTC). +* [Tuesdays at 13:00 PT (Pacific Time)](https://zoom.us/my/k8s.sig.testing) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=13:00&tz=PT%20%28Pacific%20Time%29). -Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1z8MQpr_jTwhmjLMUaqQyBk1EYG_Y_3D4y4YdMJ7V1Kk). -Meeting recordings can be found [here](https://www.youtube.com/watch?v=BbFjuxe3N4w&list=PL69nYSiGNLP0ofY51bEooJ4TKuQtUSizR). +Meeting notes and Agenda can be found [here](https://bit.ly/k8s-sig-testing-notes). +Meeting recordings can be found [here](https://bit.ly/k8s-sig-testing-videos). ## Leads -* [Aaron Crickenberger](https://github.com/spiffxp), Samsung SDS -* [Erick Feja](https://github.com/fejta), Google -* [Timothy St. Clair](https://github.com/timothysc), Heptio +* Aaron Crickenberger (**[@spiffxp](https://github.com/spiffxp)**), Samsung SDS +* Erick Feja (**[@fejta](https://github.com/fejta)**), Google +* Steve Kuznetsov (**[@stevekuznetsov](https://github.com/stevekuznetsov)**), Red Hat +* Timothy St. Clair (**[@timothysc](https://github.com/timothysc)**), Heptio ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-testing) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-testing) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Ftesting) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-testing-api-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-testing-api-reviews) | API Changes and Reviews | +| @kubernetes/sig-testing-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-testing-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-testing-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-testing-feature-requests) | Feature Requests | +| @kubernetes/sig-testing-misc | [link](https://github.com/orgs/kubernetes/teams/sig-testing-misc) | General Discussion | +| @kubernetes/sig-testing-pr-reviews | [link](https://github.com/orgs/kubernetes/teams/sig-testing-pr-reviews) | PR Reviews | +| @kubernetes/sig-testing-proposals | [link](https://github.com/orgs/kubernetes/teams/sig-testing-proposals) | Design Proposals | +| @kubernetes/sig-testing-test-failures | [link](https://github.com/orgs/kubernetes/teams/sig-testing-test-failures) | Test Failures and Triage | <!-- BEGIN CUSTOM CONTENT --> diff --git a/sig-ui/OWNERS b/sig-ui/OWNERS index bbcbafa8..cecaeb3d 100644 --- a/sig-ui/OWNERS +++ b/sig-ui/OWNERS @@ -1,6 +1,6 @@ reviewers: - - danielromlein - - bryk + - sig-ui-leads approvers: - - danielromlein - - bryk + - sig-ui-leads +labels: + - sig/ui diff --git a/sig-ui/README.md b/sig-ui/README.md index ec46ec5e..3488f842 100644 --- a/sig-ui/README.md +++ b/sig-ui/README.md @@ -4,25 +4,26 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # UI SIG Covers all things UI related. Efforts are centered around Kubernetes Dashboard: a general purpose, web-based UI for Kubernetes clusters. It allows users to manage applications running in the cluster and troubleshoot them, as well as manage the cluster itself. ## Meetings -* [Thursdays at 16:00 UTC](https://groups.google.com/forum/#!forum/kubernetes-sig-ui) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=16:00&tz=UTC). +* [Thursdays at 18:00 CET (Central European Time)](https://groups.google.com/forum/#!forum/kubernetes-sig-ui) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=18:00&tz=CET%20%28Central%20European%20Time%29). Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1PwHFvqiShLIq8ZpoXvE3dSUnOv1ts5BTtZ7aATuKd-E/edit?usp=sharing). -Meeting recordings can be found [here](). + ## Leads -* [Dan Romlein](https://github.com/danielromlein), Apprenda -* [Sebastian Florek](https://github.com/floreks), Fujitsu +* Dan Romlein (**[@danielromlein](https://github.com/danielromlein)**), Google +* Sebastian Florek (**[@floreks](https://github.com/floreks)**), Fujitsu ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-ui) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-ui) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fui) <!-- BEGIN CUSTOM CONTENT --> diff --git a/sig-windows/OWNERS b/sig-windows/OWNERS index 9f67d29e..78470a91 100644 --- a/sig-windows/OWNERS +++ b/sig-windows/OWNERS @@ -1,4 +1,6 @@ reviewers: - - michmike + - sig-windows-leads approvers: - - michmike + - sig-windows-leads +labels: + - sig/windows diff --git a/sig-windows/README.md b/sig-windows/README.md index bc0ed7e4..974fff05 100644 --- a/sig-windows/README.md +++ b/sig-windows/README.md @@ -4,25 +4,55 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Windows SIG Focuses on supporting Windows Server Containers for Kubernetes. ## Meetings -* [Tuesdays at 16:30 UTC](https://zoom.us/my/sigwindows) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=16:30&tz=UTC). +* [Tuesdays at 12:30 Eastern Standard Time (EST)](https://zoom.us/my/sigwindows) (bi-weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=12:30&tz=Eastern%20Standard%20Time%20%28EST%29). Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1Tjxzjjuy4SQsFSUVXZbvqVb64hjNAG5CQX8bK7Yda9w/edit#heading=h.kbz22d1yc431). Meeting recordings can be found [here](https://www.youtube.com/watch?v=7zawb3KT9Xk&list=PL69nYSiGNLP2OH9InCcNkWNu2bl-gmIU4). ## Leads -* [Michael Michael](https://github.com/michmike), Apprenda +* Michael Michael (**[@michmike](https://github.com/michmike)**), Apprenda ## Contact * [Slack](https://kubernetes.slack.com/messages/sig-windows) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-windows) +* [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fwindows) + +## GitHub Teams + +The below teams can be mentioned on issues and PRs in order to get attention from the right people. +Note that the links to display team membership will only work if you are a member of the org. + +| Team Name | Details | Description | +| --------- |:-------:| ----------- | +| @kubernetes/sig-windows-bugs | [link](https://github.com/orgs/kubernetes/teams/sig-windows-bugs) | Bug Triage and Troubleshooting | +| @kubernetes/sig-windows-feature-requests | [link](https://github.com/orgs/kubernetes/teams/sig-windows-feature-requests) | Feature Requests | +| @kubernetes/sig-windows-misc | [link](https://github.com/orgs/kubernetes/teams/sig-windows-misc) | General Discussion | <!-- BEGIN CUSTOM CONTENT --> +## Getting Started + +If you're looking for information on building and running containers on Windows, you can get started at http://aka.ms/windowscontainers . + +Kubernetes 1.9 includes beta support for Windows Server containers and has an up-to-date [Getting Started Guide for Windows](https://kubernetes.io/docs/getting-started-guides/windows/) . + +Some additional guides are available, but may refer to older releases: + +* Using host gateway networking: https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows +* Using OVN as demonstrated at Google Cloud Next '17: https://github.com/apprenda/kubernetes-ovn-heterogeneous-cluster + +## Contributing + +If you'd like to fix open bugs, comment on proposals or just track the project, please check [Open Code Issues and PRs for SIG/Windows](https://github.com/kubernetes/kubernetes/labels/sig%2Fwindows) on GitHub. + +## Meeting Recordings + +Past meetings are in the [Recorded Meetings Playlist on Youtube](https://www.youtube.com/playlist?list=PL69nYSiGNLP2OH9InCcNkWNu2bl-gmIU4&jct=LZ9EIvD4DGrhr2h4r0ItaBmco7gTgw) <!-- END CUSTOM CONTENT --> @@ -6,6 +6,7 @@ sigs: API CRUD semantics, admission control, encoding/decoding, conversion, defaulting, persistence layer (etcd), OpenAPI, third-party resource, garbage collection, and client libraries. + label: api-machinery leads: - name: Daniel Smith company: Google @@ -15,7 +16,8 @@ sigs: github: deads2k meetings: - day: Wednesday - utc: "18:00" + time: "11:00" + tz: "PT (Pacific Time)" frequency: biweekly meeting_url: https://staging.talkgadget.google.com/hangouts/_/google.com/kubernetes-sig meeting_archive_url: https://goo.gl/x5nWrF @@ -23,6 +25,21 @@ sigs: contact: slack: sig-api-machinery mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-api-machinery + teams: + - name: sig-api-machinery-api-reviews + description: API Changes and Reviews + - name: sig-api-machinery-bugs + description: Bug Triage and Troubleshooting + - name: sig-api-machinery-feature-requests + description: Feature Requests + - name: sig-api-machinery-misc + description: General Discussion + - name: sig-api-machinery-pr-reviews + description: PR Reviews + - name: sig-api-machinery-proposals + description: Design Proposals + - name: sig-api-machinery-test-failures + description: Test Failures and Triage - name: Apps dir: sig-apps mission_statement: > @@ -31,6 +48,7 @@ sigs: We discuss how to define and run apps in Kubernetes, demo relevant tools and projects, and discuss areas of friction that can lead to suggesting improvements or feature requests. + label: apps leads: - name: Michelle Noorali github: michelleN @@ -41,21 +59,41 @@ sigs: - name: Adnan Abdulhussein github: prydonius company: Bitnami + - name: Kenneth Owens + github: kow3ns + company: Google meetings: - day: Monday - utc: "16:00" + time: "9:00" + tz: "PT (Pacific Time)" frequency: weekly - meeting_url: https://zoom.us/j/4526666954 + meeting_url: https://zoom.us/my/sig.apps meeting_archive_url: https://docs.google.com/document/d/1LZLBGW2wRDwAfdBNHJjFfk9CFoyZPcIYGWU7R1PQ3ng/edit# meeting_recordings_url: https://www.youtube.com/watch?v=hn23Z-vL_cM&list=PL69nYSiGNLP2LMq7vznITnpd2Fk1YIZF3 contact: slack: sig-apps mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-apps + teams: + - name: sig-apps-api-reviews + description: API Changes and Reviews + - name: sig-apps-bugs + description: Bug Triage and Troubleshooting + - name: sig-apps-feature-requests + description: Feature Requests + - name: sig-apps-misc + description: General Discussion + - name: sig-apps-pr-reviews + description: PR Reviews + - name: sig-apps-proposals + description: Design Proposals + - name: sig-apps-test-failures + description: Test Failures and Triage - name: Architecture dir: sig-architecture mission_statement: > The Architecture SIG maintains and evolves the design principles of Kubernetes, and provides a consistent body of expertise necessary to ensure architectural consistency over time. + label: architecture leads: - name: Brian Grant github: bgrant0607 @@ -64,20 +102,37 @@ sigs: github: jdumars company: Microsoft meetings: - - day: Monday - utc: "17:00" - frequency: biweekly + - day: Thursday + time: "15:30" + tz: "UTC" + frequency: weekly meeting_url: https://zoom.us/j/2018742972 meeting_archive_url: https://docs.google.com/document/d/1BlmHq5uPyBUDlppYqAAzslVbAO8hilgjqZUTaNXUhKM/edit meeting_recordings_url: https://www.youtube.com/watch?v=d5ERqm3oHN0&list=PL69nYSiGNLP2m6198LaLN6YahX7EEac5g contact: slack: sig-architecture mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-architecture + teams: + - name: sig-architecture-api-reviews + description: API Changes and Reviews + - name: sig-architecture-bugs + description: Bug Triage and Troubleshooting + - name: sig-architecture-feature-requests + description: Feature Requests + - name: sig-architecture-misc-use-only-as-a-last-resort + description: General Discussion + - name: sig-architecture-pr-reviews + description: PR Reviews + - name: sig-architecture-proposals + description: Design Proposals + - name: sig-architecture-test-failures + description: Test Failures and Triage - name: Auth dir: sig-auth mission_statement: > Covers improvements to Kubernetes authorization, authentication, and cluster security policy. + label: auth leads: - name: Eric Chiang github: ericchiang @@ -90,7 +145,8 @@ sigs: company: Red Hat meetings: - day: Wednesday - utc: "18:00" + time: "18:00" + tz: "UTC" frequency: biweekly meeting_url: https://zoom.us/my/k8s.sig.auth meeting_archive_url: https://docs.google.com/document/d/1woLGRoONE3EBVx-wTb4pvp4CI7tmLZ6lS26VTbosLKM/edit# @@ -98,12 +154,28 @@ sigs: contact: slack: sig-auth mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-auth + teams: + - name: sig-auth-api-reviews + description: API Changes and Reviews + - name: sig-auth-bugs + description: Bug Triage and Troubleshooting + - name: sig-auth-feature-requests + description: Feature Requests + - name: sig-auth-misc + description: General Discussion + - name: sig-auth-pr-reviews + description: PR Reviews + - name: sig-auth-proposals + description: Design Proposals + - name: sig-auth-test-failures + description: Test Failures and Triage - name: Autoscaling dir: sig-autoscaling mission_statement: > Covers autoscaling of clusters, horizontal and vertical autoscaling of pods, setting initial resources for pods, topics related to monitoring pods and gathering their metrics (e.g.: Heapster) + label: autoscaling leads: - name: Marcin Wielgus github: mwielgus @@ -112,19 +184,36 @@ sigs: github: directxman12 company: Red Hat meetings: - - day: Thursday - utc: "15:30" + - day: Monday + time: "14:00" + tz: "UTC" frequency: biweekly/triweekly - meeting_url: https://zoom.us/j/176352399 + meeting_url: https://zoom.us/my/k8s.sig.autoscaling meeting_archive_url: https://docs.google.com/document/d/1RvhQAEIrVLHbyNnuaT99-6u9ZUMp7BfkPupT2LAZK7w/edit meeting_recordings_url: contact: slack: sig-autoscaling mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-autoscaling + teams: + - name: sig-autoscaling-api-reviews + description: API Changes and Reviews + - name: sig-autoscaling-bugs + description: Bug Triage and Troubleshooting + - name: sig-autoscaling-feature-requests + description: Feature Requests + - name: sig-autoscaling-misc + description: General Discussion + - name: sig-autoscaling-pr-reviews + description: PR Reviews + - name: sig-autoscaling-proposals + description: Design Proposals + - name: sig-autoscaling-test-failures + description: Test Failures and Triage - name: AWS dir: sig-aws mission_statement: > Covers maintaining, supporting, and using Kubernetes hosted on AWS Cloud. + label: aws leads: - name: Justin Santa Barbara github: justinsb @@ -138,7 +227,8 @@ sigs: company: Redspread meetings: - day: Friday - utc: "17:00" + time: "9:00" # Actually 9AM Pacific, but timezone support issue is https://github.com/kubernetes/community/issues/841 + tz: "PT (Pacific Time)" frequency: biweekly meeting_url: https://zoom.us/my/k8ssigaws meeting_archive_url: https://docs.google.com/document/d/1-i0xQidlXnFEP9fXHWkBxqySkXwJnrGJP9OGyP2_P14/edit @@ -146,11 +236,15 @@ sigs: contact: slack: sig-aws mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-aws + teams: + - name: sig-aws-misc + description: General Discussion - name: Azure dir: sig-azure mission_statement: > A Special Interest Group for building, deploying, maintaining, supporting, and using Kubernetes on Azure Container Service. + label: azure leads: - name: Jason Hansen github: slack @@ -163,7 +257,8 @@ sigs: company: Microsoft meetings: - day: Wednesday - utc: "16:00" + time: "16:00" + tz: "UTC" frequency: biweekly meeting_url: https://deis.zoom.us/j/2018742972 meeting_archive_url: https://docs.google.com/document/d/1SpxvmOgHDhnA72Z0lbhBffrfe9inQxZkU9xqlafOW9k/edit @@ -171,26 +266,49 @@ sigs: contact: slack: sig-azure mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-azure + teams: + - name: sig-azure-misc + description: General Discussion - name: Big Data dir: sig-big-data mission_statement: > Covers deploying and operating big data applications (Spark, Kafka, Hadoop, Flink, Storm, etc) on Kubernetes. We focus on integrations with big data applications and architecting the best ways to run them on Kubernetes. + label: big-data leads: - name: Anirudh Ramanathan github: foxish company: Google + - name: Erik Erlandson + github: erikerlandson + company: Red Hat meetings: - day: Wednesday - utc: "17:00" + time: "17:00" + tz: "UTC" frequency: weekly meeting_url: https://zoom.us/my/sig.big.data meeting_archive_url: https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit - meeting_recordings_url: + meeting_recordings_url: https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit contact: slack: sig-big-data mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-big-data + teams: + - name: sig-big-data-api-reviews + description: API Changes and Reviews + - name: sig-big-data-bugs + description: Bug Triage and Troubleshooting + - name: sig-big-data-feature-requests + description: Feature Requests + - name: sig-big-data-misc + description: General Discussion + - name: sig-big-data-pr-reviews + description: PR Reviews + - name: sig-big-data-proposals + description: Design Proposals + - name: sig-big-data-test-failures + description: Test Failures and Triage - name: CLI dir: sig-cli mission_statement: > @@ -199,6 +317,7 @@ sigs: establishment of conventions for writing CLI commands, POSIX compliance, and improving the command line tools from a developer and devops user experience and usability perspective. + label: cli leads: - name: Fabiano Franz github: fabianofranz @@ -211,19 +330,38 @@ sigs: company: Alibaba meetings: - day: Wednesday - utc: "16:00" + time: "09:00" + tz: "PT (Pacific Time)" frequency: biweekly meeting_url: https://zoom.us/my/sigcli meeting_archive_url: https://docs.google.com/document/d/1r0YElcXt6G5mOWxwZiXgGu_X6he3F--wKwg-9UBc29I/edit?usp=sharing - meeting_recordings_url: https://www.youtube.com/watch?v=X29sffrQJU4&list=PL69nYSiGNLP28HaTzSlFe6RJVxpFmbUvF + meeting_recordings_url: https://www.youtube.com/playlist?list=PL69nYSiGNLP28HaTzSlFe6RJVxpFmbUvF contact: slack: sig-cli mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-cli + teams: + - name: sig-cli-api-reviews + description: API Changes and Reviews + - name: sig-cli-bugs + description: Bug Triage and Troubleshooting + - name: sig-cli-feature-requests + description: Feature Requests + - name: sig-cli-maintainers + description: CLI Maintainers + - name: sig-cli-misc + description: General Discussion + - name: sig-cli-pr-reviews + description: PR Reviews + - name: sig-cli-proposals + description: Design Proposals + - name: sig-cli-test-failures + description: Test Failures and Triage - name: Cluster Lifecycle dir: sig-cluster-lifecycle mission_statement: > The Cluster Lifecycle SIG is responsible for building the user experience for deploying and upgrading Kubernetes clusters. + label: cluster-lifecycle leads: - name: Luke Marsden github: lukemarsden @@ -234,17 +372,35 @@ sigs: - name: Robert Bailey github: roberthbailey company: Google + - name: Lucas Käldström + github: luxas + company: Luxas Labs (occasionally contracting for Weaveworks) meetings: - day: Tuesday - utc: "16:00" + time: "09:00" + tz: "PT (Pacific Time)" frequency: weekly meeting_url: https://zoom.us/j/166836%E2%80%8B624 meeting_archive_url: https://docs.google.com/a/weave.works/document/d/1deJYPIF4LmhGjDVaqrswErIrV7mtwJgovtLnPCDxP7U/edit meeting_recordings_url: https://www.youtube.com/watch?v=ljK5dgSA7vc&list=PL69nYSiGNLP29D0nYgAGWt1ZFqS9Z7lw4 contact: slack: sig-cluster-lifecycle - full_github_teams: true mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-lifecycle + teams: + - name: sig-cluster-lifecycle-api-reviews + description: API Changes and Reviews + - name: sig-cluster-lifecycle-bugs + description: Bug Triage and Troubleshooting + - name: sig-cluster-lifecycle-feature-requests + description: Feature Requests + - name: sig-cluster-lifecycle-misc + description: General Discussion + - name: sig-cluster-lifecycle-pr-reviews + description: PR Reviews + - name: sig-cluster-lifecycle-proposals + description: Design Proposals + - name: sig-cluster-lifecycle-test-failures + description: Test Failures and Triage - name: Cluster Ops dir: sig-cluster-ops mission_statement: > @@ -252,6 +408,7 @@ sigs: focus on shared operations practices for Kubernetes clusters with a goal to make Kubernetes broadly accessible with a common baseline reference. We also organize operators as a sounding board and advocacy group. + label: cluster-ops leads: - name: Rob Hirschfeld github: zehicle @@ -261,7 +418,8 @@ sigs: company: Microsoft meetings: - day: Thursday - utc: "20:00" + time: "20:00" + tz: "UTC" frequency: biweekly meeting_url: https://zoom.us/j/297937771 meeting_archive_url: https://docs.google.com/document/d/1IhN5v6MjcAUrvLd9dAWtKcGWBWSaRU8DNyPiof3gYMY/edit# @@ -277,6 +435,7 @@ sigs: contributors are happy and productive, we need to break the commit-rate ceiling we hit in July 2015, and we need to get the monotonically growing PR merge latency and numbers of open PRs and issues under control. + label: contributor-experience leads: - name: Garrett Rodrigues github: grodrigues3 @@ -286,7 +445,8 @@ sigs: company: CoreOS meetings: - day: Wednesday - utc: "16:30" + time: "9:30" + tz: "PT (Pacific Time)" frequency: biweekly meeting_url: https://zoom.us/j/7658488911 meeting_archive_url: https://docs.google.com/document/d/1qf-02B7EOrItQgwXFxgqZ5qjW0mtfu5qkYIF1Hl4ZLI/ @@ -294,10 +454,24 @@ sigs: contact: slack: sig-contribex mailing_list: https://groups.google.com/forum/#!forum/kubernetes-wg-contribex + teams: + - name: sig-contributor-experience-bugs + description: Bug Triage and Troubleshooting + - name: sig-contributor-experience-feature-requests + description: Feature Requests + - name: sig-contributor-experience-misc-use-only-as-a-last-resort + description: General Discussion + - name: sig-contributor-experience-pr-reviews + description: PR Reviews + - name: sig-contributor-experience-proposals + description: Design Proposals + - name: sig-contributor-experience-test-failures + description: Test Failures and Triage - name: Docs dir: sig-docs mission_statement: > Covers documentation, doc processes, and doc publishing for Kubernetes. + label: docs leads: - name: Devin Donnelly github: devin-donnelly @@ -307,45 +481,61 @@ sigs: company: Google meetings: - day: Tuesday - utc: "17:30" - frequency: biweekly + time: "17:30" + tz: "UTC" + frequency: weekly meeting_url: https://zoom.us/j/678394311 meeting_archive_url: https://docs.google.com/document/d/1Ds87eRiNZeXwRBEbFr6Z7ukjbTow5RQcNZLaSvWWQsE/edit - meeting_recordings_url: + meeting_recordings_url: https://www.youtube.com/playlist?list=PL69nYSiGNLP3b5hlx0YV7Lo7DtckM84y8 contact: slack: sig-docs mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-docs - - name: Federation - dir: sig-federation + teams: + - name: sig-docs-maintainers + description: Documentation Maintainers + - name: sig-docs-pr-reviews + description: Documentation PR Reviewers + - name: GCP + dir: sig-gcp mission_statement: > - Covers the Federation of Kubernetes Clusters and related - topics. This includes: application resiliency against availability zone - outages; hybrid clouds; spanning of multiple could providers; application - migration from private to public clouds (and vice versa); and other - similar subjects. + A Special Interest Group for building, deploying, maintaining, + supporting, and using Kubernetes on the Google Cloud Platform. leads: - - name: Christian Bell - github: csbell + - name: Adam Worrall + github: abgworrall company: Google - - name: Quinton Hoole - github: quinton-hoole - company: Huawei meetings: - - day: Tuesday - utc: "16:30" + - day: Thursday + time: "16:00" + tz: "UTC" frequency: biweekly - meeting_url: https://plus.google.com/hangouts/_/google.com/k8s-federation - meeting_archive_url: https://docs.google.com/document/d/18mk62nOXE_MCSSnb4yJD_8UadtzJrYyJxFwbrgabHe8/edit - meeting_recordings_url: https://www.youtube.com/watch?v=iWKC3FsNHWg&list=PL69nYSiGNLP0HqgyqTby6HlDEz7i1mb0- + meeting_url: https://zoom.us/j/761149873 + meeting_archive_url: https://docs.google.com/document/d/1mtmwZ4oVSSWhbEw8Lfzvc7ig84qxUpdK6uHyJp8rSGU/edit contact: - slack: sig-federation - mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-federation + slack: sig-gcp + mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-gcp + teams: + - name: sig-gcp-api-reviews + description: API Changes and Reviews + - name: sig-gcp-bugs + description: Bug Triage and Troubleshooting + - name: sig-gcp-feature-requests + description: Feature Requests + - name: sig-gcp-misc + description: General Discussion + - name: sig-gcp-pr-reviews + description: PR Reviews + - name: sig-gcp-proposals + description: Design Proposals + - name: sig-gcp-test-failures + description: Test Failures and Triage - name: Instrumentation dir: sig-instrumentation mission_statement: > Covers best practices for cluster observability through metrics, logging, and events across all Kubernetes components and development of relevant components such as Heapster and kube-state-metrics. Coordinates metric requirements of different SIGs for other components through finding common APIs. + label: instrumentation leads: - name: Piotr Szczesniak github: piosz @@ -355,7 +545,8 @@ sigs: company: CoreOS meetings: - day: Thursday - utc: "16:30" + time: "16:30" + tz: "UTC" frequency: weekly meeting_url: https://zoom.us/j/5342565819 meeting_archive_url: https://docs.google.com/document/d/1gWuAATtlmI7XJILXd31nA4kMq6U9u63L70382Y3xcbM/edit @@ -363,10 +554,69 @@ sigs: contact: slack: sig-instrumentation mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-instrumentation + teams: + - name: sig-instrumentation-api-reviews + description: API Changes and Reviews + - name: sig-instrumentation-bugs + description: Bug Triage and Troubleshooting + - name: sig-instrumentation-feature-requests + description: Feature Requests + - name: sig-instrumentation-misc + description: General Discussion + - name: sig-instrumentation-pr-reviews + description: PR Reviews + - name: sig-instrumentation-proposals + description: Design Proposals + - name: sig-instrumentation-test-failures + description: Test Failures and Triage + - name: Multicluster + dir: sig-multicluster + mission_statement: > + Covers multi-cluster Kubernetes use cases and tooling. This includes: + application resiliency against availability zone outages; hybrid clouds; + spanning of multiple could providers; application migration from private + to public clouds (and vice versa); and other similar subjects. This SIG + was formerly called sig-federation and focused on the Federation project, + but expanded its charter to all multi-cluster concerns in August 2017. + label: multicluster + leads: + - name: Christian Bell + github: csbell + company: Google + - name: Quinton Hoole + github: quinton-hoole + company: Huawei + meetings: + - day: Tuesday + time: "9:30" + tz: "PT (Pacific Time)" + frequency: biweekly + meeting_url: https://zoom.us/my/k8s.mc + meeting_archive_url: https://docs.google.com/document/d/18mk62nOXE_MCSSnb4yJD_8UadtzJrYyJxFwbrgabHe8/edit + meeting_recordings_url: https://www.youtube.com/watch?v=iWKC3FsNHWg&list=PL69nYSiGNLP0HqgyqTby6HlDEz7i1mb0- + contact: + slack: sig-multicluster + mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-multicluster + teams: + - name: sig-multicluster-api-reviews + description: API Changes and Reviews + - name: sig-multicluster-bugs + description: Bug Triage and Troubleshooting + - name: sig-multicluster-feature-requests + description: Feature Requests + - name: sig-multicluster-misc + description: General Discussion + - name: sig-multicluster-pr-reviews + description: PR Reviews + - name: sig-multicluster-test-failures + description: Test Failures and Triage + - name: sig-mutlicluster-proposals + description: Design Proposals - name: Network dir: sig-network mission_statement: > Covers networking in Kubernetes. + label: network leads: - name: Tim Hockin github: thockin @@ -379,7 +629,8 @@ sigs: company: Tigera meetings: - day: Thursday - utc: "21:00" + time: "14:00" + tz: "PT (Pacific Time)" frequency: biweekly meeting_url: https://zoom.us/j/5806599998 meeting_archive_url: https://docs.google.com/document/d/1_w77-zG_Xj0zYvEMfQZTQ-wPP4kXkpGD8smVtW_qqWM/edit @@ -387,9 +638,25 @@ sigs: contact: slack: sig-network mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-network + teams: + - name: sig-network-api-reviews + description: API Changes and Reviews + - name: sig-network-bugs + description: Bug Triage and Troubleshooting + - name: sig-network-feature-requests + description: Feature Requests + - name: sig-network-misc + description: General Discussion + - name: sig-network-pr-reviews + description: PR Reviews + - name: sig-network-proposals + description: Design Proposals + - name: sig-network-test-failures + description: Test Failures and Triage - name: Node dir: sig-node mission_statement: > + label: node leads: - name: Dawn Chen github: dchen1107 @@ -399,7 +666,8 @@ sigs: company: Red Hat meetings: - day: Tuesday - utc: "17:00" + time: "10:00" + tz: "PT (Pacific Time)" frequency: weekly meeting_url: https://plus.google.com/hangouts/_/google.com/sig-node-meetup?authuser=0 meeting_archive_url: https://docs.google.com/document/d/1Ne57gvidMEWXR70OxxnRkYquAoMpt56o75oZtg-OeBg/edit?usp=sharing @@ -407,16 +675,27 @@ sigs: contact: slack: sig-node mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-node + teams: + - name: sig-node-api-reviews + description: API Changes and Reviews + - name: sig-node-bugs + description: Bug Triage and Troubleshooting + - name: sig-node-feature-requests + description: Feature Requests + - name: sig-node-pr-reviews + description: PR Reviews + - name: sig-node-proposals + description: Design Proposals + - name: sig-node-test-failures + description: Test Failures and Triage - name: On Premise dir: sig-on-premise mission_statement: > Brings together member of Kubernetes community interested in running Kubernetes on premise, on bare metal or more generally beyond cloud providers. + label: onprem leads: - - name: Tomasz Napierala - github: zen - company: Mirantis - name: Marco Ceppi github: marcoceppi company: Canonical @@ -425,7 +704,8 @@ sigs: company: CoreOS meetings: - day: Wednesday - utc: "16:00" + time: "16:00" + tz: "UTC" frequency: weekly meeting_url: https://zoom.us/my/k8s.sig.onprem meeting_archive_url: https://docs.google.com/document/d/1AHF1a8ni7iMOpUgDMcPKrLQCML5EMZUAwP4rro3P6sk/edit# @@ -433,8 +713,21 @@ sigs: contact: slack: sig-onprem mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-on-prem - full_github_teams: true - github_team_prefix: onprem + teams: + - name: sig-onprem-api-reviews + description: API Changes and Reviews + - name: sig-onprem-bugs + description: Bug Triage and Troubleshooting + - name: sig-onprem-feature-requests + description: Feature Requests + - name: sig-onprem-misc + description: General Discussion + - name: sig-onprem-pr-reviews + description: PR Reviews + - name: sig-onprem-proposals + description: Design Proposals + - name: sig-onprem-test-failures + description: Test Failures and Triage - name: OpenStack dir: sig-openstack mission_statement: > @@ -443,16 +736,18 @@ sigs: projects with OpenStack as: a deployment platform for Kubernetes; a service provider for Kubernetes; a collection of applications to run on Kubernetes. + label: openstack leads: - name: Ihor Dvoretskyi github: idvoretskyi - company: Mirantis + company: CNCF - name: Steve Gordon github: xsgordon company: Red Hat meetings: - - day: Wednesday - utc: "15:00" + - day: Thursday + time: "00:00" + tz: "UTC" frequency: biweekly meeting_url: https://zoom.us/j/417251241 meeting_archive_url: https://docs.google.com/document/d/1iAQ3LSF_Ky6uZdFtEZPD_8i6HXeFxIeW4XtGcUJtPyU/edit?usp=sharing_eixpa_nl&ts=588b986f @@ -460,7 +755,21 @@ sigs: contact: slack: sig-openstack mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-openstack - full_github_teams: true + teams: + - name: sig-openstack-api-reviews + description: API Changes and Reviews + - name: sig-openstack-bugs + description: Bug Triage and Troubleshooting + - name: sig-openstack-feature-requests + description: Feature Requests + - name: sig-openstack-misc + description: General Discussion + - name: sig-openstack-pr-reviews + description: PR Reviews + - name: sig-openstack-proposals + description: Design Proposals + - name: sig-openstack-test-failures + description: Test Failures and Triage - name: Product Management dir: sig-product-management mission_statement: > @@ -482,19 +791,21 @@ sigs: source project is very new and largely unscoped for a project as large as Kubernetes; we are learning too and we are excited to learn how we can best serve the community of users and contributors. + label: none leads: - name: Aparna Sinha github: apsinha company: Google - name: Ihor Dvoretskyi github: idvoretskyi - company: Mirantis + company: CNCF - name: Caleb Miles github: calebamiles company: CoreOS meetings: - day: Tuesday - utc: "15:00" + time: "15:00" + tz: "UTC" frequency: biweekly meeting_url: https://zoom.us/j/845373595 meeting_archive_url: https://docs.google.com/document/d/1YqIpyjz4mV1jjvzhLx9JYy8LAduedzaoBMjpUKGUJQo/edit?usp=sharing @@ -504,6 +815,7 @@ sigs: mailing_list: https://groups.google.com/forum/#!forum/kubernetes-pm - name: Release dir: sig-release + label: release leads: - name: Phillip Wittrock github: pwittrock @@ -513,7 +825,8 @@ sigs: company: CoreOS meetings: - day: Tuesday - utc: "21:00" + time: "21:00" + tz: "UTC" frequency: biweekly meeting_url: https://zoom.us/j/664772523 meeting_archive_url: https://docs.google.com/document/d/1vhsixdT58iJFfoGZbpmvI_xnK59XyAjtadu3h6hHPpY/edit?usp=sharing @@ -521,6 +834,25 @@ sigs: contact: slack: sig-release mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-release + teams: + - name: sig-release-admins + description: Release Team Admins + - name: sig-release-api-reviews + description: API Changes and Reviews + - name: sig-release-bugs + description: Bug Triage and Troubleshooting + - name: sig-release-feature-requests + description: Feature Requests + - name: sig-release-members + description: Release Team Members + - name: sig-release-misc + description: General Discussion + - name: sig-release-pr-reviews + description: PR Reviews + - name: sig-release-proposals + description: Design Proposals + - name: sig-release-test-failures + description: Test Failures and Triage - name: Scalability dir: sig-scalability mission_statement: > @@ -532,7 +864,8 @@ sigs: control system reasonably consume? For more details about our objectives please review our - [Scaling And Performance Goals](https://github.com/kubernetes/community/blob/master/sig-scalability/goals.md) + [Scaling And Performance Goals](https://git.k8s.io/community/sig-scalability/goals.md) + label: scalability leads: - name: Wojciech Tyczynski github: wojtek-t @@ -540,22 +873,36 @@ sigs: - name: Bob Wise github: countspongebob company: Samsung SDS - - name: Joe Beda - github: jbeda - company: Heptio meetings: - day: Thursday - utc: "16:00" - frequency: weekly + time: "16:00" + tz: "UTC" + frequency: bi-weekly meeting_url: https://zoom.us/j/989573207 meeting_archive_url: https://docs.google.com/a/bobsplanet.com/document/d/1hEpf25qifVWztaeZPFmjNiJvPo-5JX1z0LSvvVY5G2g/edit?usp=drive_web meeting_recordings_url: https://www.youtube.com/watch?v=NDP1uYyom28&list=PL69nYSiGNLP2X-hzNTqyELU6jYS3p10uL contact: slack: sig-scalability mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-scale + teams: + - name: sig-scalability-api-reviews + description: API Changes and Reviews + - name: sig-scalability-bugs + description: Bug Triage and Troubleshooting + - name: sig-scalability-feature-requests + description: Feature Requests + - name: sig-scalability-misc + description: General Discussion + - name: sig-scalability-pr-reviews + description: PR Reviews + - name: sig-scalability-proprosals + description: Design Proposals + - name: sig-scalability-test-failures + description: Test Failures and Triage - name: Scheduling dir: sig-scheduling mission_statement: > + label: scheduling leads: - name: David Oppenheimer github: davidopp @@ -565,10 +912,12 @@ sigs: company: Red Hat meetings: - day: Monday - utc: "20:00" + time: "20:00" + tz: "UTC" frequency: biweekly - day: Wednesday - utc: "07:30" + time: "07:30" + tz: "UTC" frequency: biweekly meeting_url: https://zoom.us/zoomconference?m=rN2RrBUYxXgXY4EMiWWgQP6Vslgcsn86 meeting_archive_url: https://docs.google.com/document/d/13mwye7nvrmV11q9_Eg77z-1w3X7Q1GTbslpml4J7F3A/edit @@ -576,11 +925,27 @@ sigs: contact: slack: sig-scheduling mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-scheduling + teams: + - name: sig-scheduling-api-reviews + description: API Changes and Reviews + - name: sig-scheduling-bugs + description: Bug Triage and Troubleshooting + - name: sig-scheduling-feature-requests + description: Feature Requests + - name: sig-scheduling-misc + description: General Discussion + - name: sig-scheduling-pr-reviews + description: PR Reviews + - name: sig-scheduling-proposals + description: Design Proposals + - name: sig-scheduling-test-failures + description: Test Failures and Triage - name: Service Catalog dir: sig-service-catalog mission_statement: > To develop a Kubernetes API for the CNCF service broker and Kubernetes broker implementation. + label: service-catalog leads: - name: Paul Morie github: pmorie @@ -596,18 +961,35 @@ sigs: company: IBM meetings: - day: Monday - utc: "20:00" + time: "20:00" + tz: "UTC" frequency: weekly meeting_url: https://zoom.us/j/7201225346 - meeting_archive_url: http://goo.gl/A0m24V + meeting_archive_url: https://docs.google.com/document/d/17xlpkoEbPR5M6P5VDzNx17q6-IPFxKyebEekCGYiIKM/edit meeting_recordings_url: https://www.youtube.com/watch?v=ukPj1sFFkr0&list=PL69nYSiGNLP2k9ZXx9E1MvRSotFDoHUWs contact: slack: sig-service-catalog mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-service-catalog + teams: + - name: sig-service-catalog-api-reviews + description: API Changes and Reviews + - name: sig-service-catalog-bugs + description: Bug Triage and Troubleshooting + - name: sig-service-catalog-feature-requests + description: Feature Requests + - name: sig-service-catalog-misc + description: General Discussion + - name: sig-service-catalog-pr-reviews + description: PR Reviews + - name: sig-service-catalog-proposals + description: Design Proposals + - name: sig-service-catalog-test-failures + description: Test Failures and Triage - name: Storage dir: sig-storage mission_statement: > Covers storage and volume plugins. + label: storage leads: - name: Saad Ali github: saad-ali @@ -617,7 +999,8 @@ sigs: company: Red Hat meetings: - day: Thursday - utc: "16:00" + time: "9:00" + tz: "PT (Pacific Time)" frequency: biweekly meeting_url: https://zoom.us/j/614261834 meeting_archive_url: https://docs.google.com/document/d/1-8KEG8AjAgKznS9NFm3qWqkGyCHmvU6HVl0sk5hwoAE/edit?usp=sharing @@ -625,6 +1008,21 @@ sigs: contact: slack: sig-storage mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-storage + teams: + - name: sig-storage-api-reviews + description: API Changes and Reviews + - name: sig-storage-bugs + description: Bug Triage and Troubleshooting + - name: sig-storage-feature-requests + description: Feature Requests + - name: sig-storage-misc + description: General Discussion + - name: sig-storage-pr-reviews + description: PR Reviews + - name: sig-storage-proposals + description: Design Proposals + - name: sig-storage-test-failures + description: Test Failures and Triage - name: Testing dir: sig-testing mission_statement: > @@ -632,6 +1030,7 @@ sigs: interested specifically in making it easier for the community to run tests and contribute test results, to ensure Kubernetes is stable across a variety of cluster configurations and cloud providers. + label: testing leads: - name: Aaron Crickenberger github: spiffxp @@ -639,19 +1038,38 @@ sigs: - name: Erick Feja github: fejta company: Google + - name: Steve Kuznetsov + github: stevekuznetsov + company: Red Hat - name: Timothy St. Clair github: timothysc company: Heptio meetings: - day: Tuesday - utc: "20:00" + time: "13:00" + tz: "PT (Pacific Time)" frequency: weekly - meeting_url: https://zoom.us/j/2419653117 - meeting_archive_url: https://docs.google.com/document/d/1z8MQpr_jTwhmjLMUaqQyBk1EYG_Y_3D4y4YdMJ7V1Kk - meeting_recordings_url: https://www.youtube.com/watch?v=BbFjuxe3N4w&list=PL69nYSiGNLP0ofY51bEooJ4TKuQtUSizR + meeting_url: https://zoom.us/my/k8s.sig.testing + meeting_archive_url: https://bit.ly/k8s-sig-testing-notes + meeting_recordings_url: https://bit.ly/k8s-sig-testing-videos contact: slack: sig-testing mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-testing + teams: + - name: sig-testing-api-reviews + description: API Changes and Reviews + - name: sig-testing-bugs + description: Bug Triage and Troubleshooting + - name: sig-testing-feature-requests + description: Feature Requests + - name: sig-testing-misc + description: General Discussion + - name: sig-testing-pr-reviews + description: PR Reviews + - name: sig-testing-proposals + description: Design Proposals + - name: sig-testing-test-failures + description: Test Failures and Triage - name: UI dir: sig-ui mission_statement: > @@ -659,16 +1077,18 @@ sigs: Dashboard: a general purpose, web-based UI for Kubernetes clusters. It allows users to manage applications running in the cluster and troubleshoot them, as well as manage the cluster itself. + label: ui leads: - name: Dan Romlein github: danielromlein - company: Apprenda + company: Google - name: Sebastian Florek github: floreks company: Fujitsu meetings: - day: Thursday - utc: "16:00" + time: "18:00" + tz: "CET (Central European Time)" frequency: weekly meeting_url: https://groups.google.com/forum/#!forum/kubernetes-sig-ui meeting_archive_url: https://docs.google.com/document/d/1PwHFvqiShLIq8ZpoXvE3dSUnOv1ts5BTtZ7aATuKd-E/edit?usp=sharing @@ -680,20 +1100,29 @@ sigs: dir: sig-windows mission_statement: > Focuses on supporting Windows Server Containers for Kubernetes. + label: windows leads: - name: Michael Michael github: michmike company: Apprenda meetings: - day: Tuesday - utc: "16:30" - frequency: weekly + time: "12:30" + tz: "Eastern Standard Time (EST)" + frequency: bi-weekly meeting_url: https://zoom.us/my/sigwindows meeting_archive_url: https://docs.google.com/document/d/1Tjxzjjuy4SQsFSUVXZbvqVb64hjNAG5CQX8bK7Yda9w/edit#heading=h.kbz22d1yc431 meeting_recordings_url: https://www.youtube.com/watch?v=7zawb3KT9Xk&list=PL69nYSiGNLP2OH9InCcNkWNu2bl-gmIU4 contact: slack: sig-windows mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-windows + teams: + - name: sig-windows-bugs + description: Bug Triage and Troubleshooting + - name: sig-windows-feature-requests + description: Feature Requests + - name: sig-windows-misc + description: General Discussion workinggroups: - name: Resource Management dir: wg-resource-management @@ -707,8 +1136,9 @@ workinggroups: github: derekwaynecarr company: Red Hat meetings: - - day: Tuesday - utc: "18:00" + - day: Wednesday + time: "11:00" + tz: "PT (Pacific Time)" frequency: weekly (On demand) meeting_url: https://zoom.us/j/4799874685 meeting_archive_url: https://docs.google.com/document/d/1j3vrG6BgE0hUDs2e-1ZUegKN4W4Adb1B6oJ6j-4kyPU @@ -729,11 +1159,113 @@ workinggroups: company: Google meetings: - day: Tuesday - utc: "15:00" + time: "15:00" + tz: "UTC" frequency: bi-weekly (On demand) meeting_url: TBD - meeting_archive_url: https://docs.google.com/document/d/1bCK-1_Zy2WfsrMBJkdaV72d2hidaxZBhS5YQHAgscPI/edit + meeting_archive_url: https://docs.google.com/document/d/1uH60pNr1-jBn7N2pEcddk6-6NTnmV5qepwKUJe9tMRo/edit meeting_recordings_url: contact: slack: wg-container-identity mailing_list: https://groups.google.com/forum/#!forum/kubernetes-wg-container-identity + - name: Kubeadm Adoption + dir: wg-kubeadm-adoption + mission_statement: > + Boost adoption of the kubeadm tool. kubeadm is a tool for creating new Kubernetes clusters easily for new users, but can also be used as a toolbox for higher-level deployment + solutions. This working group makes sure kubeadm meets the extensibility requirements of those higher-level Kubernetes installers. + leads: + - name: Lucas Käldström + github: luxas + company: Luxas Labs (occasionally contracting for Weaveworks) + - name: Justin Santa Barbara + github: justinsb + meetings: + - day: Tuesday + time: "18:00" + tz: "UTC" + frequency: bi-weekly + meeting_url: https://zoom.us/j/166836%E2%80%8B624 + meeting_archive_url: https://docs.google.com/document/d/1KdXsLYiJYJdiRbtgZsx6qbHF4g_K-gAScB9Zs4avgzg/edit + meeting_recordings_url: https://www.youtube.com/watch?v=-Xlcrm5iT80&list=PLPgAK4Icr0ehh93BiMC3djAc5KoW7WIkl + contact: + slack: sig-cluster-lifecycle + mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-lifecycle + - name: Cluster API + dir: wg-cluster-api + mission_statement: > + Define a portable API that represents a Kubernetes cluster. The API will contain the control plane and its configuration and the underlying infrastructure (nodes, node pools, etc). + leads: + - name: Kris Nova + github: kris-nova + company: Microsoft + - name: Jacob Beacham + github: pipejakob + company: Google + - name: Robert Bailey + github: roberthbailey + company: Google + meeting_archive_url: https://docs.google.com/document/d/16ils69KImmE94RlmzjWDrkmFZysgB2J4lGnYMRN89WM/edit + contact: + slack: sig-cluster-lifecycle + mailing_list: https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-lifecycle + - name: App Def + dir: wg-app-def + mission_statement: > + Improve UX of declarative primitives in the API and/or primary client libraries and tools, + understand the needs of other parts of the ecosystem on this layer, and improve the interoperability + of application management tools through better separation of concerns, common conventions, + and common principles. + + Charter can be found [here](https://docs.google.com/document/d/1TzRwzWYRulx4o8Fii8k7ToIx4LR4MSncxxLdJ9TkOAs/edit#) + leads: + - name: Antoine Legrand + github: ant31 + company: CoreOS + - name: Sebastien Goasguen + github: sebgoa + company: Bitnami + meetings: + - day: Wednesday + time: "16:00" + tz: "UTC" + frequency: bi-weekly + meeting_url: https://zoom.us/j/748123863 + meeting_archive_url: https://docs.google.com/document/d/1Pxc-qwAt4FvuISZ_Ib5KdUwlynFkGueuzPx5Je_lbGM/edit + contact: + slack: wg-app-def + mailing_list: https://groups.google.com/forum/#!forum/kubernetes-wg-app-def + - name: Cloud Provider + dir: wg-cloud-provider + mission_statement: > + Charter can be found [here](https://docs.google.com/document/d/1m4Kvnh_u_9cENEE9n1ifYowQEFSgiHnbw43urGJMB64/edit#) + leads: + - name: Sidhartha Mani + github: wlan0 + company: Caascade Labs + - name: Jago Macleod + github: jagosan + company: Google + meetings: + - day: Wednesday + time: "10:00" + tz: "PT (Pacific Time)" + frequency: weekly + meeting_url: https://zoom.us/my/cloudprovider + meeting_archive_url: https://docs.google.com/document/d/1OZE-ub-v6B8y-GuaWejL-vU_f9jsjBbrim4LtTfxssw/edit#heading=h.w7i4ksrweimp + contact: + slack: wg-cloud-provider + mailing_list: https://groups.google.com/forum/#!forum/kubernetes-wg-cloud-provider + - name: Multitenancy + dir: wg-multitenancy + mission_statement: > + Define the models of multitenancy that Kubernetes will support. Discuss and execute upon any remaining work that needs to be done to support these models. Create conformance tests that will prove that these models can be built and used in production environments. + leads: + - name: David Oppenheimer + github: davidopp + company: Google + - name: Jessie Frazelle + github: jessfraz + company: Microsoft + contact: + slack: wg-multitenancy + mailing_list: https://groups.google.com/forum/#!forum/kubernetes-wg-multitenancy diff --git a/wg-app-def/OWNERS b/wg-app-def/OWNERS new file mode 100644 index 00000000..cfe36752 --- /dev/null +++ b/wg-app-def/OWNERS @@ -0,0 +1,4 @@ +reviewers: + - wg-app-def-leads +approvers: + - wg-app-def-leads diff --git a/wg-app-def/README.md b/wg-app-def/README.md new file mode 100644 index 00000000..dd40027b --- /dev/null +++ b/wg-app-def/README.md @@ -0,0 +1,30 @@ +<!--- +This is an autogenerated file! + +Please do not edit this file directly, but instead make changes to the +sigs.yaml file in the project root. + +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md +--> +# App Def Working Group + +Improve UX of declarative primitives in the API and/or primary client libraries and tools, understand the needs of other parts of the ecosystem on this layer, and improve the interoperability of application management tools through better separation of concerns, common conventions, and common principles. +Charter can be found [here](https://docs.google.com/document/d/1TzRwzWYRulx4o8Fii8k7ToIx4LR4MSncxxLdJ9TkOAs/edit#) + +## Meetings +* [Wednesdays at 16:00 UTC](https://zoom.us/j/748123863) (bi-weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=16:00&tz=UTC). + +Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1Pxc-qwAt4FvuISZ_Ib5KdUwlynFkGueuzPx5Je_lbGM/edit). + + +## Organizers +* Antoine Legrand (**[@ant31](https://github.com/ant31)**), CoreOS +* Sebastien Goasguen (**[@sebgoa](https://github.com/sebgoa)**), Bitnami + +## Contact +* [Slack](https://kubernetes.slack.com/messages/wg-app-def) +* [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-wg-app-def) + +<!-- BEGIN CUSTOM CONTENT --> + +<!-- END CUSTOM CONTENT --> diff --git a/wg-cloud-provider/README.md b/wg-cloud-provider/README.md new file mode 100644 index 00000000..81f1a20a --- /dev/null +++ b/wg-cloud-provider/README.md @@ -0,0 +1,29 @@ +<!--- +This is an autogenerated file! + +Please do not edit this file directly, but instead make changes to the +sigs.yaml file in the project root. + +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md +--> +# Cloud Provider Working Group + +Charter can be found [here](https://docs.google.com/document/d/1m4Kvnh_u_9cENEE9n1ifYowQEFSgiHnbw43urGJMB64/edit#) + +## Meetings +* [Wednesdays at 10:00 PT (Pacific Time)](https://zoom.us/my/cloudprovider) (weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=10:00&tz=PT%20%28Pacific%20Time%29). + +Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1OZE-ub-v6B8y-GuaWejL-vU_f9jsjBbrim4LtTfxssw/edit#heading=h.w7i4ksrweimp). + + +## Organizers +* Sidhartha Mani (**[@wlan0](https://github.com/wlan0)**), Caascade Labs +* Jago Macleod (**[@jagosan](https://github.com/jagosan)**), Google + +## Contact +* [Slack](https://kubernetes.slack.com/messages/wg-cloud-provider) +* [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-wg-cloud-provider) + +<!-- BEGIN CUSTOM CONTENT --> + +<!-- END CUSTOM CONTENT --> diff --git a/wg-cloud-provider/draft-20171128-controller-manager.md b/wg-cloud-provider/draft-20171128-controller-manager.md new file mode 100644 index 00000000..68bd078c --- /dev/null +++ b/wg-cloud-provider/draft-20171128-controller-manager.md @@ -0,0 +1,295 @@ +``` +kep-number: draft-20171127 +title: Cloud Provider Controller Manager +authors: + - "@cheftako" + - "@calebamiles" +owning-sig: sig-apimachinery +participating-sigs: + - sig-storage + - sig-apps + - sig-network +reviewers: + - "@wlan0" + - "@calebamiles" +approvers: + - TBD +editor: TBD +creation-date: 2017-11-27 +last-updated: 2017-11-27 +status: draft +see-also: +replaces: + - contributors/design-proposals/cloud-provider/cloud-provider-refactoring.md +``` + +# Remove Cloud Provider Code From Kubernetes Core + +## Table of Contents + + * [Remove Cloud Provider Code From Kubernetes Core](#remove-cloud-provider-code-from-kubernetes-core) + * [Table of Contents](#table-of-contents) + * [Summary](#summary) + * [Motivation](#motivation) + * [Goals](#goals) + * [Intermediary Goals](#intermediary-goals) + * [Non-Goals](#non-goals) + * [Proposal](#proposal) + * [Controller Manager Changes](#controller-manager-changes) + * [Kubelet Changes](#kubelet-changes) + * [API Server Changes](#api-server-changes) + * [Volume Managent Changes](#volume-managent-changes) + * [Deployment Changes](#deployment-changes) + * [Implementation Details/Notes/Constraints [optional]](#implementation-detailsnotesconstraints-optional) + * [Security Considerations](#security-considerations) + * [Graduation Criteria](#graduation-criteria) + * [Implementation History](#implementation-history) + * [Drawbacks [optional]](#drawbacks-optional) + * [Alternatives [optional]](#alternatives-optional) + +## Summary + +We want to remove any cloud provider specific logic from the kubernetes/kubernetes repo. We want to restructure the code +to make is easy for any cloud provider to extend the kubernetes core in a consistent manner for their cloud. New cloud +providers should look at the [Creating a Custom Cluster from Scratch](https://kubernetes.io/docs/getting-started-guides/scratch/#cloud-provider) +and the [cloud provider interface](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/cloud.go#L31) +which will need to be implemented. + +## Motivation + +We are trying to remove any dependencies from Kubernetes Core to any specific cloud provider. Currently we have seven +such dependencies. To prevent this number from growing we have locked Kubernetes Core to the addition of any new +dependencies. This means all new cloud providers have to implement all their pieces outside of the Core. +However everyone still ends up consuming the current set of seven in repo dependencies. For the seven in repo cloud +providers any changes to their specific cloud provider code requires OSS PR approvals and a deployment to get those +changes in to an official build. The relevant dependencies require changes in the following areas. + +- [Kube Controller Manager](https://kubernetes.io/docs/reference/generated/kube-controller-manager/) - Track usages of [CMServer.CloudProvider](https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-controller-manager/app/options/options.go) +- [API Server](https://kubernetes.io/docs/reference/generated/kube-apiserver/) - Track usages of [ServerRunOptions.CloudProvider](https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/options/options.go) +- [Kubelet](https://kubernetes.io/docs/reference/generated/kubelet/) - Track usages of [KubeletFlags.CloudProvider](https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/options/options.go) +- [How Cloud Provider Functionality is deployed to and enabled in the cluster](https://kubernetes.io/docs/setup/pick-right-solution/#hosted-solutions) - Track usage from [PROVIDER_UTILS](https://github.com/kubernetes/kubernetes/blob/master/cluster/kube-util.sh) + +For the cloud providers who are in repo, moving out would allow them to more quickly iterate on their solution and +decouple cloud provider fixes from open source releases. Moving the cloud provider code out of the open source +processes means that these processes do not need to load/run unnecessary code for the environment they are in. +We would like to abstract a core controller manager library so help standardize the behavior of the cloud +controller managers produced by each cloud provider. We would like to minimize the number and scope of controllers +running in the cloud controller manager so as to minimize the surface area for per cloud provider deviation. + +### Goals + +- Get to a point where we do not load the cloud interface for any of kubernetes core processes. +- Remove all cloud provider specific code from kubernetes/kubernetes. +- Have a generic controller manager library available for use by the per cloud provider controller managers. +- Move the cloud provider specific controller manager logic into repos appropriate for those cloud providers. + +### Intermediary Goals + +Have a cloud controller manager in the kubernetes main repo which hosts all of +the controller loops for the in repo cloud providers. +Do not run any cloud provider logic in the kube controller manager, the kube apiserver or the kubelet. +At intermediary points we may just move some of the cloud specific controllers out. (Eg. volumes may be later than the rest) + +### Non-Goals + +Forcing cloud providers to use the generic cloud manager. + +## Proposal + +### Controller Manager Changes + +For the controller manager we would like to create a set of common code which can be used by both the cloud controller +manager and the kube controller manager. The cloud controller manager would then be responsible for running controllers +whose function is specific to cloud provider functionality. The kube controller manager would then be responsible +for running all controllers whose function was not related to a cloud provider. + +In order to create a 100% cloud independent controller manager, the controller-manager will be split into multiple binaries. + +1. Cloud dependent controller-manager binaries +2. Cloud independent controller-manager binaries - This is the existing `kube-controller-manager` that is being shipped +with kubernetes releases. + +The cloud dependent binaries will run those loops that rely on cloudprovider in a separate process(es) within the kubernetes control plane. +The rest of the controllers will be run in the cloud independent controller manager. +The decision to run entire controller loops, rather than only the very minute parts that rely on cloud provider was made +because it makes the implementation simple. Otherwise, the shared data structures and utility functions have to be +disentangled, and carefully separated to avoid any concurrency issues. This approach among other things, prevents code +duplication and improves development velocity. + +Note that the controller loop implementation will continue to reside in the core repository. It takes in +cloudprovider.Interface as an input in its constructor. Vendor maintained cloud-controller-manager binary could link +these controllers in, as it serves as a reference form of the controller implementation. + +There are four controllers that rely on cloud provider specific code. These are node controller, service controller, +route controller and attach detach controller. Copies of each of these controllers have been bundled together into +one binary. The cloud dependent binary registers itself as a controller, and runs the cloud specific controller loops +with the user-agent named "external-controller-manager". + +RouteController and serviceController are entirely cloud specific. Therefore, it is really simple to move these two +controller loops out of the cloud-independent binary and into the cloud dependent binary. + +NodeController does a lot more than just talk to the cloud. It does the following operations - + +1. CIDR management +2. Monitor Node Status +3. Node Pod Eviction + +While Monitoring Node status, if the status reported by kubelet is either 'ConditionUnknown' or 'ConditionFalse', then +the controller checks if the node has been deleted from the cloud provider. If it has already been deleted from the +cloud provider, then it deletes the nodeobject without waiting for the `monitorGracePeriod` amount of time. This is the +only operation that needs to be moved into the cloud dependent controller manager. + +Finally, The attachDetachController is tricky, and it is not simple to disentangle it from the controller-manager +easily, therefore, this will be addressed with Flex Volumes (Discussed under a separate section below) + + +The kube-controller-manager has many controller loops. [See NewControllerInitializers](https://github.com/kubernetes/kubernetes/blob/release-1.9/cmd/kube-controller-manager/app/controllermanager.go#L332) + + - [nodeController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/node) + - [volumeController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/volume) + - [routeController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/route) + - [serviceController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/service) + - replicationController + - endpointController + - resourceQuotaController + - namespaceController + - deploymentController + - etc.. + +Among these controller loops, the following are cloud provider dependent. + + - [nodeController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/node) + - [volumeController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/volume) + - [routeController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/route) + - [serviceController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/service) + +The nodeController uses the cloudprovider to check if a node has been deleted from the cloud. If cloud provider reports +a node as deleted, then this controller immediately deletes the node from kubernetes. This check removes the need to +wait for a specific amount of time to conclude that an inactive node is actually dead. + +The volumeController uses the cloudprovider to create, delete, attach and detach volumes to nodes. For instance, the +logic for provisioning, attaching, and detaching a EBS volume resides in the AWS cloudprovider. The volumeController +uses this code to perform its operations. + +The routeController configures routes for hosts in the cloud provider. + +The serviceController maintains a list of currently active nodes, and is responsible for creating and deleting +LoadBalancers in the underlying cloud. + +### Kubelet Changes + +Moving on to the kubelet, the following cloud provider dependencies exist in kubelet. + + - Find the cloud nodename of the host that kubelet is running on for the following reasons : + 1. To obtain the config map for the kubelet, if one already exists + 2. To uniquely identify current node using nodeInformer + 3. To instantiate a reference to the current node object + - Find the InstanceID, ProviderID, ExternalID, Zone Info of the node object while initializing it + - Periodically poll the cloud provider to figure out if the node has any new IP addresses associated with it + - It sets a condition that makes the node unschedulable until cloud routes are configured. + - It allows the cloud provider to post process DNS settings + +The majority of the calls by the kubelet to the cloud is done during the initialization of the Node Object. The other +uses are for configuring Routes (in case of GCE), scrubbing DNS, and periodically polling for IP addresses. + +All of the above steps, except the Node initialization step can be moved into a controller. Specifically, IP address +polling, and configuration of Routes can be moved into the cloud dependent controller manager. + +[Scrubbing DNS was found to be redundant](https://github.com/kubernetes/kubernetes/pull/36785). So, it can be disregarded. It is being removed. + +Finally, Node initialization needs to be addressed. This is the trickiest part. Pods will be scheduled even on +uninitialized nodes. This can lead to scheduling pods on incompatible zones, and other weird errors. Therefore, an +approach is needed where kubelet can create a Node, but mark it as "NotReady". Then, some asynchronous process can +update it and mark it as ready. This is now possible because of the concept of Taints. + +This approach requires kubelet to be started with known taints. This will make the node unschedulable until these +taints are removed. The external cloud controller manager will asynchronously update the node objects and remove the +taints. + +### API Server Changes + +Finally, in the kube-apiserver, the cloud provider is used for transferring SSH keys to all of the nodes, and within an a +dmission controller for setting labels on persistent volumes. + +Kube-apiserver uses the cloud provider for two purposes + +1. Distribute SSH Keys - This can be moved to the cloud dependent controller manager +2. Admission Controller for PV - This can be refactored using the taints approach used in Kubelet + +### Volume Managent Changes + +Volumes need cloud providers, but they only need **specific** cloud providers. The majority of volume management logic +resides in the controller manager. These controller loops need to be moved into the cloud-controller manager. The cloud +controller manager also needs a mechanism to read parameters for initilization from cloud config. This can be done via +config maps. + +There are two entirely different approach to refactoring volumes - +[Flex Volumes](https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md) and +[CSI Container Storage Interface](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md). There is an undergoing effort to move all +of the volume logic from the controller-manager into plugins called Flex Volumes. In the Flex volumes world, all of the +vendor specific code will be packaged in a separate binary as a plugin. After discussing with @thockin, this was +decidedly the best approach to remove all cloud provider dependency for volumes out of kubernetes core. Some of the discovery +information for this can be found at [https://goo.gl/CtzpVm](https://goo.gl/CtzpVm). + +### Deployment Changes + +This change will introduce new binaries to the list of binaries required to run kubernetes. The change will be designed +such that these binaries can be installed via `kubectl apply -f` and the appropriate instances of the binaries will be +running. + +Issues such as monitoring, configuring the new binaries will generally be left to cloud provider. However they should +ensure that test runs upload the logs for these new processes to [test grid](https://k8s-testgrid.appspot.com/). + +Applying the cloud controller manager is the only step that is different in the upgrade process. +In order to complete the upgrade process, you need to apply the cloud-controller-manager deployment to the setup. +A deployment descriptor file will be provided with this change. You need to apply this change using + +``` +kubectl apply -f cloud-controller-manager.yml +``` + +This will start the cloud specific controller manager in your kubernetes setup. + +The downgrade steps are also the same as before for all the components except the cloud-controller-manager. +In case of the cloud-controller-manager, the deployment should be deleted using + +``` +kubectl delete -f cloud-controller-manager.yml +``` + +### Implementation Details/Notes/Constraints [optional] + +### Security Considerations + +Make sure that you consider the impact of this feature from the point of view of Security. + +## Graduation Criteria + +How will we know that this has succeeded? +Gathering user feedback is crucial for building high quality experiences and SIGs have the important responsibility of +setting milestones for stability and completeness. +Hopefully the content previously contained in [umbrella issues][] will be tracked in the `Graduation Criteria` section. + +[umbrella issues]: https://github.com/kubernetes/kubernetes/issues/42752 + +## Implementation History + +Major milestones in the life cycle of a KEP should be tracked in `Implementation History`. +Major milestones might include + +- the `Summary` and `Motivation` sections being merged signaling SIG acceptance +- the `Proposal` section being merged signaling agreement on a proposed design +- the date implementation started +- the first Kubernetes release where an initial version of the KEP was available +- the version of Kubernetes where the KEP graduated to general availability +- when the KEP was retired or superseded + + +## Alternatives [optional] + +One alternate to consider is the use of a side-car. The cloud-interface in tree could then be a [GRPC](https://github.com/grpc/grpc-go) +call out to that side-car. We could then leave the Kube API Server, Kube Controller Manager and Kubelet pretty much as is. +We would still need separate repos to hold the code for the side care and to handle cluster setup for the cloud provider. +However we believe that different cloud providers will (already) want different control loops. As such we are likely to need +something like the cloud controller manager anyway. From the perspective it seems easier to centralize the effort in that +direction. In addition it should limit the proliferation of new processes across the entire cluster. diff --git a/wg-cluster-api/OWNERS b/wg-cluster-api/OWNERS new file mode 100644 index 00000000..9f4deffe --- /dev/null +++ b/wg-cluster-api/OWNERS @@ -0,0 +1,4 @@ +reviewers: + - wg-cluster-api-leads +approvers: + - wg-cluster-api-leads diff --git a/wg-cluster-api/README.md b/wg-cluster-api/README.md new file mode 100644 index 00000000..1cdd75d6 --- /dev/null +++ b/wg-cluster-api/README.md @@ -0,0 +1,29 @@ +<!--- +This is an autogenerated file! + +Please do not edit this file directly, but instead make changes to the +sigs.yaml file in the project root. + +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md +--> +# Cluster API Working Group + +Define a portable API that represents a Kubernetes cluster. The API will contain the control plane and its configuration and the underlying infrastructure (nodes, node pools, etc). + +## Meetings + +Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/16ils69KImmE94RlmzjWDrkmFZysgB2J4lGnYMRN89WM/edit). + + +## Organizers +* Kris Nova (**[@kris-nova](https://github.com/kris-nova)**), Microsoft +* Jacob Beacham (**[@pipejakob](https://github.com/pipejakob)**), Google +* Robert Bailey (**[@roberthbailey](https://github.com/roberthbailey)**), Google + +## Contact +* [Slack](https://kubernetes.slack.com/messages/sig-cluster-lifecycle) +* [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-lifecycle) + +<!-- BEGIN CUSTOM CONTENT --> + +<!-- END CUSTOM CONTENT --> diff --git a/wg-container-identity/OWNERS b/wg-container-identity/OWNERS index 60681e16..e51715f4 100644 --- a/wg-container-identity/OWNERS +++ b/wg-container-identity/OWNERS @@ -1,6 +1,4 @@ reviewers: - - destijl - - smarterclayton + - wg-container-identity-leads approvers: - - destijl - - smarterclayton + - wg-container-identity-leads diff --git a/wg-container-identity/README.md b/wg-container-identity/README.md index e2b7ee32..2f23fc2e 100644 --- a/wg-container-identity/README.md +++ b/wg-container-identity/README.md @@ -4,7 +4,7 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Container Identity Working Group @@ -13,17 +13,18 @@ Ensure containers are able to interact with external systems and acquire secure ## Meetings * [Tuesdays at 15:00 UTC](TBD) (bi-weekly (On demand)). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=15:00&tz=UTC). -Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1bCK-1_Zy2WfsrMBJkdaV72d2hidaxZBhS5YQHAgscPI/edit). -Meeting recordings can be found [here](). +Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1uH60pNr1-jBn7N2pEcddk6-6NTnmV5qepwKUJe9tMRo/edit). + ## Organizers -* [Clayton Coleman](https://github.com/smarterclayton), Red Hat -* [Greg Gastle](https://github.com/destijl), Google +* Clayton Coleman (**[@smarterclayton](https://github.com/smarterclayton)**), Red Hat +* Greg Gastle (**[@destijl](https://github.com/destijl)**), Google ## Contact * [Slack](https://kubernetes.slack.com/messages/wg-container-identity) * [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-wg-container-identity) <!-- BEGIN CUSTOM CONTENT --> +The working group proposal can be found [here](https://docs.google.com/document/d/1bCK-1_Zy2WfsrMBJkdaV72d2hidaxZBhS5YQHAgscPI/edit). <!-- END CUSTOM CONTENT --> diff --git a/wg-kubeadm-adoption/OWNERS b/wg-kubeadm-adoption/OWNERS new file mode 100644 index 00000000..23b6aa6c --- /dev/null +++ b/wg-kubeadm-adoption/OWNERS @@ -0,0 +1,4 @@ +reviewers: + - wg-kubeadm-adoption-leads +approvers: + - wg-kubeadm-adoption-leads diff --git a/wg-kubeadm-adoption/README.md b/wg-kubeadm-adoption/README.md new file mode 100644 index 00000000..9427c0c4 --- /dev/null +++ b/wg-kubeadm-adoption/README.md @@ -0,0 +1,29 @@ +<!--- +This is an autogenerated file! + +Please do not edit this file directly, but instead make changes to the +sigs.yaml file in the project root. + +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md +--> +# Kubeadm Adoption Working Group + +Boost adoption of the kubeadm tool. kubeadm is a tool for creating new Kubernetes clusters easily for new users, but can also be used as a toolbox for higher-level deployment solutions. This working group makes sure kubeadm meets the extensibility requirements of those higher-level Kubernetes installers. + +## Meetings +* [Tuesdays at 18:00 UTC](https://zoom.us/j/166836%E2%80%8B624) (bi-weekly). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=18:00&tz=UTC). + +Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1KdXsLYiJYJdiRbtgZsx6qbHF4g_K-gAScB9Zs4avgzg/edit). +Meeting recordings can be found [here](https://www.youtube.com/watch?v=-Xlcrm5iT80&list=PLPgAK4Icr0ehh93BiMC3djAc5KoW7WIkl). + +## Organizers +* Lucas Käldström (**[@luxas](https://github.com/luxas)**), Luxas Labs (occasionally contracting for Weaveworks) +* Justin Santa Barbara (**[@justinsb](https://github.com/justinsb)**) + +## Contact +* [Slack](https://kubernetes.slack.com/messages/sig-cluster-lifecycle) +* [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-lifecycle) + +<!-- BEGIN CUSTOM CONTENT --> + +<!-- END CUSTOM CONTENT --> diff --git a/wg-multitenancy/OWNERS b/wg-multitenancy/OWNERS new file mode 100644 index 00000000..644b961a --- /dev/null +++ b/wg-multitenancy/OWNERS @@ -0,0 +1,4 @@ +reviewers: + - wg-multitenancy-leads +approvers: + - wg-multitenancy-leads diff --git a/wg-multitenancy/README.md b/wg-multitenancy/README.md new file mode 100644 index 00000000..40ee5a88 --- /dev/null +++ b/wg-multitenancy/README.md @@ -0,0 +1,28 @@ +<!--- +This is an autogenerated file! + +Please do not edit this file directly, but instead make changes to the +sigs.yaml file in the project root. + +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md +--> +# Multitenancy Working Group + +Define the models of multitenancy that Kubernetes will support. Discuss and execute upon any remaining work that needs to be done to support these models. Create conformance tests that will prove that these models can be built and used in production environments. + +## Meetings + + + + +## Organizers +* David Oppenheimer (**[@davidopp](https://github.com/davidopp)**), Google +* Jessie Frazelle (**[@jessfraz](https://github.com/jessfraz)**), Microsoft + +## Contact +* [Slack](https://kubernetes.slack.com/messages/wg-multitenancy) +* [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-wg-multitenancy) + +<!-- BEGIN CUSTOM CONTENT --> + +<!-- END CUSTOM CONTENT --> diff --git a/wg-resource-management/OWNERS b/wg-resource-management/OWNERS index dd22d5fc..60221854 100644 --- a/wg-resource-management/OWNERS +++ b/wg-resource-management/OWNERS @@ -1,6 +1,4 @@ reviewers: - - vishh - - derekwaynecarr + - wg-resource-management-leads approvers: - - vishh - - derekwaynecarr + - wg-resource-management-leads diff --git a/wg-resource-management/README.md b/wg-resource-management/README.md index a5522594..daf2e9bb 100644 --- a/wg-resource-management/README.md +++ b/wg-resource-management/README.md @@ -4,21 +4,21 @@ This is an autogenerated file! Please do not edit this file directly, but instead make changes to the sigs.yaml file in the project root. -To understand how this file is generated, see generator/README.md. +To understand how this file is generated, see https://git.k8s.io/community/generator/README.md --> # Resource Management Working Group Designing and shepherding cross-cutting features around compute resource isolation and utilization. ## Meetings -* [Tuesdays at 18:00 UTC](https://zoom.us/j/4799874685) (weekly (On demand)). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=18:00&tz=UTC). +* [Wednesdays at 11:00 PT (Pacific Time)](https://zoom.us/j/4799874685) (weekly (On demand)). [Convert to your timezone](http://www.thetimezoneconverter.com/?t=11:00&tz=PT%20%28Pacific%20Time%29). Meeting notes and Agenda can be found [here](https://docs.google.com/document/d/1j3vrG6BgE0hUDs2e-1ZUegKN4W4Adb1B6oJ6j-4kyPU). Meeting recordings can be found [here](https://www.youtube.com/watch?v=FUUJeWIEej0&list=PL69nYSiGNLP2uTrVwZCFtdEvLQvsbG2w4). ## Organizers -* [Vishnu Kannan](https://github.com/vishh), Google -* [Derek Carr](https://github.com/derekwaynecarr), Red Hat +* Vishnu Kannan (**[@vishh](https://github.com/vishh)**), Google +* Derek Carr (**[@derekwaynecarr](https://github.com/derekwaynecarr)**), Red Hat ## Contact * [Slack](https://kubernetes.slack.com/messages/wg-resource-mgmt) |
