1.7.0-beta.1

First beta release of the kops 1.7 series.

Significant changes

• Manifests are rewritten by default, which includes a normalization phase. This can make it hard to understand the actual changes (as opposed to just the formatting changes). A feature flag has been added, export KOPS_FEATURE_FLAGS="-RewriteManifests" which can be used to disable manifest rewriting. A recommendation: you can run kops update twice, once without manifest formatting to show the real changes, and then immediately afterwards with manifest changes, which will be just formatting changes. Run KOPS_FEATURE_FLAGS="-RewriteManifests" kops update cluster, to show the real changes, apply them with KOPS_FEATURE_FLAGS="-RewriteManifests" kops update cluster --yes, then run kops update cluster to show the formatting changes, followed by kops update cluster --yes

• Default disk size increased to 64GB (masters) and 128GB (nodes). This does have a higher cost, but also gives us more inodes & more iops (and more disk space, of course!)

• Calico now configured with the correct pod CIDR: #2768. Please refer to the Required Actions section for details regarding this.

Required Actions

• Existing Calico users on clusters that were created prior to kops 1.7 are suspectible to IP conflict between Pods and Services due to an overlap of the two IP ranges. Migration to a new Pod CIDR is recommended, and is a manual procedure due to risk of potential downtime during this operation. For the migration procedure, please refer to this document.

For more details, please see the full release notes

1.7.0-alpha.1

Early access (alpha.1) for the kops 1.7 release, supporting kubernetes 1.7

Release notes to follow.

1.6.2

Please see 1.6-NOTES.md for known issues

• Weave upgraded to 1.9.8, to fix NodePort issue (thanks @jordanjennings, @justinsb)
• Fixes for (experimental) k8s.local DNS-free configurations (thanks @justinsb)
• Weave now configured with the correct pod CIDR (thanks @jordanjennings)
• Initial support for kube-router networking (thanks @murali-reddy)
• Apply cloud-labels to EBS volumes (thanks @pastjean)
• Support empty --resolv-conf (thanks @austinmoore-)
• Add --subnet and --role flags to create ig command (thanks @dtan4)
• Improvments to kops delete output (thanks @chrislovecnm)
• Match type (public/private) of DNS zones when matching (thanks @justinsb)
• CoreOS command now finds the latest image (thanks @gianrubio)
• Protokube now checks if kubelet is already running before calling systemctl start (thanks @aledbf)
• Added index to make documentation much easier to navigate (thanks @WillemMali)
• Makefile improvements (thanks @WillemMali)
• Refactor instance group / rolling-update code (thanks @andrewsykim)
• Lots of documentation and polish (thanks @chrislovecnm, @cordoval, @justinsb, @WillemMali)

1.6.1

Please see known issues for known issues with this release (primarily around upgrading from kubernetes 1.4/1.5 to kubernetes 1.6).

Features:

kops get can now output a complete cluster spec (thanks @geojaz)
kops create can set master/node volume size (thanks @matthew-marchetti)
Add ability to set cross-subnet mode in Calico (thanks @ottoyiu)
Make Weave MTU configurable and configure jumbo frame support for new clusters on AWS (thanks @jordanjennings)
Initial support for external-dns project (thanks @sethpollack)

Fixes:

Fix calico bootstrapping problems (thanks @ottoyiu, @ozdanborne)
Update to latest release of calico (thanks @mad01)
Update canal manifests for 1.6 & RBAC (thanks @heschlie)
Mark calico-node pods as critical (thanks @andreychernih)
Fix log rotation of apiserver audit logs (thanks @ottoyiu)
Update cluster autoscaler addon (thanks @sethpollack)
Set hairpin mode for flannel (thanks @justinsb)
Fix GCE disk cleanup on cluster deletion (thanks @andrewsykim)
Prevent "unbound variable" errors in kops-mfa (thanks @hugocf)
Fix e2e tests for kubernetes 1.8 and 1.9 (thanks @justinsb)
Fix directory permissions for .kube directory on master (thanks @chrislovecnm)
Lots of documentation and polish (thanks @andrewsykim, @caarlos0, @chrislovecnm, @gianrubio, @grillz, @justinsb, @mikesplain, @Shimi, @tanner-bruce, @WillemMali, @zanhsieh)

1.6.0

kops 1.6.0 adds support for kubernetes 1.6.x

Features and Bug Fixes

• Only run multiple masters if master-count or multiple master-zones are specified

• Don't run dns-controller with ingress enabled; caused issues in existing setups

• Expose apiserver audit log by setting it as a Host Path (thanks @ottoyiu)

• Add api-loadbalancer option to cli set public or internal loadbalancer (thanks @mikesplain)

• Update the terraform generator to use the value "role" instead of "roles" for the aws_iam_instance_profile resource (thanks @while1eq1)

• Add required terraform version declaration & require terraform 0.9.3

• Add instance role names/ARNs to terraform output (thanks @pastjean)

• Recognize networking=kopeio as alias for kopeio-vxlan (thanks @rdtr)

• Add critical pod annotations to kopeio networking

• Creates .kube dir at /home/admin with group/owner as admin (thanks @dolftax)

• Remove babysit-daemons flag from 1.7

• Fix channel version recommendations

• Add missing error handling (thanks @caarlos0)

• Update calico to 2.1.5 (thanks @blakebarnett)

• Add ingress nginx example addons for 1.6

• Add heapster addon for 1.6 (thanks @sethpollack, @BradErz)

• Update alpha channel

• Implemented a Git hook for the CI tests and related installer (thanks @WillemMali)

• Ensure state store not set when generating docs (thanks @rdtr)

• CI now verifies that docs have been generated (thanks @chrislovecnm)

• Added help target to Makefile and fixed small path ordering issue (thanks @WillemMali)

• Mark all phony targets with ".PHONY" in Makefile (thanks @WillemMali)

• Adds instructions for developing in Docker (thanks @svozza)

• Improved documentation & polish (thanks @BradErz, @dolftax, @four43, @lcrisci, @pierreozoux, @sethpollack, @sngchlko, @snoby, @svozza)

• Review of all command docs / help (thanks @gianrubio, @chrislovecnm)

Known Issues

• Kubernetes upgrade from 1.5 -> 1.6 requires a configmap to be created in the kube-system namespace. If kubectl -n kube-system get configmap does not return a configmap with the name "kube-dns". BEFORE upgrade please execute kubectl create configmap -n kube-system kube-dns.
• Kubernetes upgrades from 1.5 -> 1.6 users have noticed some issues with taints when running HA masters. Editing the taints by hand is a workaround. See: #2594
• Some Kubernetes upgrades from 1.6.x -> 1.6.x have experience issues with secrets and service accounts. Deleting the secrets have been a workaround (not validated). See: #2576
• Calico users have noticed problems using Calico with Kubernetes 1.6, this has been fixed in master, and will be released in kops 1.6.1.

1.6.0-beta.1

Deploys dns-controller with --watch-ingress=true (thanks @geojaz)
Dockerfile for running kops (thanks @denniswebb)
zsh autocompletion (thanks @chrislovecnm)
Add support for m4.16xlarge instance types (thanks @ryanlitalien)
Set default master size for ap-northeast-2 (thanks @buo)
kops create cluster flag for encrypted etcd volumes (thanks @pronix)
Enable CertificateSigner API on k8s 1.6
Fix validation when we have fewer nodes than minimum count (thanks @GauntletWizard)
Refactor cluster deletion (thanks @andrewsykim)
Misc fixes for golang vet warnings (thanks @pronix)
Use explicit tag management in network tasks
Fix private DNS zone creation

Experimental support for gossip-backed DNS
Initial vmware backend (thanks @abrarshivani, @luomiao, @prashima, @SandeepPissay, @vmware,
Initial EFK addon, and fix version in EFK manifest (thanks @archseer)
Experimental support for hooks

Update flannel & canal to 0.7.1, containing k8s 1.6 fixes
Update heapster to 1.3.0 and addon-resizer to 1.7 (thanks @blackstar257)
Update CNI version for k8s 1.6 (thanks @chrislovecnm)
Update route53-mapper addon (thanks @a1dutch, @LeandroCR)
Use go 1.8.1 (thanks @chrislovecnm)
Updated k8s dashboard to 1.6 (thanks @mstump)

Cleaned up CLI help and output (thanks @robertojrojas)
Clarified brew instructions around --HEAD (thanks @chrislovecnm)
Lots more polish & documentation (thanks @arthurlm44, @BrentDorsey, @caarlos0, @chrislovecnm, @geojaz, @pierreozoux, @snoby, @tazjin, @voyalab, @while1eq1)

1.6.0-alpha.2

Known issues:

• We are tracking a potential issue with flannel & calico which seems to cause the masters to lose their taints
• Please review the 1.6.0-alpha.1 notes

Major changes & fixes in alpha.2:

• Update calico, canal, flannel, kopeio-networking and weave for 1.6 (thanks @felipejfc, @ottoyiu, @pronix)
• Update cluster-autoscaler for 1.6 (thanks @ottoyiu)
• Support version selection in the kops addon manager
• Reuse the public key if it exists, so upgrade doesn't require rekeying
• Support private dns with terraform (thanks @ahl)
• Make sure we start docker-healthcheck.service
• Update to docker 1.12.6 for k8s 1.6
• Fix CNI bin & conf paths (thanks @prachetasp)
• AWS Instance Tenancy support(thanks @dacohen)
• Fix GCE disk cleanup (thanks @andrewsykim)

More goodies:

• Add AWS CloudConfig DisableSecurityGroupIngress Configuration Parameter (thanks @waldman)
• Update protokube's embedded kubectl to 1.6.1 (thanks @chrislovecnm)
• Implement --filename option for kops delete (thanks @MrTrustor)
• Update route53-mapper v.1.3.0 (thanks @a1dutch)
• Set nf_conntrack_max in kubeproxy (thanks @andrewsykim)
• Feature flags support for kubelet (thanks @chrislovecnm)
• Expose options.CreateKubecfg to the CLI (thanks @kris-nova)
• kopeio networking should not set configure-cloud-routes
• Fix node role logic in validate cluster command
• Docs & polish fixes (thanks @adrianmoisey, @caarlos0, @chrislovecnm, @kris-nova, @MrTrustor, @pronix, @twpayne )

1.6.0-alpha.1

1.6.0-alpha.1

1.6.0-alpha.1 is a prerelease early-access of kops 1.6, which is the release with full support for kubernetes 1.6.
This version of kops & kubernetes has not yet undergone extensive validation, and there will be improvements
made before release of kops 1.6.0.

This is not a full set of release notes, but rather a summary of the highest impact changes in the 1.6 release:

• RBAC can be enabled by passing the --authorization=rbac parameter to kops create cluster,
  or via kops edit cluster and change authorization from alwaysAllow: {} to rbac: {}

• The standard RBAC policy for 1.6 means that all access to the Kubernetes API using the default
  service account method will be denied.

• The taints & tolerations have changed as part of their graduation from alpha. The taint is now a field on the node:

spec:
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master

An example toleration (as used in dns-controller) is:

spec:
  tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master

Note that the annotation form is ignored. To schedule a pod on the master, the toleration must be updated
and moved from an annotation to the field.

• A new label for nodes, mirroring the toleration, is added and is now preferred: node-role.kubernetes.io/master=
  (node-role.kubernetes.io/master with an empty value). kubernetes.io/role=master is still present, but
  the node-role.kubernetes.io/<role>= form is preferred. kubernetes.io/role=node and node-role.kubernetes.io/node=
  are also present.

Workaround: create the configmap with kubectl create configmap -n kube-system kube-dns before updating.

Known Issues

Rolling updates

Rolling update to 1.6 does not succeed because new kube-dns pods mount a configmap with an optional volume map,
but that is enforced by the kubelets, which are upgraded after the master.

etcd3

kops is not yet recommending etcd3. Right now we are working on resolving issues such as HA upgrade support.

1.5.3

• Important for Terraform Users Make ELB naming unambiguous by including the full cluster name. This will cause the ELBs to be recreated if using Terraform with private topologies, causing disruption of external access to the API and of external access to the bastion (if enabled). Expected disruption is less than 5 minutes. Use export KOPS_FEATURE_FLAGS=+UseLegacyELBName to keep the legacy naming and avoid disruption. Fix #1899

• Fix terraform output of shared subnets. Fix #1977

• Add support for i3 instances (thanks @geojaz)

• Experimental drain rolling-update,

• Experimental GCE support

• Update Weave to v1.9.3

• Put flannel in guaranteed class (thanks @mihok)

• DNS autoscaler fixes (thanks @MrHohn)

• Remove legacy flags (thanks @mtaufen)

• Add route53 mapper addon (thanks @itskingori)

• Build fixes (thanks @zmerlynn)

• Disable cloudformation delete (thanks @kris-nova)

• Docs fixes (thanks @bowei, @jonchiu, @dosullivan, @DualSpark, @foxylion, @kris-nova

1.5.2-beta.2

Breaking changes:

• Route53 permissions are scoped only to the hosted zone in use, rather than all zones. You can add additionalPolicies
  to expand the permissions available.

Changelog:

• Scope route53 permissions to DNS_ZONE only (thanks @zytek)
• Improved kops edit when errors are made (thanks @geojaz, @sethpollack, @yissacharcw)
• Initial support for CoreOS (thanks @aledbf)
• Initial support for CloudFormation output
• Add support for adjusting ELB idle timeout for apiserver (thanks @zacblazic)
• Cloud labels can be defined at cluster level, and through CLI flags (thanks @robinpercy)
• More flag mapping - auditing & volume mapping (thanks @aledbf, @sethpollack)
• Fix issue with leaking elastic IPs on cluster delete (thanks @ikropotov)
• Fix terraform render for additional security groups (thanks @philk)
• Improve suggestions after cluster creation, and validation output (thanks @kamilhristov)
• Unset kubectl current-context as part of cluster delete (thanks @kamilhristov)
• Terraform: include output variables, for reuse in a module
• Allow general access from nodes to masters, but block sensitive ports (etcd)
• Bump dns-controller to 1.5.2: include namespace in keys
• Update alpha channel with k8s 1.4.9 and 1.5.3 (thanks @itskingori)
• Run Weave Net in Guaranteed Quality of Service (thanks @bboreham)
• Bump Weave Net to 1.9.0 (thanks @bboreham)
• Flannel support (thanks @mihok)
• Bump calico version to 2.0.2 (thanks @heschlie)
• Canal support (thanks @heschlie)
• Autoscaling improvements (thanks @andrewsykim)
• Increase conntrack table size using sysctl (thanks @aledbf)
• Update cgroup hierarchies for k8s 1.6 (thanks @derekwaynecarr)
• Improved validation (thanks @geojaz, @sethpollack, @sstarcher)
• Resolve DNS Hosted Zone ID while building IAM policy (thanks @zytek)
• Makefile fixes (thanks @aledbf, @apenney, @zytek)
• Choose only one subnet per AZ for API ELB
• Prevent spurious changes around IAM roles
• Delete CloudFormation stack on kops delete cluster
• More tolerant parsing of IAM policies
• Allow bastion image override when creating cluster
• Install ethtool via nodeup, so it is always installed
• Docs fixes - thanks @ahawkins, @andrewwatson, @chrislovecnm, @crigor, @fate-grand-order, @geojaz, @heartlock, @itskingori, @kris-nova, @mshenhera, @yissachar, @zacblazic, @zytek