Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Upgrade errors with Helm 2.1 when TLS enabled

Description

The Helm Chart upgrade from 2.0.x to 2.1 encounters the following error:

{"level":"error","ts":1610489073.7747614,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"couchbase- controller","request":"default/cb-test-couchbase-cluster","error":"secrets \"cb-test-couchbase-cluster\" already exists","stacktrac e":"github.com/go-logr/zapr.(*zapLogger)

*Steps to Reproduce*

1) helm install cb-test couchbase/couchbase-operator --version 2.0.2
this will install the standard default

2) then:

kubectl replace -f crd.yaml
kubectl create -f crd.yaml

3) then upgrade the chart:

helm upgrade cb-test couchbase/couchbase-operator

We will hit the error above.

In addition, If we have the following set to true in the values.yaml

# TLS Certs that will be used to encrypt traffic between operator and couchbase tls: # enable to auto create certs generate: false # Expiry time of CA in days for generated certs expiration: 365

Then upgrading to 2.1 operator from previous version will encounter the following error:

{"level":"error","ts":1611102051.5212724,"logger":"cluster","msg":"Reconciliation failed","cluster":"default/demo","error":"certificate cannot be verified for zone: x509: certificate is valid for localhost, *.demo-couchbase-cluster.default.svc, *.demo-couchbase-cluster.default, *.demo-couchbase-cluster, *.demo-couchbase-cluster-srv.default.svc, *.demo-couchbase-cluster-srv.default, *.demo-couchbase-cluster-srv, demo-couchbase-cluster-srv.default.svc, demo-couchbase-cluster-srv.default, demo-couchbase-cluster-srv, *.demo-couchbase-cluster-srv.default.svc.cluster.local, host.demo-couchbase-cluster.default.svc.cluster.local, not host.demo

Environment

None

Release Notes Description

None

Activity

Show:

Eric Schneider May 12, 2021 at 6:06 PM

assigned https://couchbasecloud.atlassian.net/browse/K8S-1955#icft=K8S-1955 to handle QE review of the docs. I'll leave it up to you with what to do with this ticket. Perhaps you can close this one since the final review is being handled in the other ticket?

Patrick Stephens April 28, 2021 at 9:48 AM

Cherry picked change over from master branch

Matt Ingenthron April 27, 2021 at 3:53 PM

Need to cherry pick this back to 2.1.

Patrick Stephens April 27, 2021 at 3:51 PM

I think this should be resolved in the linked docs now - do we need to backport it as well?  or

Tin Tran January 20, 2021 at 11:53 PM
Edited

Hi Thank you for the workaround, I will make a note here with more details for record keeping:

1) Install the Operator 2.0.2 without the cluster

helm install cb-op couchbase/couchbase-operator --set install.couchbaseCluster=false --version 2.0.2

2) Deploy Couchbase Cluster with TLS.generate set to true.

helm install --values values.yaml demo couchbase/couchbase-operator --version 2.0.2

3) we see the operator-tls and server-tls secrets

kubect get secrets NAME TYPE DATA AGE demo Opaque 4 7m13s demo-demo Opaque 2 11m demo-demo-operator-tls Opaque 1 11m demo-demo-server-tls Opaque 2 11m

4) At this point, before we upgrade, we must re-generate the new certs with the correct format, we can do that by:

helm template demo --values values.yaml couchbase/couchbase-operator > secretsdemo.yaml

Please note that we must match the helm's release name (demo in this case) of the couchbase server cluster and use the same values.yaml.

5) From the secretsdemo.yaml, remove everything except the operator and server tls secrets then replace the current secrets with:

kubectl replace -f secretsdemo.yaml

We should now see the following lines in the Operator logs:

{"level":"info","ts":1611184007.8059275,"logger":"cluster","msg":"Reloading certificate chain","cluster":"default/demo","name":"demo-0000"} {"level":"info","ts":1611184007.941515,"logger":"cluster","msg":"Reloading certificate chain","cluster":"default/demo","name":"demo-0001"} {"level":"info","ts":1611184008.0639791,"logger":"cluster","msg":"Reloading TLS client configuration"}
Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Story Points

Components

Sprint

Fix versions

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created January 13, 2021 at 7:25 PM
Updated July 21, 2021 at 3:09 PM
Resolved April 28, 2021 at 9:48 AM
Configure
Instabug