[BP 7.2.1] - XDCR - rebalancing progress visibility

Description

Speaking with support team, one of the asks that support would like to see is to enhance the visibility of XDCR rebalancing.
Currently, when a cluster is being rebalanced, it provides a percentage of the rebalancing progress.

Current issue
Once rebalances happen either on the source or target, XDCR has a "count down" mechanism to wait for VB movement to stabilize. Once it stablizes, XDCR pipeline is restarted and data movement will continue.
In the meantime, before pipeline is restated, there could be fewer data movements as the VBs that are rebalanced are no longer streaming. From customer's pov, without feedback, it looks like as if XDCR stalled and no indication is given for the reason why it stalled.

Source side rebalance
However, once the source cluster has finished rebalancing, XDCR itself will undergo its own rebalancing mechanism.
Internally, it has a few counters. And, depending on the situation, will restart the pipeline after a certain amount of time.

Target side rebalance
Moreover, when a target cluster is rebalanced, source XDCR currently does not show any status either, even if it is aware of target topology changes.
Adding this knowledge to XDCR status could also help the visibility into why docs aren't replicated, and when the pipeline restart will happen next.

It would be considered an improvement for supportability if XDCR can showcase each node’s rebalance progress.

One way to do it is to provide this as a “status” in the pipeline warning/status if any VBs are not being replicated, and is pending pipeline restart. In the meantime, which wouldn’t require any UI changes, and can come later.

 

Issue

Resolution

It appeared that XDCR had stalled and an explanation was not provided. For rebalances on the source or target, the XDCR pipeline should be restarted, and data movement should continue. Before the pipeline was restated, there might have been fewer data movements as the rebalanced VBs were no longer streaming.

An ETA is now provided in the Server UI to show when the pipeline is due to be restarted.

Components

None

None

Labels

Environment

None

Release Notes Description

None

Attachments

2

is a backport of

Activity

Show:

Ayush Nayyar July 24, 2023 at 11:27 AM

Verified on 7.2.1-5861. Pipeline restart estimates were popped up in activity log for source and target cluster rebalance/topology change.

CB robot June 22, 2023 at 11:15 PM

Build couchbase-server-7.2.1-5812 contains goxdcr commit 8573af6 with commit message:
: notify UI of rebalancing event

CB robot June 22, 2023 at 11:15 PM

Build couchbase-server-7.2.1-5812 contains goxdcr commit 6f17964 with commit message:
: Fix a panic

Fixed

Details

Assignee

Reporter

Story Points

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created May 5, 2023 at 11:05 PM
Updated September 18, 2023 at 3:01 PM
Resolved June 22, 2023 at 9:55 PM
Loading...