When a pipeline/replication is configured with bandwidth limit and the pipeline stops the Xmem nozzles do a cleanup. This clean-up is stuck because the writers (i.e. xmem nozzle writing to socket) wait for bandwidth throttler (referred to as only throttler henceforth) to release some capacity/quota.

However due to pipeline stopping, the throttler goroutine also exits. So we now have a situation where the writers are waiting on a non-existent throttler. 

The stacktrace for Xmem Nozzle

following stacktrace shows waiting on the bandwidth throttler:

Steps to reproduce:

1. Create replication with bandwidth usage limit

2. Ensure that usage limit is such that the writers block all the time. The following log line will indicate such situation

1. Pause the replication.

2. Check goroutine stack trace