We have seem this behavior when using constant learning rates with logistic regression models.
In theory the iterates should not diverge with constant rates but bounce around the true value.
This behavior is still not confirmed but would be helpful if anyone could run an independent experiment.