I found the culprit.
pinchzoomimage is an Img inside a Container.
when setting width and height of the image, you also end up setting width and height of the container, which for some reason causes the snap back when you try to pan to the bottom.
I added the workaround by defining config parameters imageWidth and imageHeight which are applied to Img only. Now it pans like a charm.