Border layout generally throws a bit of a strop if you try to add or remove components from it. The standard approach for handling this is to wrap your center region in an extra level of container with a fit layout and then switch the child item of the container.
You can then do remove/add on the container.
// Exactly how you choose to instantiate the viewport is up to you
// The item you want in the center goes here
This is touched upon in the docs: