Routing to multiple Wan ip's on the same network port



I have a cloud server setup with multiple internet ip’s on the one network card.

I have setup and esxi server using one of the ip’s and then built a vyos client on this which will use another of the ip’s as it’s outside ip address.

I then have setup an internal network address which vyos connects too via another internal esx network card to a few vm’s.

I have configured basic Nat and my hosts can see the internet through the vyos and all seems to be working well.

However if I try and get to the public ip address of my esx host from one of the internal hosts it will not route to this address.

Does anyone know how to resolve this?