after changing the zone interface for mgmt-zone to vrf name (as below ) the same started working
Set firewall zone mgmt-zone interface mgmt
my observation is
i have to also enable the established and related rules on the firewall for the traffic between zone.
why is this still needed even though same is mentioned in global-options as below
set firewall global-options state-policy established action ‘accept’
set firewall global-options state-policy related action ‘accept’