Peter Andre was afraid his wife Emily would leave him