The question is u wanna course option or thesis option. For course option, maybe u need to start from diploma. For thesis option, it mainly depends on the supervisor. If u can fine a supervisor, then it is fine. I think math background is not bad. Actually, some professors perfer this. If u don't wanna financal support, things are easier. Otherwise, you may meet some problem.