Yusuf said working to get a Reform government was no longer “a good use of my time” after a row about banning the burqa.