The Tories’ tough stance on immigration was once relied upon – now polls show Labour are more trusted