Four in 10 voters think the Labour party’s Brexit policy is to rejoin the European Union, according to a new poll.