Without the Brexit glue, support for the Conservative Party is coming unstuck