A relatively recent reference is found at
http://www.theguardian.com/politics/2013/may/18/david-cameron-ally-activists-loons,
Answering this sort of question can take a lot of research beyond items on the internet, which I don't plan to do, but a good start is found here (inspired by the 2013 remark), with much more on the linked site itself:
'How did the word "swivel-eyed" enter the British political lexicon, and when did we first get "swivel-eyed loons"? Here's a preliminary report, armed with access to the Lexis database, and the help of some friends on the Twitter with very good memories.
As long ago as 1983, Michael Meacher was described in the unlamented Punch as a "swivel-eyed Leftie lunatic", so the term has been in circulation for a while...' -see
http://virtualstoa.net/2013/05/18/a-short-history-of-swivel-eyed-loons/ by Chris Brooke
On that site (which in the Comments section traces "swivel-eyed" to the early 19th Century)and elsewhere, there is reference to this term being used to describe Labour politician Tony Benn:
'And so to Tony Benn's consistent and ideologically pure euroscepticism that began long before the word was invented. There can be no doubt about its sincerity or that he was motivated to a great extent by a love for Parliament and parliamentary democracy. He was also motivated by a fear that, once in the EEC, Britain would never become the socialist country he wanted it to be. That, together with his rather intimidating way of speaking (he was the original swivel-eyed loon before he became the benign elder statesman) meant that the BBC used him freely to scare people. It worked. Benn's participation in the NO campaign in the referendum of 1975 contributed to its heavy defeat.'
-see
http://tee2i.org/articles/1030
