You don't need planning permission to build a castle in the UK. Truth or Myth?

This link is to a recent story in the news about a man who tried to get away with building a castle in haystacks to avoid planning rules. One would assume then that it is a myth that you don't need planning permission to build a castle, as Robert Fidler has been refused planning permission by Reigate and Banstead Council.

http://www.dailymail.co.uk/news/article-2280764/Farmer-told-MUST-pull-castle-built-secretly-haystacks-loses-year-planning-row.html

I believe it would be illegal to fortify any building for military purposes without special permission, but merely building something that looked like a castle would need ordinary planning permission. For more details on UK planning, http://www.direct.gov.uk/en/HomeAndCommunity/Planning/PlanningPermission/DG_4018203
and http://www.direct.gov.uk/en/HomeAndCommunity/Planning/index.htm