Oh, I LOVE castles! I admit it fully. I'm a princess at heart.
The one I really want to go to, but haven't yet, is Neuschwanstein Castle in Germany. It's the one the Disney castles were modeled from. And, speaking of Disney castles, I've visited them at Disneyland (California), Disney Paris, DisneyWorld (Florida), and Tokyo Disney.
From the list, I've been to Leeds, which is really quite something. If you like the Tudor period, then it's a great place to go.
I've been to Kensington Palace, Windsor, Warwick and Tower of London also in England, of course. Warwick was amazing, but I'm afraid they've made it rather Disneyesque which ruins it a bit to me. [If I want Disney...I'll go to one of the parks!]
There's Hearst Castle in the U.S. (California) which is fun to visit...I went as a kid and hope to go back with my kids someday.
I have also been to Casa Loma in Toronto...it was ok.
Prague Castle was nice (but you can't actually go INSIDE). But Cesky Krumlov Castle in the southern part of Czech Rep was very different and interesting to see as was the medieval town surrounding it.
Himeji Castle in Himeji, Japan is amazing; it is the best example of Japanese architecture and structure of the time (1300s).
In France, we recently went to the fortified city of Carcassonne and explored the old walls and "castle" inside. There wasn't much to the castle, but the walls and everything else were incredible.
I'm looking forward to many more castle visits...