Hi Frank,
I was in the Netherlands a couple of weeks ago for the weekend and crammed so much in (it's in my 'Dots round Europe' link below) The train system in the Netherlands is fantastic, cheap & regular.
Nowhere is more than 3 hours away from Amsterdam, so if you want to get out of the main city to the country, it's really easy. Places like The Hague; Rotterdam are only half an hour to an hour away from Amsterdam.
You can hire bikes from any train station in the Netherlands - the rental places are outside the stations (despite there being more bikes than Dutch people in this country, make sure you lock them securely, as they do get stolen, and usually chucked in a canal!)
Although I didn't cycle it, if you go to Gouda by train, you could rent a bike and cycle to a place called Oudewater - it's really pretty, and really Dutch - plus as you mentioned liking quirky stuff, there's an ancient Waag (weighhouse) there where they used to weigh people to see if they were a witch in the 18th century - you can still get weighed there, and get a certificate to prove you're not a witch or warlock - if indeed, you are not

Similarly, there's a place called Kinderdijk where there are 18 Dutch windmills all in a row - I was gutted not to make it there, as I think it would be soooo pretty!! I think it's cyclable from a place called Dordrecht.
In Amsterdam itself, as well as what the others said, don't miss out on the Anne Frank Museum. There's also a free walking tour that you can do, that leave from the torist office near the Centraal Station. It's very good - the guys work for tips, so they do a really good job to get good tips. I would recommend at least one more night based in Amsterdam (or book in a surrounding town where it's still convenient, but is cheaper than the capital - I stayed in Delft when I was there)
How are you getting from Paris to Amsterdam? If you're going by train then you could take in Bruges plus Antwerp on the way (1 day in each?) But I would recommend more than 2 days in the Netherlands. I crammed a lot in by train in 3 days, but I want to see more of that country! I don't know much about Paris I'm afraid - if someone else is as keen on Paris as I am on the Netherlands, you might find your trip becomes a 50:50 split recommendation!
For the places that are set in stone, I'd suggest missing Germany out unless you are flying / don't mind not having much time there. You'll get so much out of the Netherlands by devoting more portion of your trip there.
I hope you like it as much as I do!!