Wander down from the castle grounds into this very peaceful shrine with the most wonderful wooden shrine entrance.
It's on the south east side of Ohori park, roughly behind the castle, next to NHK. The closest subway station is probably Ohori, but that is a way off. However, if you have got as far as Ohori Jpaanese garden, it is just a little beyond that.