Technically a Castle

Not a temple or a shrine.

This was the headquarters for the dominant families who ruled the area before areas were amalgamated under central Japanese rule (Shoguns).