I would guess that you tie into the Castle Townshend group back in England rather than Ireland. I am assuming you looked at their printed genealogy fairly thoroughly and found no unidentified males.
Unfortunately, they don't know where Col. Townsend came from in England itself but I think your Richard is only, say, 4 generations separated from him and thus you are fairly far back. It is thought he came from the west of England.
Richard seems to have married Anna Stone at about 34, do you know if he had an earlier wife? The Stones were a prominent family in VA so he must have had some status, too.