Hi davej,
I had a look at the 1881 entry, and had a thought that maybe Samuel and Mary were father and daughter.
So I scratched back to 1841 and found this.
Oswestry, Shropshire.
Reg,dist. Oswestry
Sub.reg. St.Martins.
Samuel Lee. -30
Charlotte Lee.- 35
Mary Lee. - 5
Sarah Lee. - 1
Another family from Redbourn, Herts. 1851.
ED 3c
House. 40
Samuel Lee.39
Esther. 36
Charles. 16
Henry. 12
Eliza. 9
Mary Ann. 6
Your Samuel Lee and family appears to be travellers in 1881, and that may be why I can't find them back down the various census.
Sorry I can't be much more help to you, but I'm sure someone here can be of more help.
Steve.
