Obtaining accurate addresses from census

[tigger449]

I'm confused by the house numbers in the census returns for 1851 to 1901. Although my ancestors John & Patience EMENEY lived in the same street in each census, the number against their household varies. In 1851 it's 32, 1861 is 53, 1871 is 55, 1881 is 49, 1891 is 48 and 1901 is 53 again. Can I assume that the numbers are allocated by the enumerator, rather than the actual address of the property? How can I work out which house they actually lived in?
Many thanks,
Brian

 

[davelambert271]

I think the numbers are Schedule numbers not the number of the house. If recorded, they are in the first col, the house number would be in the 2nd col along with the street

dave