I'm confused by the house numbers in the census returns for 1851 to 1901. Although my ancestors John & Patience EMENEY lived in the same street in each census, the number against their household varies. In 1851 it's 32, 1861 is 53, 1871 is 55, 1881 is 49, 1891 is 48 and 1901 is 53 again. Can I assume that the numbers are allocated by the enumerator, rather than the actual address of the property? How can I work out which house they actually lived in? 
Many thanks,
Brian
Many thanks,
Brian