When the windmill was built, Brixton Hill would have been relatively rural, surrounded by open fields, but during the 1850s, the area around the mill had become built up and the new houses sheltered the mill from the strong winds needed to drive it. The mill could no longer work efficiently and in 1862 Ashby and Sons transferred their milling business to watermills at Mitcham. The sails were removed two years later and the mill was used for storage.

In 1902, when the lease on the Mitcham mill expired, a steam engine was fitted to power Brixton Windmill. This was later replaced by a gas engine but the mill was finally closed down in 1934.

Sometime after it ceased to be a working windmill a wooden gallery was built around the top. London Borough of Lambeth, Archives Department.