Construction of the house was started in 1822 by one of the first landowners, John W. Freeman or Josiah Huie. It was completed by the next owner, William Montgomery, in 1844. Mary Emma Scott Stewart purchased the property in 1910 from the Montgomery family. Jacob Janes then owned the house for two years before it was sold to the Fred Whittaker family in 1937. The Whittaker family restored and added modern amenities to the house. They owned it until 1978, when the house was inherited by M.W. Petrey, Jr. It was sold again in 1982 to the Union Camp Corporation, a pulp and paper company that allowed the Autauga County Heritage Association to preserve and maintain the house. Following Union Camp's acquisition by International Paper, the new owner donated the house to the association in 2007.