Temple Hall was constructed in 1810 for William Temple Thomson Mason (24 July 1782–1862), a son of Thomson Mason and his second wife Elizabeth Westwood Wallace of nearby Raspberry Plain, and nephew of George Mason. The estate became a hub of Leesburg society and was visited by Gilbert du Motier, marquis de Lafayette during his grand tour of the United States on 9 August 1825. Lafayette was accompanied by President John Quincy Adams and former President James Monroe, who was then residing at his Oak Hill plantation in southern Loudoun County. The three gentlemen witnessed the baptism of Mason's two youngest daughters at Temple Hall with Lafayette serving as godfather for Mary Carroll, and Adams and Monroe serving as Maria Louisa's godfathers. Altogether, Mason and his wife, Ann Eliza Carroll, raised ten children at Temple Hall. In addition to the Mason family, approximately twenty enslaved African-Americans resided on the property. Under Mason's management, the Temple Hall estate was a modestly successful farm. Mason cultivated orchards and raised corn, wheat, and livestock.
Mason retired to Washington, D.C. in 1857 and sold his farm to Henry A. Ball, who farmed it with using approximately 20 enslaved young men, 17 under age 16. Ball favored secession, and after Virginia voted for secession as the American Civil War began, two of Henry Ball's enlisted in the Confederate States Army's cavalry. In 1862, Henry Ball was arrested for refusing to take an oath of allegiance to the Federal Government. Ball was released after spending nearly a year in the Old Capitol Prison. John S. Mosby visited Temple Hall on 5 July 1864 and dined with the Ball family. While at the house, Mosby received information that led to the Action at Mount Zion Church. Temple Hall remained in the Ball family until 1878 when it was sold at auction.