Frascati was built between 1821 and 1823 for Philip Pendleton Barbour, Associate Justice of the Supreme Court of the United States and statesman. Barbour's father, Thomas Barbour arrived in Orange County as a young man from Culpeper County. In 1806, Thomas Barbour and his wife, Mary, sold their home tract of 885 acres (3.58 km2) to their son, Philip. Barbour entered into his "Articles of Agreement" with master builder John M. Perry on 7 November 1821. Barbour was familiar with Perry's capabilities through his work at the University of Virginia and was attracted to Somerset, the home of his neighbor, Thomas Macon. Like Somerset, Frascati was influenced by the more conventional, central hall Federal house style. This conventional format contrasted with the tastes of Barbour's brother, Senator James Barbour of Barboursville. Barboursville was designed by Thomas Jefferson in 1822 and displayed Jefferson's mixing of French and Palladian forms similar to that at Monticello.
Frascati was Barbour's home until his death in 1841. Barbour's wife sold the estate in 1848 to Captain James Magruder, another local builder who had worked under Jefferson on the University of Virginia. Frascati was then long the home of Mrs. William H. Lyne, followed by Mr. A. D. Irving, a distant relation to Washington Irving. The residence is now owned by the Barrow family, who have preserved its farm setting.