Upper Brandon plantation was part of an original land patent known as Brandon, granted to Captain John Martin, one of the founders of Jamestown. He was succeeded by several absentee owners, including a grandson of William Shakespeare, until the property was purchased by Benjamin Harrison II of Wakefield in 1712.
In 1807, at the death of Benjamin Harrison III (1743-1807), his will divided the 7,000 acre property between his two sons, George Evelyn Harrison (1797-1839) and William Byrd Harrison (1800-1870), who were to receive their inheritance when they reached the age of twenty-one. When he came of age, George Evelyn Harrison took over the original Brandon house and divided the land with his brother. William Byrd Harrison, an 1820 graduate of Harvard University inherited the 3,555 acres now known as Upper Brandon and completed his mansion in 1825. The main house at Upper Brandon is a five bay center hall, red brick structure built in the Federal style with a low hip roof with a widows walk. The design of the mansion and its woodwork was influenced by The American Builder's Companion by Asher Benjamin. It has two matching three bay dependencies at either end that are connected to the main structure by low hyphens that are partially below grade. William Byrd Harrison and his family lived there until the outbreak of the Civil War. After the Civil War, William Byrd Harrison never returned to live at Upper Brandon and upon his death in 1870, the plantation was purchased by his nephew, George Harrison Byrd. In 1948, the last Harrison descendant, Francis Otway Byrd, sold the estate to Petersburg businessman Harry C. Thompson.
Following the death of Mr. Thompson, the property was sold to Fred E. Watkins of Curles Neck Farms in 1961. Watkins used the farmland at Upper Brandon to grow feed for his large dairy herd at Curles Neck, but the manor house remained unoccupied from the 1960s until 1984 when the property was purchased by the James River Corporation who restored the mansion and used the property as a conference center and corporate retreat. James River Corporation later acquired the adjoining Edlow plantation.