The first structures on the site of Morven Park date to about 1780. A fieldstone house built by Wilson Cary Seldon is now a part of the north wing of the main house, stuccoed over to match the rest of the mansion. Judge Thomas Swann acquired the property around 1800. Around 1830 Swann built the center two-story portion of the house, with flanking pavilions. It is not known whether the pavilions were initially linked to the house, but the renovations included the prominent tetrastyle Greek Revival portico that dominates the front portico. While the brick structure remains, now stuccoed, none of the Swann interiors exist. Judge Swann's son, Thomas Swann Jr., began a remodeling program around 1850, using the Baltimore firm of E.G. Lind and William T. Murdock as architects, converting the Palladian house to the Italianate style with four towers, including one that was five stories tall, stated by The Buildings of Virginia to resemble Queen Victoria's Osborne House on the Isle of Wight. The pavilions were by this time linked to the main house. The towers were later removed by Thomas Swann Jr.'s daughter, Mary Mercer Swann Carter, and her husband, Dr. Shirley Carter. They made more changes, raising the height of the hyphens to two stories and reworking the interior.

Westmoreland Davis, a New York lawyer, had roots in Virginia and purchased Morven Park in 1903. He and his wife were avid equestrians, and they quickly became involved in the Virginia fox hunting community. Davis was a founding member of the Loudoun Hunt and served as Master of Foxhounds. He later made Morven Park into an agricultural showpiece, while his wife developed formal boxwood gardens near the house. He served as Virginia's governor from 1918–1922. He and his wife were the last private owners of the estate, which is now run as a nonprofit organization that raises money to keep the park open to the public.

From 1967–1991, Morven Park was home to the world-renowned Morven Park International Equestrian Institute. The Institute was a training center for riding instructors, teaching advanced dressage, 3-day-eventing and show jumping. A number of expert riders trained at Morven Park went on to international equestrian competition. In its day, Morven Park was considered the most prestigious riding school in the United States, and flourished under the direction of Major John Lynch, an instructor at the British Army Equitation School at Weedon, as well as at the English military academies of Sandhurst and Woolrich. He had been coach and trainer to British, Irish and American Olympic teams.

Plantation or merely a mansion?