Benjamin Morgan, originally from Philadelphia and of a prominent American Revolutionary War family, purchased the land in 1804. After his death in 1826, his son Thomas Ashton Morgan inherited the estate and began growing sugarcane there.
Between 1847 and 1853, Thomas A. Morgan commissioned the noted Philadelphia architect William L. Johnston to design the plantation house. Many of the elements of the house, including the bricks, were produced in Philadelphia and transported to the building site, where they were assembled by slaves.
The house was notable for its plumbing and central heating systems. The attic had a 700-gallon copper boiler with wood and coal burners; the home featured hot and cold running water year-round, making it one of the most technologically advanced buildings of the time. The unique architectural style, featuring Gothic revival and Tudor elements, led to it being described as "a proper English manor house that looked as if it had been set down on the Louisiana landscape by mistake and forgotten about."
In 1876, the house was sold to Louis Fasnacht. The plantation was named Orange Grove around this time for the long row of orange trees that grew on the property.
The home was purchased in 1957 by the Southern Railway system with the intention of its restoration.
In 1984, the house was destroyed by a fire when four vandals entered the home. One of them started a fire.