The James Arnold House, a Federal Style home, was built in 1821, undergoing many additions and changes throughout the years including the Mansard third story in 1873. James Arnold was born in Providence, R.I. in 1781, moving to New Bedford to work for merchant William Rotch, Jr. Arnold became a successful businessman with a distinct interest in agriculture and horticulture, helping found the New Bedford Horticultural Society in 1847. He curated extensive gardens, importing trees and exotic fauna from his travels abroad. He opened his gardens to the public, which was an unusual act of hospitality and charity for the time. His cultural impact did not stop there as he donated some of his fortune to the advancement of horticultural and agricultural studies. In 1872 the Arnold Arboretum was founded on land belonging to Harvard in Jamaica Plain to celebrate his stewardship.
The Arnold Mansion is now the property of the Wamsutta Club and named to the National Historic Register of Historic Places.