Asa Mann purchased this property in 1841 and the house was built around 1845. In 1863, the structure became home to Mary Ann Silliman-Ayers-Lindsay-Richardson-Lapsley. Mary Ann had eight children - none of whom reached maturity - and also outlived her four husbands, which left her quite wealthy. Elias Ayers, her first husband, was a successful dry goods merchant who donated $15,000 in 1847 for the establishment of the New Albany Theological Seminary. The house was updated in 1885 in the Second Empire style, with the Mansard roof, top floor and ornament added at that time.