Hamblen House, Pre Civil War era to 1969