This was the home of lifelong Westport Islander Charles E. Colby (1861-1939) and his wife Annie May Perkins (1866-1955). Colby was a carpenter, and did many renovations on this family home where he and his wife raised six children -- converting it from a cape with added space and embellishments. He also ran a farm and a saw mill and was involved in town government, serving as a selectman for many years.

In 2000, Dennis Cromwell bought the Colby homestead from Hazel Colby, wife of Clarence Colby, one of Charles E. Colby's grandsons. The house Dennis bought had two chimneys. When he saw a photograph that showed three chimneys, he became curious about the third chimney pictured on the south (left) addition of the house. He examined the ceilings, floors and crawlspace for cutouts or signs of a chimney foundation, and found none. When Clarence Colby saw him under the crawlspace and learned what he was doing, he laughed. Colby said there was never a third chimney, it was a "chimney" made of wood that was painted and faced to look like the other two chimneys so that people would think there was heat in the south addition of the house -- which some considered a sign of affluence.