Mary Hirst Pepperrell and William Pepperrell moved into this garrison-type house with stone and wood fences, and a gambrel roof with four chimneys, after the 1734 death of William Pepperrell Senior. Pepperrell Senior built this house in 1682; as a merchant, he also owned and traded in slaves, and was noted as one of the wealthiest men in New England.

In the 17th and 18th centuries, owning enslaved people was a mark of wealth and success in Maine, as was owning a four-chimney mansion. The enslaved people who worked as house servants and laborers greatly benefitted their owners and Maine's economy, supporting grand homes such as the Pepperrell mansion at Kittery Point, which still stands in 2021.