The manor was originally located on Kahiluoto island in Taivassalo, Southwest Finland. The precise year of the building is not known, but the architectural style of the house points out to the late 18th century. Some details - such as the two front entrances - then again to Carl Wijnblad's architectural book of designs published in Sweden 1755.

The manor houses are often associated with ghost stories in folklore. According to the story, Kahiluoto Manor is haunted by the 'White Lady'. It is thought that the ghost is Agneta Eleonora de la Myle, the lady who let the manor be built. She lost her son and soon afterwards her husband. The haunted 'White Lady' is therefore mourning the loss of her loved ones.