A reminder of the history of Linnansaari Island is the Tähtitorni ("Observatory"), which has functioned as a summer cafe for nearly 120 years. It was built in the place of Oulu Castle, finished in the 17th century. The castle itself was destroyed after a fire broke out as the result of a lightning stire, which caused the gun powder cellar to explode. As a result of the famous "Oulu Big Bang" in 1793, a part of the office building's base, a cellar, a southside wall and a gateway were all that was left of the castle. The observatory for teaching astronomy at Oulu Maritime School was built on top of the office building.

There is a persistent urban legend about a secret passage that supposedly leads from Linnansaari to the town side near the Market Square ans the Radisson Blu hotel, or possibly near Oulu Lyceum. On the other hand, plenty of ghost stories about the castle cellar, once also a prison, and the observatory have originated from dramatic human fates.