The Little Castle was built in 1612 by Sir Charles Cavendish as a retreat from his principal seat at Welbeck, a few miles away. The Little Castle was inherited by his son William Cavendish who decorated the Little Castle from 1619-1634.
The design of the Little Castle was intended to evoke a Norman great tower, which it clearly resembles viewed from a distance, rising sheer from the cliff.