The known history of Island Harbour goes back to the Middle Ages when the land was owned by the medieval abbey of St Mary of the Quarry (or Quarr Abbey), which was founded in 1132. The abbey was granted areas of the nearby parish of Whippingham in the early to middle of the 13th century by Henry de Clavill. Shortly afterwards, the abbey also acquired the lands of Cleybrokes or Cleybroc, which would have encompassed the whole of the Island Harbour site. The prominent features of the site are the two 'lucks' or streams which flow into the River Medina. One of these has been known at various times as both Claybrooks Luck and Great Luck, whilst the other is North Fairlee Luck.
