Starting in the 11th century, the Byzantines began fortification. Saint Hilarion formed the defense of the island with the castles of Buffavento and Kantara against Arab pirates raiding the coast. Some sections were further upgraded under the Lusignan rule, who may have used it as a summer residence. Much of the castle was dismantled by the Venetians in the 15th century to reduce the up-keeping cost of garrisons.