After the death of Count Friderik III of Ortenburg in 1418 at the castle, the lordship of Lipnica was inherited by the counts of Celje; after their extinction in 1456 the castle became a ducal holding. Its castellans included the houses of Heunsberg, Dulacer, Velacer and Stain (of Kamen pri Begunjah), and the Lambergs. The lordship of Lipnica was managed by the Kacijanar family, the counts of Dietrichstein, and others, until the counts of Thurn merged it with Radovljica around 1600.
