The ruins on the hill are the remains of the best-preserved and most important castle in the Tolmin area.

The Tolmin area, especially the important roads, was controlled from its ramparts between the 12th and 17th centuries and during WWI.

Many powerful feudal lords, the Aquileian Patriarchs, the Counts of Gorica, the Town of Cividale, the Venetians and finally the Hapsburgs, all strove to occupy Kozlov Rob.

The provincial governor entrusted the protection of the castle to powerful noble families.

The first count of the castle was first mentioned in 1188; and the first mention of the St. Martin Chapel in the castle dates to 1194.

The castle building was damaged during earthquakes in 1348 and 1511.

At the end of the 16th century the castle was still being renewed.

After 1651, when the Coronini's took over the Tolmin nobility and governorship the castle was finally abandoned and a new castle was built in the town of Tolmin (a part of which now houses the museum).