From the Museum of Jaén, the creation of some museums in the province was promoted, such as the Cazorla Museum, as a section of the Jaén museum. It was founded in 1971, although after the transfer of powers was settled, it depended directly on the Provincial Delegation of the Department of Culture. It was inaugurated on 23 September 1983.

The Castillo de la Yedra was acquired for its headquarters; intervention works lasted many years and were even being carried out during its inauguration, although the keep was prepared for exhibitions.

This castle which serves as the museum?s headquarters is the most important architectural monument in Cazorla. This monument, known since the 17th century as the Castillo de la Yedra, is also called "Castillo de las Cuatro Esquinas", a name which it gets from the floor in the main tower, which is square (cuatro esquinas), with angles directed towards the four cardinal points.

The castle is mainly formed by three enclosures, although there are also barbican remains, remains of walls and other defensive elements outside the present enclosure.

The highest enclosure incorporates the keep, with a square floor, which has four levels and a crenellated terrace, the lowest level corresponding to the dungeons, to which the "Leyenda de la Tragantía" is associated.

In the south-east angle of the tower, there is a construction with three floors of unequal size, previously used as a dwelling and through which the museum is currently accessed. The other two enclosures, included within the castle walls, do not contain buildings, but do have gardens.

There is no exact information about their construction, although the scholars seem to agree that the Arabs must have built it on top of the remains of a primitive Roman fortress. The current castle is from the age of the Archbishop Pedro Tenorio, who restored and extended the fortress in 1394, a time in which the border was threatened by the kingdom of Granada at the height of the reign of Muhammad V. Its importance in the wars and skirmishes between the Arabs and Castilian is proven by documents.

During the 12th and 13th centuries, it lost its strategic and military importance and only became significant again in the War of Independence.

During the confiscation of the 19th century, it became the property of the municipality. Through a sale, it passed into the hands of Mr. Mariano Extremera, a local politician, and was later sold again to Mr. Miguel Marín, whose wife left it to the Marín García Foundation, run by the Mercedarian Sisters. Finally, in 1972, it was expropriated and restored to install the Museum of Popular Arts and Traditions there.