The area of Panzano was already inhabited in Etruscan times as evidenced by the discovery of a stele dating from the 6th-5th centuries B.C. near the parish church of San Leolino, a stele later dispersed. Even in Roman times the area was densely inhabited; numerous traces of that period remain in place names including Panzano itself. In the early 10th century the Pieve di San Leolino in Flacciano, later to become Panzano, is mentioned.

In the 12th century there is the first evidence of the name Panzano, which is mentioned in the plebe Sancti Leolini sitam in Panzano, while in the tithes of the 13th century the church of Santa Maria located in the castle is also mentioned. The castle of Panzano certainly had already developed before the 12th century and was among the possessions of the Firidolfi family.

Of the historical events of the castle there are not many traces left. In the mid-13th century when the Florentine countryside was organized into leagues Panzano was included in the League of the Val di Greve. After the Battle of Montaperti in 1260 the castle was sacked and had two towers destroyed by the victorious Ghibelline troops. During the war that opposed Florence with the Visconti of Milan Panzano was occupied and again sacked by Alberico da Barbiano’s troops in 1397.

In 1478, the Sienese troops and their allies, the troops of the King of Naples Ferdinand I of Aragon invaded Chianti for the second time. On that occasion the castle of Panzano was one of the most important bulwarks in defense of the Republic of Florence so much so that it became the seat of the Commissioner of the Republic.