The mission of Liberty Hall Historic Site is to educate the public by interpreting the life and times of Senator John Brown and his descendants.  The site will collect, document, preserve, conserve, and exhibit the social, domestic, and horticultural aspects of Kentucky life, beginning with 1796 at Liberty Hall and 1835 at the Orlando Brown House.

Liberty Hall Historic Site is a non-profit organization owned and operated by Liberty Hall, Inc., and The National Society of the Colonial Dames of America in the Commonwealth of Kentucky (NSCDA-KY).

The history of Liberty Hall Historic Site can be traced back to 1786, when General James Wilkinson purchased much of the land that is downtown Frankfort. Wilkinson laid out the town of Frankfort, naming the streets for friends, famous people and places, and even himself. Some of Wilkinson's original streets, Wilkinson, Wapping, and Montgomery (now Main) form the boundaries of three of the four sides of Liberty Hall Historic Site. Wilkinson ultimately sold the tract that includes Liberty Hall to Frankfort resident Andrew Holmes. In 1796,  Holmes sold the four acres to Senator John Brown.

John Brown began construction of a home on the property shortly after purchasing it, though he was often away in Philadelphia. The architect of Liberty Hall is unknown (John himself may have done some of the design) but it is clear that someone with great skill and understanding designed the Federal style home. One of the earliest brick homes in Frankfort, the bricks were fired locally from clay dug from the cellar. The construction continued until 1800 when the house was substantially complete, lacking only the glass windows, which were added in 1804. In 1801, John Brown moved into the home with his wife, Margaretta Mason Brown, and infant son Mason.

In addition to the main house, John Brown built several dependent structures on the property, including a kitchen and laundry, smokehouse, a privy, stables, carriage house, and slave quarters.  Eventually a set of stairs was installed from the garden level to the river level and a boat landing was created. Ultimately, a fence was built around the property.  After this, Liberty Hall remained substantially unchanged until the early 1830's.

In order to give his two sons equal inheritance, in 1835 John Brown divided his property. His elder son, Mason, would inherit Liberty Hall. For his younger son, Orlando, Brown hired Gideon Shryock, designer of the Kentucky Capitol, to design a new house. Constructed Greek Revival style, the Orlando Brown House was built by local contractor Harrison Blanton. The entire project cost just $5,000.00.

In the 1870s, Brown family members sold a parcel of land between the two houses. William Chinn built a house on the property, which was later sold to the Sutterlin family. The Chinn-Sutterlin House stood until the 1960s, when it was razed to reconnect the Liberty Hall and Orlando Brown properties. Three more lots, on the Wapping Street side of the Orlando Brown property, were sold near the turn of the 20th century. These properties remain privately owned and separate from the Liberty Hall Historic Site property.

In 1934, Mary Mason Scott, John Brown's great granddaughter and the last resident of Liberty Hall, passed away leaving Liberty Hall to her brother, John Matthew Scott. He sold Liberty Hall to a group of concerned citizens who had formed Liberty Hall, Inc., a nonprofit organization. They opened the house as a museum in 1937. The Orlando Brown House was occupied until 1955. At her death, Orlando Brown's last remaining descendant, Anne Hord Brown left the house to the National Society of the Colonial Dames of America in the Commonwealth of Kentucky (NSCDA-KY). The Dames opened the house as a museum shortly thereafter.