In 1870 General Sylvanus Thayer proposed to establish a public library within the town. Although they had rejected previous propositions from him, the Town accepted this one. Thayer, along with Asa French and Francis A. Hobart purchased a site for the library from Nathaniel M. Hobart in early 1871. This land was directly adjacent to the newly constructed Town Hall.
Thayer personally loaned the town $10,000 to see the project through. Unfortunately Thayer passed away in 1872, but the inheritors of his estate saw the library through to completion. The Victorian style building built in brick and stone was completed in 1874. It was designed by Boston architect Hammet Billings.