Schools were established in the small communities throughout Van Buren County in mid-to-late 1800s. Beginning in the 1920s, these community schools began to be consolidated into larger school buildings to house the growing instructional needs of the students.

A school in the town of Spencer has existed since the late nineteenth century. In 1936, county officials took over the education of elementary age students in the town of Spencer from Burritt College. The restructured Spencer Elementary school was housed in rented buildings until a permanent building could be secured. Under the leadership of Superintendent J. Bouldin Brady, a new brick Spencer Elementary School was planned for and constructed on the southeastern edge of town. Following the Christmas break of 1949, staff and students moved into the new Spencer Elementary building in January 1950.

Spencer Elementary remained in this building until a new school was constructed that adjoined the high school building. Construction was completed and the new Spencer Elementary School building was opened in 1998.

The old Spencer Elementary building completed in 1950. The building would serve the students of Spencer until a new building was completed across town in 1998. The building would later serve the citizens of Van Buren County as the administrative building housing all of the county offices and historical archives until it was destroyed by fire in January 2015.

The new Spencer Elementary School building completed in 1998.