The original building was a Baptist church. It burned in 1862 and was rebuilt.
President Lincoln was assassinated in the auditorium in 1865.
Later in the 19th century, the War Department purchased the building and used it as an office building/storage facility. Part of it collapsed in the 1890s, killing 22 people.
It sat vacant for many years and was in danger of being condemned when it was purchased and restored, reopening in January of 1968.