Limited knowledge is available on the relationship between antigen-specific immune responses and COVID-19 disease severity. We completed a combined examination of all three branches of adaptive immunity at the level of SARS-CoV-2-specific CD4+ and CD8+ T cell and neutralizing antibody responses in acute and convalescent subjects. SARS-CoV-2-specific CD4+ and CD8+ T cells were each associated with milder disease. Coordinated SARS-CoV-2-specific adaptive immune responses were associated with milder disease, suggesting roles for both CD4+ and CD8+ T cells in protective immunity in COVID-19. Notably, coordination of SARS-CoV-2 antigen-specific responses was disrupted in individuals ≥ 65 years old. Scarcity of naive T cells was also associated with aging and poor disease outcomes. A parsimonious explanation is that coordinated CD4+ T cell, CD8+ T cell, and antibody responses are protective, but uncoordinated responses frequently fail to control disease, with a connection between aging and impaired adaptive immune responses to SARS-CoV-2.