Hippocampal volume shrinks in late adulthood, but the neuromolecular factors that trigger hippocampal decay in aging humans remains a matter of speculation. In rodents, brain-derived neurotrophic factor (BDNF) promotes the growth and proliferation of cells in the hippocampus and is important in long-term potentiation and memory formation. In humans, circulating levels of BDNF decline with advancing age, and a genetic polymorphism for BDNF has been related to gray matter volume loss in old age. In this study, we tested whether age-related reductions in serum levels of BDNF would be related to shrinkage of the hippocampus and memory deficits in older adults. Hippocampal volume was acquired by automated segmentation of magnetic resonance images in 142 older adults without dementia. The caudate nucleus was also segmented and examined in relation to levels of serum BDNF. Spatial memory was tested using a paradigm in which memory load was parametrically increased. We found that increasing age was associated with smaller hippocampal volumes, reduced levels of serum BDNF, and poorer memory performance. Lower levels of BDNF were associated with smaller hippocampi and poorer memory, even when controlling for the variation related to age. In an exploratory mediation analysis, hippocampal volume mediated the age-related decline in spatial memory and BDNF mediated the age-related decline in hippocampal volume. Caudate nucleus volume was unrelated to BDNF levels or spatial memory performance. Our results identify serum BDNF as a significant factor related to hippocampal shrinkage and memory decline in late adulthood.