Aging comes along with reduced gray matter (GM) volume in several cerebral areas and with cognitive performance decline in different cognitive domains. Moreover, regional GM volume is linked to specific cognitive sub processes in older adults. However, it remains unclear which regional changes in older individuals are directly associated with decreased cognitive performance. Moreover, most of the studies on this topic focused on hippocampal and prefrontal brain regions and their relation to memory and executive functioning. Interestingly, there are only a few studies that reported an association between striatal brain volume and cognitive performance. This is insofar surprising that striatal structures are (1) highly affected by age and (2) involved in different neural circuits that serve intact cognition. To address these issues, voxel-based morphometry (VBM) was used to analyze GM volume in 18 younger and 18 older adults. Moreover, several neuropsychological tests from different neuropsychological test batteries were applied to assess a broad range of cognitive domains. Older adults showed less GM volume than younger adults within frontal, striatal, and cerebellar brain regions. In the group of older adults, significant correlations were found between striatal GM volume and memory performance and between prefrontal/temporal GM volume and executive functioning. The only direct overlap between brain regions associated with regional atrophy and cognitive performance in older adults was found for the right caudate: older adults showed reduced caudate volume relative to younger adults. Moreover, caudate volume was positively correlated with associative memory accuracy in older adults and older adults showed poorer performances than younger adults in the respective associative memory task. Taken together, the current findings indicate the relevance of the caudate for associative memory decline in the aging brain.