Spatial position of an object can be represented in the human brain based on two types of reference frames: allocentric and egocentric. The perception/action hypothesis of the ventral/dorsal visual stream proposed that allocentric reference frame codes object positions relative to another object/background subserving conscious perception of the external world while egocentric reference frame codes object positions relative to the observer's body/body parts subserving goal-directed actions towards the objects. In three experiments of the present study, by asking congenitally deaf participants and hearing controls to perform allocentric and egocentric judgment tasks on the same stimulus set and by using the spatial congruency effect between allocentric and egocentric positions of the same target object to indicate the extent of influences between the two frames, we aimed to investigate whether the two frames and the potential interaction between them are altered after early deafness. Our results suggested that deaf participants' responses were significantly slower in the egocentric tasks as compared to hearing controls while the two groups showed comparable task performance in the allocentric tasks, indicating that egocentric reference frame was impaired after early deafness. Moreover, the pattern of interaction between the two frames was different between deaf and hearing groups: irrelevant egocentric positions caused more interference to allocentric processing than vice versa in the hearing group while the two frames caused equivalent interference to each other in the deaf group. Further control experiments suggested that the above effects were not caused by the impaired sense of balance in the congenitally deaf participants (via open-loop pointing test), and was independent of whether the speed of allocentric and egocentric processing was equivalent or not in the hearing group.