Heavy metals or metalloids (HMs) in agricultural soils at high geological background areas could inherit the characteristics of parent materials, which would affect HM accumulation in crops. However, the specific impact of different parent materials on the behavior of HMs was unclear. This study focused on the bioavailability and risk variations of HMs in paddy soils derived from eluvial and alluvial parent materials in the Xitiaoxi catchment based on the total concentration, fraction, and risk assessment of HMs. The results showed that the paddy soil in the alluvial area had a higher bioavailability and risk level of HMs than those in the eluvial area, due to the increased proportion of activated HMs in the alluvial paddy soil. The increase in activated HMs was associated with a decrease in soil acidity and fertility as well as intensified anthropogenic activities within the alluvial study area. The spatial distribution of HM risk in soil samples demonstrated that the risk level of HMs was related to the exposure of black shale in eluvial soil, while the risk level of HMs in alluvial soil was negatively correlated with distance from the riverside. The source apportionment results further indicated that the alluvial paddy soil had lower geological but higher anthropogenic sources, especially for Cd, Pb, and Cu. In summary, the alteration in the properties of parent materials during river transportation and more anthropogenic sources raised the bioavailability and risk of HMs in the alluvial paddy ecosystem. This study provided new insights into HM risk assessment and control in agricultural soils developed from different parent materials within a catchment with a high geological background.