Heavily doped oxide semiconductors can be tailored for widespread application in near-infrared (NIR) and mid-infrared (mid-IR) wavelength ranges because of both functional and fabrication advantages. Here, the ultrafast and broadband nonlinear saturable absorption of Al-doped zinc oxide nanocrystals (AZO NCs) is investigated by using the Z-scan technique and the pump–probe technique. The nonlinear absorption coefficient is as high as −1.90 × 10³ cm GW⁻¹ in the wide infrared (IR) wavelength range (from 800 to 3000 nm). Furthermore, a maximum optically induced refractive index of −1.85 × 10⁻¹ cm² GW⁻¹ in the dielectric region and 2.09 × 10⁻¹ cm² GW⁻¹ in the metallic region leads to an ultrafast nonlinear optical response (less than 350 femtoseconds). Mode-locked fiber lasers at 1064 nm and 1550 nm as well as Q-switched fiber lasers near 2000 nm and 3000 nm prove the use of employing AZO NCs as a broadband and ultrafast nonlinear optical device, which provides a valuable strategy and intuition for the development of nanomaterial-based photonic and optoelectronic devices in the NIR and mid-IR wavelength ranges.