In this study, we investigated the size and photoluminescence (PL) evolution of CdTe nanoparticles (NPs) in different polymer media under thermal annealing. A quick growth and maintenance of strong PL were observed. By analyzing the transmission electron microscopy (TEM) images of NPs in polymer media, we discovered that the size evolution of NPs was the combination of Ostwald ripening and dynamic coalescence throughout the growth process. Moreover, the experimental results also revealed that the nature of polymers determined the dynamic coalescence of NPs from three aspects; the mobility of polymer chains, the compatibility of NPs with polymer media, and the interaction between NPs and polymer network. Thus by altering the glass transition temperature (T_g) of polymers, the molar mass (M_n) of the polymers, the phase separation of NPs in polymer media, as well as the interaction between NPs and polymers, the growth rate of NPs was controllable.