Upconverting nanoparticles (UCNPs) compose a class of luminescent materials that utilize the unique wavelength-converting properties of lanthanide (Ln) ions for light-harvesting applications, photonics technologies, and biological imaging and sensing experiments. Recent advances in UCNP design have shed light on the properties of local color centers, both intrinsic and controllably induced, within these materials and their potential influence on UCNP photophysics. In this review, we describe fundamental studies of color centers in Ln-based materials, including research into their origins and their roles in observed photodarkening and photobrightening mechanisms. We place particular focus on the new functionalities that are enabled by harnessing the properties of color centers within Ln-doped nanocrystals, illustrated through applications in afterglow-based bioimaging, X-ray detection, all-inorganic nanocrystal photoswitching, and fully rewritable optical patterning and memory.