Controlling optical and electronic properties of graphene via substitutional doping is central to many fascinating applications. Doping graphene with boron (B) or nitrogen (N) has led to p- or n-type graphene; however, the electron mobility in doped-graphene is severely compromised due to increased electron-defect scattering. Here, we demonstrate through Raman spectroscopy, nonlinear optical and ultrafast spectroscopy, and density functional theory that the graphitic dopant configuration is stable in graphene and does not significantly alter electron–electron or electron–phonon scattering, that is otherwise present in doped graphene, by preserving the crystal coherence length (L_a).