A method to measure thermal diffusivity around a single heated gold nanoparticle is presented. It is based on photothermal single particle microscopy and employs the phase delay of temperature modulation due to finite thermal diffusivity. The phase delay is detected optically averaging over the focal volume of a diffraction limited beam of light. Thermal diffusivity is extracted by comparison to electromagnetic scattering calculations of the photothermal signal. Measurements in the solid (polymer) and liquid (water) are presented and compare well with literature data. The method paves the way for extended measurements of non-diffusive and heterogeneous heat transport in complex media.