Point defects can be used to tailor the properties of semiconductors, but can also have undesired effects on electronic and thermal transport, particularly in ultrascaled nanostructures, such as nanowires. Here we use all-atom molecular dynamics to study the effect that different concentrations and spatial distributions of vacancies have on the thermal conductivity of Si nanowires, overcoming the limitations of previous studies. Although vacancies are not as effective as the nanovoids found in e.g. porous Si, they can still reduce the thermal conductivity in ultrathin Si nanowires by more than a factor of two, when found in concentrations smaller than 1%. We also present arguments against the so-called self-purification mechanism, which is sometimes suggested to take place and proposes that vacancies have no influence on transport phenomena in nanowires.