Colloidal gold nanostructures are nowadays widely involved in sensor applications. One of the most interesting techniques that takes advantage of them is certainly the Surface Enhanced Raman Scattering (SERS) effect, even if it is often considered a tricky technique due to structural constraints required by the nanostructured substrates to obtain high enhancement factors (EFs), i.e. the presence of hot spots. Because of the easy preparation and high number of hot spots, aggregated gold nanospheres seem to be the most efficient through the SERS colloids, but their characteristic high disorder makes them unpredictable and difficult to compare between different batches. For this reason, less SERS effective, but more regular and organized substrates are usually preferred. In this study, a method based on Boundary Element Method (BEM) simulation is used to accurately predict the colloidal SERS EFs of gold nanoparticle (AuNP) aggregates, starting from their experimental extinction spectra. Surprisingly, it was found that larger aggregates do not exhibit stronger hot spots, but rather higher amounts of them, influencing the overall predicted EFs, which well reflect the results obtained experimentally.