Magnetite (Fe₃O₄) nanoparticles grafted with thiol groups through silanization of (3-mercaptopropyl) trimethoxy-silane (Fe₃O₄-SH) were used for preparing vinyl ester resin (VER) nanocomposites with different Fe₃O₄ nanoparticle loading levels. The calculated thiol group graft percentage of only ∼2.25% did show a significant effect on the physicochemical properties of VER nanosuspensions and nanocomposites. The exothermal curing peak temperature was shifted from 80 °C for pure VER to 75 °C for liquid VER with 10 wt% Fe₃O₄-SH. The tensile strength (72.6 MPa) of 5 wt% Fe₃O₄-SH reinforced nanocomposites was increased by 26.3% as compared with that (57.5 MPa) of pure VER composites. However, the tensile strength (62.7 MPa) of 5 wt% Fe₃O₄ was just increased by 9.0%. The polymer matrix and surface coating were observed to have negligible effects on the magnetic properties of nanoparticles, exhibiting superparamagnetic behaviors.