For the Mg–Ni–Y system, a typical Mg-based bulk metallic glass forming system, glass formation compositions are first predicted by thermodynamic calculations based on the extended Miedema’s model, suggesting that metallic glasses in the system are favored over a large composition range. Assisted by ab initio calculations, a realistic Mg–Ni–Y n-body potential is then constructed under a proposed modified tight-binding scheme. Based on the potential, an atomistic modeling scheme is further formulated for designing the favored, and even pinpointing the optimized, compositions at the atomic level. A hexagonal glass formation region is located for the Mg–Ni–Y system, reflecting the possible compositions energetically favoring metallic glass formation. An optimized stoichiometry sub-region is further pinpointed, within which the driving force for glass formation is prominently larger than that outside. The present study evaluates glass formation in the Mg–Ni–Y system from two different perspectives, and the results have implications for the entire family of Mg-based systems. The prediction schemes could be of great help for guiding the composition design of ternary glass formers.