A series of different topological nanostructures are fabricated on silicon wafer using metal-assisted chemical etching. The modulation effect of these nanostructures on the size, filopodia generation and growth orientation of the rat mesenchymal stem cells (MSCs) are studied. These topological nanostructures tend to induce the MSCs to have smaller size, but they generate much more filopodia compared to the flat silicon control. The modulation effects of these nanostructures are dependent on their surface convexity, as analysed by grey-level value histogram and fast Fourier transformation (FFT). A surface with a higher portion of convex area is better at supporting larger cell sizes. The wavenumber analysis by FFT further determines its effect on filopodia generation. In addition, the growth orientation of the cells are also guided by the surface convexity. On the porous and spongy surface, the cell filopodia extend and grow in avoidance of large sinking pits. On the columnar and spiny surface, the cell body and filopodia extend only on the tips of these nanostructures. Our study reveals that surface convexity is an important factor modulating cell behavior, and convexity analysis by image processing can work as a fast and simple evaluation standard to design topological nanostructures.