Polydimethylsiloxane (PDMS) sponges with nanofiller modification are gaining considerable attention because of their extraordinary performances in improving the qualities of polymer nanocomposites. However, the current synthesis methods limit the flexibility and stability of functional PDMS sponges. Herein, we present an ultrasound-assisted in situ polymerization approach to convert a novel clay-like PDMS nanocomposite into a functional PDMS sponge. Accessible ultrasound with low intensity can moderately regulate the cross-linking degree of PDMS oligomer to obtain only shape-controllable clay-like units for the scalable synthesis of a PDMS sponge with a high elasticity. In order to enhance the functionality, graphene nanosheets serve as a nanofiller uniformly distributed into the three-dimensional porous PDMS skeleton. The graphene-reinforced PDMS sponge (GRPS) exhibits mechanical stability and wettability, and is a better alternative to typically recyclable adsorbents in an oil–water separation system. Thus, the present synthesis method may pave a facile way for manufacturing functional PDMS sponges on a large scale.