Amphiphilic polymers can self-assemble to form nanoparticles with different structures under suitable conditions. Polymer nanoparticles functionalized with aromatic azo groups are endowed with photo-responsive properties. In recent years, a variety of photoresponsive polymers and nanoparticles have been developed based on azobenzene, using different molecular design strategies and synthetic routes. This article reviews the progress of this rapidly developing research field, focusing on the structure, synthesis, assembly and response of photo-responsive polymer assemblies. According to the molecular structure, photo-responsive polymers can be divided into linear polymers containing azobenzene in a side chain, linear polymers containing azobenzene in the main chain, linear polymers containing azobenzene in an end group, branched polymers containing azobenzene and supramolecular polymers containing azobenzene. These systems have broad biomedical application prospects in the field of drug delivery and imaging applications.