Solid-state batteries have gained increasing attention with the discovery of new inorganic solid electrolytes, some of which rival the ionic conductivity of liquid electrolytes. With the additional benefit of being single-ion conductors, several inorganic solid electrolytes achieve the lithium ion conduction required for solid-state batteries to become the next generation of energy storage device in combination with lithium metal. However, the challenges faced when preparing thin layers and stable interfaces of solely inorganic and brittle materials limit the performance of lithium solid-state batteries that are made purely of inorganic materials. Therefore, the best-performing solid-state batteries also introduce polymers to the system to improve the interfaces, cohesion, manufacture and mechanical properties of the cell as a whole. This article highlights recent developments made with the combination of polymer and inorganic materials in the form of composite electrolytes, interlayers, protective coatings and binders. The role of polymers regarding interface chemistry, interface resistance and lithium transfer is discussed and the importance of polymers for the processing of solid-state batteries is described. Taken as a whole, the article surveys the relevance of polymers at each cell component and discerns how polymers may provide the key to access the full potential of solid-state batteries with inorganic solid electrolytes.