Today, there is an urgent demand to develop all solid-state lithium-ion batteries (LIBs) with a high energy density and a high degree of safety. The core technology in solid-state batteries is a solid-state electrolyte, which determines the performance of the battery. Among all the developed solid electrolytes, composite polymer electrolytes (CPEs) have been deemed as one of the most viable candidates because of their comprehensive performance. In this review, the limitations of traditional solid polymer electrolytes and the recent progress of CPEs are introduced. The effect and mechanism of inorganic fillers to the various properties of electrolytes are discussed in detail. Meanwhile, the factors affecting ionic conductivity are intensively reviewed. The recent representative CPEs with synthetic fillers and natural clay-based fillers are highlighted because of their great potential. Finally, the remaining challenges and promising prospects are outlined to provide strategies to develop novel CPEs for high-performance LIBs.