The influence of phenolic compound extracts from three colored rice cultivars on the gut microbiota was investigated. The results revealed that protocatechuic acid, chlorogenic acid, caffeic acid and p-coumaric acid were the major metabolites after gut microbiota fermentation. The presence of phenolic compounds led to a significantly decreased ratio of Firmicutes and Bacteroidetes, while the abundance of Proteobacteria decreased. At the genus level, phenolic compounds promoted an increase of Prevotella, Megamonas and Bifidobacterium, while the abundance of Bacteroides and Escherichia–Shigella was inhibited. The concentration of ferulic acid and syringic acid was positively correlated with Bifidobacterium, while Megamonas was positively correlated with catechin and caffeic acid. The abundance of Escherichia–Shigella and Citrobacter was found to be significantly negatively correlated with chlorogenic acid. More importantly, this study revealed that the presence of phenolic compounds generated more propionate, followed by acetate, but not butyrate after gut microbiota fermentation.