In this study, three polysaccharides including water-soluble polysaccharide (WSP), dilute alkali-soluble polysaccharide (DASP) and concentrated alkali-soluble polysaccharide (CASP) were extracted from purple sweet potatoes and then administered to normal and cyclophosphamide (CTX) treated mice by gavage. The results showed that WSP and CASP could restore the spleen index and immune cytokine (IL-2 and IL-6) levels in CTX treated mice, while DASP could enhance the levels of TNF-α, IL-2 and IL-6. As compared to the normal control group, WSP and CASP treatment groups exhibited increased levels of Bacteroidetes, Lachnospiraceae and Oscillospira, but decreased levels of Firmicutes, Alcaligenaceae and Sutterella in normal mice. When compared with a model control group, all the three polysaccharide treatment groups showed relatively higher abundances of Bacteroidetes, Ruminococcaceae, Lachnospiraceae, Ruminococcus and Oscillospira but lower levels of Firmicutes, Proteobacteria, Alcaligenaceae and Sutterella in CTX treated mice. Moreover, all the polysaccharides could enhance the production of acetic acid, propionic acid and butyric acid in normal mice, while WSP could upregulate the production of these short chain fatty acids in CTX treated mice.