Spectral imaging of pharmaceutical material using a compact ultra-broadband (1–4 THz) terahertz semiconductor source was demonstrated. False-color RGB images could be obtained using a simple procedure (calibration free). The ability to distinguish the polymorphism of carbamazepine (CBZ), the hydrate forms of D-(+)-glucose and caffeine, and the crystallinity of nifedipine was demonstrated using the THz DFG source. Crystal forms of pharmaceutical materials can be distinguished using this method.