Colloidal self-assembly is a viable solution to making advanced metamaterials. While the physicochemical properties of the particles affect the properties of the assembled structures, particle configuration is also a critical determinant factor. Colloidal self-assembly state classification is typically achieved with order parameters, which are aggregate variables normally defined with nontrivial exploration and validation. Here, we present an image-based framework to classify the state of a 2-D colloidal self-assembly system. The framework leverages deep learning algorithms with unsupervised learning for state classification and a supervised learning-based convolutional neural network for state prediction. The neural network models are developed using data from an experimentally validated Brownian dynamics simulation. Our results demonstrate that the proposed approach gives a satisfying performance, comparable and even outperforming the commonly used order parameters in distinguishing void defective states from ordered states. Given the data-based nature of the approach, we anticipate its general applicability and potential automatability to different and complex systems where image or particle coordination acquisition is feasible.