Graph neural networks (GNNs) have been shown to be extremely flexible and accurate in predicting the physical properties of molecules and crystals. However, traditional invariant GNNs are not compatible with directional properties, which currently limits their usage to the prediction of only invariant scalar properties. To address this issue, here we propose a general framework, i.e., an edge-based tensor prediction graph neural network, in which a tensor is expressed as the linear combination of the local spatial components projected on the edge directions of clusters with varying sizes. This tensor decomposition is rotationally equivariant and exactly satisfies the symmetry of the local structures. The accuracy and universality of our new framework are demonstrated by the successful prediction of various tensor properties from first to third order. The framework proposed in this work will enable GNNs to step into the broad field of prediction of directional properties.