Deep learning has achieved wide success in solving Partial Differential Equations (PDEs), with particular strength in handling high dimensional problems and parametric problems. Nevertheless, there is still a lack of a clear picture on the designing of network architecture and the training of network parameters. In this work, we develop Residual-Informed Neural Networks (RINN) to solve partial differential equations. Compared to the widely used method PINN (Physics-Informed Neural Networks), RINN avoids calculating the high-order derivatives of neural networks, thus significantly reduces the computational cost.