Neural networks provide a powerful tool for machine learning and other data science techniques. Although there have been significant developments in neural network methods, the analysis of relevant matrices is typically overlooked. In this talk, we aim to show why interesting matrix analysis and computations may be performed for some neural network methods, especially function approximations by shallow ReLU neural networks. We present rigorous analysis for some relevant matrices and show the challenges (in the conditioning and frequency modes) faced by traditional matrix computations and show why it is feasible to design new fast and reliable solvers based on certain underlying structures.