Long-context deep neural networks (DNNs) have shown significant effectiveness in natural language processing and video applications, etc., becoming a prevailing trend. However, they introduce extremely large intermediate tensors, resulting in substantial memory overhead. While considerable efforts have been made to optimize DNNs, insufficient awareness of tensor properties has hindered effective memory optimization and may lead to inefficient computations in the long context scenario. In this paper, we present TA, a DNN optimization system that reduces memory overhead and improves inference performance by exploiting tensor properties. We first extract and identify essential tensor properties from the computation graph, such as reduce dependency and broadcastability, then apply various optimizations involving transformations and kernel mapping based on these properties. Experiments conducted on seven models demonstrate that TA achieves speedups of 1.50× and 3.24× on average for end-to-end and core module performance, respectively, compared to eight state-of-the-art works, on H100 (1.86× and 3.70× on A100).