Knowing the physical properties and scattering mechanisms contributes to Synthetic Aperture Radar (SAR) image interpretation. For single-polarized SAR data, however, it is difficult to extract the physical scattering mechanisms due to lack of polarimetric information. Time-frequency analysis (TFA) on complex-valued SAR image provides extra information in frequency perspective beyond the ’image’ domain. Based on TFA theory, we propose to generate the sub-band scattering pattern for every object in complex-valued SAR image as the physical property representation, which reveals backscattering variations along slant-range and azimuth directions. In order to discover the inherent patterns and generate a scattering classification map from single-polarized SAR image, an unsupervised hierarchical deep embedding clustering algorithm based on time-frequency analysis (HDEC-TFA) is proposed to learn the embedded fea- tures and cluster centers simultaneously and hierarchically. The polarimetric analysis result for quad-pol SAR images is applied as reference data of physical scattering mechanisms. In order to compare the scattering classification map obtained from single- polarized SAR data with the physical scattering mechanism result from full-polarized SAR, and to explore the relationship and similarity between them in a quantitative way, an information theory based evaluation method is proposed. We take Gaofen- 3 quad-polarized SAR data for experiments and the results and discussions demonstrate that the proposed method is able to learn valuable scattering properties from single-polarization complex- valued SAR data, and to extract some specific targets as well as polarimetric analysis. At last, we give a promising prospect to future applications.