Cross-modal Sentiment Information Expression of Voice Source Characteristics using Image Texture Features

Abstract

Following the successful findings of high correlations between speech and color such as F0 and Value, Loudness and Saturation and Spectrum and Hue, we analyzed the correlations between voice source characteristics and the image parameters showing textural differences in this paper for better scientific understanding of their correlations and effective use in visualization of speech information. Through sentiment association experiments, we could have observed high positive correlations between H1*- H2* (amplitude difference between first and second harmonics corrected for vocal tract effects), H1-A1 (amplitude difference between first harmonic and first formant) and Contrast, high negative correlations between H1*-H2*, H1-A1, H1-A2, H1-A3, Harmonicto- Noise Ratio (HNR) in 0 to 3500Hz frequency band and Variance, Prominence and negative correlations between H1*-A3*, HNR in 0 to 500 Hz and Prominence. These results show the possibility of direct visualization of speech characteristics which cannot be effectively carried out by conventional mapping using discrete language expressions.