Abstract: As an important aspect of acoustic environment perception research, the audio-visual interaction has been put under extensive studies from the quantitative perspective. Only a few qualitative studies have revealed the audio-visual interaction mechanism. Taking the soundscape of Wulingyuan World Natural Heritage Site as the object, this research, based on the conclusion of “low audio-visual matching in semi-natural environment” obtained in the previous studies, selects the semi-natural soundscape with low audio-visual matching accuracy to obtain 72 effective samples via audio-visual experiments. It analyzes the audio-visual interaction mechanism based on the grounded theory. The results reveal: 1) The audio-visual matching is related to the advantage perception. In audio-visual conformity, the subject’s hearing is more likely to guide the vision. In the case of audio-visual inconsistency, the vision is more likely to guide hearing. 2) Visual judgment affects acoustic evaluation. For the noisy-quiet emotional perception in particular, visual judgment is more important than auditory judgment. 3) The sound source type is an element to judge the environmental authenticity. The breadth of visual elements and the distance of sounds are also complementary factors to judge environmental authenticity. Based on these findings, this research puts forward suggestions on the management of the existing acoustic and visual environments in the Wulingyuan World Natural Heritage Site and similar protected areas.