Abstract: From the angle of social environment, combined with literature research, field investigation and logic deduction, the paper analyzes the changes and inner impetus of Yiyang county Dieshan Academy garden. From the Yuan Dynasty to the Qing Dynasty, in aspect of transitional form, phase-ground of Dieshan Academy garden moved from mountain forest to city resort, the spatial distribution changed from the architecture space with single axis featuring front sacrifice hall and back teaching rooms to the garden space with double axis and lots of courtyards, and the culture transformed from Confucianism dominance to coexistence of various cultures. In aspect of changing impetus, the political environment and economic support were the foundation of the academy garden construction development. The attitude of the government decided the academy garden survival and its social significance. The economic support had significant influence on its maintenance and officialization. The cultural environment was influenced by political environment, which allowed the continuation of the Song Confucianism if inclusive, or banned free style of literature if harsh. Therefore, the cultural environment played a key role in the site selection, spatial layout and cultural establishment of the academy garden.