Garden writing and painting is an important means to continue the material existence of garden culture; needless to say, it also reflects the surplus spiritual and cultural value of garden culture. Wu Liang, a literati in the Ming Dynasty, lived in seclusion in Changzhou’s Zhiyuan Garden, and constructed a unique imagination of peach-blossom springs with his own life experience, thus shifting the cultural image of peach-blossom springs from philosophical thought to literary paradigm, and then to the physical garden landscape. Through the interaction between Zhiyuan Garden Paintings by Zhang Hong and Zhiyuan Garden Collections written by Wu Liang and from the simple and wild style of Wu Liang’s imagination of peach-blossom springs, it can be seen that Zhiyuan Garden built by Wu Liang was different from the luxurious and noble gardens of his times. In this sense, Wu Liang’s seclusion in Zhiyuan Garden not only conformed to the shift and development of the seclusion culture and the social trends of building gorgeous gardens among the elite class, but also was his helpless action to avoid potential disasters caused by factional strife. In addition, the literati’s love of landscape and sentiment of filial obedience are also important factors that prompted Wu Liang to live in seclusion. Therefore, the construction of Zhiyuan Garden is not a manifestation of Wu Liang’s desperation to the world but his determination to return from an earthly life to a life in seclusion. Accordingly, his retirement and seclusion reflected a literati’s sense of responsibility and self-restraint instead of a good reputation for seclusion.