In order to investigate the environmental pollution of heavy metals in coal gangue, samples of the coal gangue and shallow groundwater in surrounding areas were collected in Gongyi City, Henan Province. The coal gangue samples were treated by leaching experiment first. Then the contents of metallic elements in groundwater and treated coal gangue leaching water samples were measured by inductively coupled plasma mass spectrometry (ICP-MS). The results showed that the contents of Cr and Pb were high in coal gangue, and the amount of the metallic elements increased as the coal gangue stacking over time with variable dissolution trends. The contents of most metallic elements in shallow groundwater were higher in the collapsed area than in the non-collapsed area, except for Cr and Sr. The contents of Cr and Cu in shallow groundwater samples from collapsed area, and Cr from non-collapsed area were high, as 17.6, 59.4 and 22.8 g/L, respectively. Based on the leaching characteristics, and comparison between the contents of metallic elements in the collapsed and non-collapsed areas, we suggested that the high contents of Cu and Pb in shallow groundwater in collapsed area were closely related to the coal gangue dump.