Due to a lack of data and practical models, few studies have extracted tailings pond margins in large areas. In addition, there is no public dataset of tailings ponds available for relevant research. This study proposed a new deep learning‐based framework for extracting tailings pond margins from high spatial resolution (HSR) remote sensing images by combining You Only Look Once (YOLO) v4 and the random forest algorithm. At the same time, we created an open source tailings pond dataset based on HSR remote sensing images. Taking Tongling city as the study area, the proposed model can detect tailings pond locations with high accuracy and efficiency from a large HSR remote sensing image (precision = 99.6%, recall = 89.9%, mean average precision = 89.7%). An optimal random forest model and morphological processing were utilized to further extract accurate tailings pond margins from the target areas. The final map of the entire study area was obtained with high accuracy. Compared with the random forest algorithm, the total extraction time was reduced by nearly 99%. This study can be beneficial to mine monitoring and ecological environmental governance.