Abstract: This paper studies the classification of buildings by using object-oriented remote sensing image information extraction method. Five medium-sized rural buildings in different areas of Zhangjiakou were selected as research objects. According to the building characteristic information, the building is extracted from the remote sensing image, and the buildings are classified according to the results of information extraction and adjustment, and the classification results are compared with the survey results. After the comparison, the classification accuracy of buildings reached more than 80%, meeting the needs for the classification of rural buildings in Zhangjiakou area. It can be used to assist in the field research of building structure types, which can effectively improve the efficiency of the research and serve the research on seismic fortification of buildings.