Urban land use information plays an important role in urban management, government policy-making, and population activity monitoring. However, the accurate classification of urban functional zones is challenging due to the complexity of urban systems. Many studies have focused on urban land use classification by considering features that are extracted from either high spatial resolution (HSR) remote sensing images or social media data, but few studies consider both features due to the lack of available models. In our study, we propose a novel scene classification framework to identify dominant urban land use type at the level of traffic analysis zone by integrating probabilistic topic models and support vector machine. A land use word dictionary inside the framework was built by fusing natural-physical features from HSR images and socioeconomic semantic features from multisource social media data. In addition to comparing with manual interpretation data, we designed several experiments to test the land use classification accuracy of our proposed model with different combinations of previously acquired semantic features. The classification results (overall accuracy = 0.865, Kappa = 0.828) demonstrate the effectiveness of our strategy that blends features extracted from multisource geospatial data as semantic features to train the classification model. This method can be applied to help urban planners analyze fine urban structures and monitor urban land use changes, and additional data from multiple sources will be blended into this proposed framework in the future.