A huge effort has been applied in image classification to create high-quality thematic maps and to establish precise inventories about land cover use. The peculiarities of remote sensing images (RSIs) combined with the traditional image classification challenges made RSI classification a hard task. Our aim is to propose a kind of boost-classifier adapted to multiscale segmentation. We use the paradigm of boosting, whose principle is to combine weak classifiers to build an efficient global one. Each weak classifier is trained for one level of the segmentation and one region descriptor. We have proposed and tested weak classifiers based on linear support vector machines (SVM) and region distances provided by descriptors. The experiments were performed on a large image of coffee plantations. We have shown in this paper that our approach based on boosting can detect the scale and set of features best suited to a particular training set. We have also shown that hierarchical multiscale analysis is able to reduce training time and to produce a stronger classifier. We compare the proposed methods with a baseline based on SVM with radial basis function kernel. The results show that the proposed methods outperform the baseline.