Traditionally, a cut-test is used to assess the cocoa fermentation degree for a quality control aims. However, this method is subjective and presents several drawbacks. In this paper, a reliable machine vision system was proposed to automatically identify and classify cocoa beans (Theobroma cacao L.). The approach developed in this study uses color features and a support vector machine-based method for cocoa beans classification according to the fermentation degree. To outline this approach, firstly, images were acquired, and beans were separately identified from the background. After that, color features were extracted in each component of RGB, HSV and YCbCr color spaces and were used to describe cocoa beans fermentation degree. Then, a selection procedure of the best cocoa beans descriptor combination was developed. Finally, SVM model was built to discriminate unfermented, partly fermented and well fermented cocoa beans. This model was 10-fold cross-validated to ensure its stability. Using selected descriptors, our approach had a discrimination rate of 100% in both training and prediction set. The results show that, machine vision system coupled with SVM model can rapidly, accurately, and reliably discriminate cocoa beans according to the fermentation degree compared to the traditional classification methods.