Image pattern classification is considered a significant step for image and video processing. Although various image pattern algorithms have been proposed so far that achieved adequate classification, achieving higher accuracy while reducing the computation time remains challenging to date. A robust image pattern classification method is essential to obtain the desired accuracy. This method can be accurately classify image blocks into plain, edge, and texture (PET) using an efficient feature extraction mechanism. Moreover, to date, most of the existing studies are focused on evaluating their methods based on specific orthogonal moments, which limits the understanding of their potential application to various Discrete Orthogonal Moments (DOMs). Therefore, finding a fast PET classification method that accurately classify image pattern is crucial. To this end, this paper proposes a new scheme for accurate and fast image pattern classification using an efficient DOM. To reduce the computational complexity of feature extraction, an election mechanism is proposed to reduce the number of processed block patterns. In addition, support vector machine is used to classify the extracted features for different block patterns. The proposed scheme is evaluated by comparing the accuracy of the proposed method with the accuracy achieved by state-of-the-art methods. In addition, we compare the performance of the proposed method based on different DOMs to get the robust one. The results show that the proposed method achieves the highest classification accuracy compared with the existing methods in all the scenarios considered.