118 Imavis D 18 00055 Text

Image and Vision Computing Manuscript Draft Manuscript Number: IMAVIS-D-18-00055 Title: Learning Discriminative Subregions and Pattern Orders for Facial Gender Classification Article Type: Full Length Article Keywords: Discriminative subregions; Pattern order; Chain-type SVM; Feature vector selection; Facial gender classification Abstract: Facial image-based gender classification has been widely used in many real-world applications. Most of the existing work, however, focuses on designing sophisticated or specific feature descriptors for the entire face, which neglects the discriminative information carried by facial components and pattern order combinations. To address the issue, in this paper, we first propose a generalized texture operator, i.e., the multi-spatial multi-order interlaced pattern (MMIP) matrix, to represent the gender information possessed by the facial subregions with textural pattern orders. A chain-type support vector machine (CSVM) based feature vector selection schema, is then developed to highlight the gender characteristics. As a result, the discriminative subregions and pattern orders are constructed as the feature representation for facial gender classification. Empirical studies on FRGC 2.0 dataset demonstrate that the proposed method is able to pinpoint the discriminative facial subregions and pattern orders between the female and male groups, leading to a 95.6% (or 96.1%) classification accuracy on symmetrically (or asymmetrically) segmented facial images. Moreover, experimental results on FRGC 2.0 and FERET datasets validate the effectiveness of the proposed MMIP matrix compared to the state-of-the-art feature descriptors including LBP, LDP, IDP and DWT. Finally, extensive experiments on four benchmark datasets (i.e., FRGC 2.0, FERET, LFW and UND) show that the proposed method achieves competitive classification performance compared with seven state-of-the-art baselines. Suggested Reviewers: Buckles Bill Bill.Buckles@unt.edu Cover Letter Dear Editors of Image and Vision Computing: Facial image-based gender classification has been widely used in many real-world applications. Most of the existing work, however, focuses on designing sophisticated or specific feature descriptors for the entire face, which neglects the discriminative information carried by facial components and pattern order combinations. To address the issue, in this paper, we first propose a generalized texture operator, i.e., the multi-spatial multi-order interlaced pattern (MMIP) matrix, to represent the gender information possessed by the facial subregions with textural pattern orders. A chain-type support vector machine (CSVM) based feature vector selection schema, is then developed to highlight the gender characteristics. As a result, the discriminative subregions and pattern orders are constructed as the feature representation for facial gender classification. Empirical studies on FRGC 2.0 dataset demonstrate that the proposed method is able to pinpoint the discriminative facial subregions and pattern orders between the female and male groups, leading to a 95.6% (or 96.1%) classification accuracy on symmetrically (or asymmetrically) segmented facial images. Moreover, experimental results on FRGC 2.0 and FERET datasets validate the effectiveness of the proposed MMIP matrix compared to the state- of-the-art feature descriptors including LBP, LDP, IDP and DWT. Finally, extensive experiments on four benchmark datasets (i.e., FRGC 2.0, FERET, LFW and UND) show that the proposed method achieves competitive classification performance compared with seven state-of- the-art baselines. Therefore, this work is of general interest to the community of Image and Vision Computing, and should be published in an academic journal. Sincerely Zhong Chen, Andrea Edwards, Yongsheng Gao, Kun Zhang Cover Letter Learning Discriminative Subregions and Pattern Orders for Facial Gender Classification Zhong Chena, Andrea Edwardsa, Yongsheng Gaob, Kun Zhanga,∗ aDepartment of Computer Science, Xavier University of Louisiana, LA 70125, USA bSchool of Engineering, Griffith University, QLD 4111, Australia Abstract Facial image-based gender classification has been widely used in many real-world applications. Most of the existing work, however, focuses on designing sophisticated or specific feature descriptors for the entire face, which neglects the discriminative information carried by facial components and pattern order combinations. To address the issue, in this paper, we first propose a generalized texture operator, i.e., the multi-spatial multi-order interlaced pattern (MMIP) matrix, to represent the gender information possessed by the facial subregions with textural pattern orders. A chain-type support vector machine (CSVM) based feature vector selection schema, is then developed to highlight the gender characteristics. As a result, the discriminative subregions and pattern orders are constructed as the feature representation for facial gender classification. Empirical studies on FRGC 2.0 dataset demonstrate that the proposed method is able to pinpoint the discriminative facial subregions and pattern orders between the female and male groups, leading to a 95.6% (or 96.1%) classification accuracy on symmetrically (or asymmetrically) segmented facial images. Moreover, experimental results on FRGC 2.0 and FERET datasets validate the effectiveness of the proposed MMIP matrix compared to the state- of-the-art feature descriptors including LBP, LDP, IDP and DWT. Finally, extensive experiments on four benchmark datasets (i.e., FRGC 2.0, FERET, LFW and UND) show that the proposed method achieves competitive classification performance compared with seven state-of- the-art baselines. Abstract Learning Discriminative Subregions and Pattern Orders for Facial Gender Classification Zhong Chena, Andrea Edwardsa, Yongsheng Gaob, Kun Zhanga,∗ aDepartment of Computer Science, Xavier University of Louisiana, LA 70125, USA bSchool of Engineering, Griffith University, QLD 4111, Australia Abstract Facial image-based gender classification has been widely used in many real- world applications. Most of the existing work, however, focuses on designing sophisticated or specific feature descriptors for the entire face, which neglects the discriminative information carried by facial components and pattern order combinations. To address the issue, in this paper, we first propose a generalized texture operator, i.e., the multi-spatial multi-order interlaced pattern (MMIP) matrix, to represent the gender information possessed by the facial subregions with textural pattern orders. A chain-type support vector machine (CSVM) based feature vector selection schema, is then developed to highlight the gender characteristics. As a result, the discriminative subregions and pattern orders are constructed as the feature representation for facial gender classification. Empir- ical studies on FRGC 2.0 dataset demonstrate that the proposed method is able to pinpoint the discriminative facial subregions and pattern orders between the female and male groups, leading to a 95.6% (or 96.1%) classification accuracy on symmetrically (or asymmetrically) segmented facial images. Moreover, ex- perimental results on FRGC 2.0 and FERET datasets validate the effectiveness of the proposed MMIP matrix compared to the state-of-the-art feature descrip- tors including LBP, LDP, IDP and DWT. Finally, extensive experiments on four benchmark datasets (i.e., FRGC 2.0, FERET, LFW and UND) show that ∗Corresponding author. Tel.: (+1) 504-520-6700. Fax: (+1) 504-520-7906. Email address: kzhang@xula.edu (Kun Zhang) Preprint submitted to Image and Vision Computing February 7, 2018 *Manuscript Click here to view linked References the proposed method achieves competitive classification performance compared with seven state-of-the-art baselines. Keywords: Discriminative subregions, Pattern order, Chain-type SVM, Feature vector selection, Facial gender classification 1. Introduction 1 The human face is an information rich source of traits (e.g., gender, age, 2 emotion, and ethnicity), which play an important role in many real-world i- 3 dentification and verification applications such as customer-oriented advertis- 4 ing, dynamic marketing surveys, video surveillance, human-computer interac- 5 tion [1, 2]. Among them, gender clue is the most fundamental and important 6 demographic attribute of human beings, which in the vast majority of cases 7 remains unchanged through a lifetime. In the last two decades, face-based gen- 8 der recognition has received much research interest in the scientific community. 9 In general, facial gender classification schemes can be classified into two main 10 categories: geometry-based methods and appearance-based methods [3]. The 11 former focuses on extracting the shape structure of the face to give prominence 12 to gender differences, and the representative methods include AAM [4], 3DGF 13 [5], and DCT [6]. The latter uses texture or statistics-oriented descriptors to 14 characterize the facial gender information, which have gained increasing atten- 15 tion due to the robustness to illumination variation and high discrimination 16 capability. 17 Numerous texture and statistics-oriented descriptors have been proposed 18 in the literature to effectively describe facial features for gender classification. 19 Representative methods include LBP [7], LDP [8], IDP [9], and DWT [10]. A- 20 mong them, LBP, LDP, and IDP are multiple pattern orders-based descriptors, 21 which have been validated more efficient in extracting discriminative features 22 for gender classification. In fact, LBP is regarded as the first-order circular 23 derivation pattern, while LDP and IDP are higher-order local patterns that en- 24 code distinctive spatial relationships for facial images. Although both texture 25 2 and statistical-oriented descriptors are widely used in various face recognition 26 applications, they often use a certain pattern order from the entire or partial 27 face to tackle gender recognition problem, where the best pattern order is often 28 determined by the best performance from different empirical studies [9]. Then, 29 how to automatically choose a suitable pattern order for specific tasks is stil- 30 l an unsolved problem. In the literature, these methods are applied to facial 31 gender classification under a global or local fashion. The global region based 32 feature descriptors usually use the entire face to extract discriminative features 33 without any distinctions among different facial components, leading to inferior 34 performance even for slight variations in illumination and pose. On the con- 35 trary, local region based feature descriptors have obtained impressive results by 36 investigating discriminative subregions (i.e., hair, forehead, cheeks, ears, chin, 37 etc.) for gender classification. For instance, Ueki et al. [11] integrate facial, 38 hairstyle, and clothing images for gender classification. Their empirical studies 39 show that the integration strategy significantly reduces the classification error 40 rate compared to the conventional approaches. Similar results are reported in 41 [12], where they compare facial features from internal zones (i.e., eyes, nose, 42 and mouth) and external zones (i.e., hair, chin, and ears). Experiments on the 43 FRGC database show that the external face zones contribute more useful in- 44 formation for gender classification. Later on, Lian & Lu [13] integrate facial 45 features from eyes, nose, mouth, and hair information for accurate gender clas- 46 sification. Li et al. [14] utilize both local features on five facial components 47 (i.e., forehead, eyes, nose, mouth and chin) and external features (i.e., hair and 48 clothing) to improve the classification performance. These studies show that 49 the external zones (i.e., clothing, head, shoulder, hair, and ears) give sufficient 50 information for gender recognition, while only using facial information is more 51 challenging for gender classification problem. 52 To address the facial gender classification problem, a large number of in- 53 novative, yet controversial methods have been introduced in recent years. For 54 example, Lu et al. [15] investigate significance of different facial regions for gen- 55 der classification. Experimental results show that the upper region of the face 56 3 proves to be the most significant part for gender classification. On the contrary, 57 Hasnat et al. [16] verify that the lower region of the face is more important 58 for gender classification as mouth and chin carry more important facial gender 59 information. Specifically, the male face is hairy and rough while the female face 60 is non-hairy and smooth. Additionally, some more discriminative analysis on 61 facial components are presented with controversy. For example, Merkow et al. 62 [17] apply the information obtained from the periocular region to identify gen- 63 der and achieve 85% accuracy on 936 low resolution images collected from the 64 web. When combining other facial components (i.e., nose, lip, and chin), the 65 dominant regions are clearly biased toward the ocular region. Therefore, they 66 claim that brow and eyes contain more valuable information for reliably gender 67 classification than the other facial components. Meanwhile, ¨Ozbudak et al. [18] 68 conduct an experimental study on examining the effects of facial components 69 for gender classification. To investigate the most influential component, parts 70 of these facial components (i.e., forehead, eyebrows, eyes, nose, lip, and chin) 71 are masked. Experimental results on the masked samples indicate that nose is 72 the most influential part for gender classification. 73 In summary, existing studies on facial gender classification address that both 74 texture or statistics-oriented descriptors and facial components are essential to 75 decrease redundant information and improve classification accuracy. However, 76 since datasets used in the above-mentioned techniques have different resolution- 77 s, quality, and resources, how to choose discriminative facial components and 78 suitable texture operator is still a challenging issue for facial gender classifica- 79 tion. Motivated by the issue, in this paper, we attempt to solve the following 80 two challenges: 1) how to develop a uniform and effective texture operator to 81 describe facial components and multiple pattern orders for facial image; and 2) 82 how to automatically select discriminative facial components and pattern orders 83 for gender classification. To the end, we first introduce a generalized texture op- 84 erator, i.e., the multi-spatial multi-order interlaced pattern (MMIP) matrix, to 85 enhance facial gender information, and then propose a chain-type SVM (CSVM) 86 based feature vector selection algorithm to investigate discriminative facial com- 87 4 Figure 1: The flowchart of the proposed gender classification method. ponents and pattern orders for gender classification. As a result, the proposed 88 approach addresses these challenges in the following aspects: 1) the proposed 89 MMIP matrix that considers both facial components and pattern orders brings 90 benefit to describing spatial and high-order facial gender information; and 2) 91 the proposed feature vector selection algorithm can figure out indispensable 92 information for gender classification. 93 The flowchart of the proposed gender classification system is shown in Figure 94 1, which consists of four seamless steps in the training and test stages: 1) 95 partition facial images into facial subregions; 2) extract multi-spatial and multi- 96 order texture information of facial images; 3) use CSVM-based feature vector 97 selection algorithm to form the feature representation; and 4) incorporate the 98 feature representation into the SVM classifier for gender classification. 99 The main contributions of this paper are summarized as follows. 100 • A uniform and efficient framework for describing facial subregions and 101 pattern orders of facial images is proposed. 102 • A CSVM-based feature vector selection algorithm for acquiring the most 103 informative feature vectors in a lower feature space is developed. 104 • The proposed method investigates the discriminative facial subregions and 105 pattern orders for facial gender classification. 106 • Experimental results on four different datasets (i.e., FRGC 2.0 [19], FER- 107 5 ET [20], LFW [21], and UND [22]) clearly demonstrate the efficacy of the 108 proposed method compared to the state-of-the-art baselines. 109 The remainder of this paper is organized as follows. A brief review of exist- 110 ing studies for facial gender classification is presented in Section 2. A detailed 111 description of the proposed gender classification approach is elaborated in Sec- 112 tion 3. Experimental results and analysis are reported in Section 4. Finally, a 113 short summary and an outlook on future work are summarized in Section 5. 114 2. Related Work 115 Generally, gender classification encompasses two main steps: feature extrac- 116 tion and classification. The classification step demands efficient feature repre- 117 sentation as it heavily affects the performance of a classifier. The key issue in 118 feature representation is a well-designed feature description, which should have 119 low computational cost, high robustness, and perform well on unseen test sam- 120 ples. State-of-the-art methods are summarized in Table 1, which compares the 121 features, classifiers and datasets of recent studies for face-based gender classifi- 122 cation. 123 These methods try to grasp the best features to distinguish female and male 124 groups. Table 1 summarizes some geometric-based, appearance-based, and some 125 hybrid methods for gender classification. For example, Moghaddam & Yang 126 [23] utilize raw image pixels of facial images as appearance-based features and 127 achieve an accuracy of 96.6% on the FERET database. Makinen & Raisamo 128 [1] systematically evaluate diverse face alignment and gender classification ap- 129 proaches on the FERET database. Baluja & Rowley [24] propose an efficient 130 gender classification method by boosting pixel comparisons of human face. Yang 131 et al. [25] use the texture raw pixels for feature extraction and adopt the Ad- 132 aBoost classifier for gender classification on the FERET database. All these 133 methods are not suitable to learn discriminative information between female 134 and male groups as they only use pixel intensity feature of facial image, leading 135 to being sensitive to pose, illumination and expression variations [3]. 136 6 Table 1: Overview of recent studies on gender classification. Method Feature Classifier Dataset Moghaddam & Yang [23] Raw pixels SVM FERET Makinen & Raisamo [1] Raw pixels SVM FERET Baluja & Rowly [24] Pixel comparison AdaBoost FERET Yang et al. [25] Texture AdaBoost FERET Tapia & P´erez [26] Fusion SVM FERET, LFW Shan [7] Boosted LBP SVM LFW Gallagher & Chen [27] Contextual Features GML Groups Rai & Khanna [28] Gabor+(2D)2PCA SVM FERET Mery & Bowyer [29] Random Patch Adaptive SRC FERET, LFW, Groups Hadid et al. [30] LBP, LPQ, BSIF SVM Groups Moeini et al. [31] LGBP SVM FERET, LFW Han et al. [32] BIF SVM LFW Jain & Huang [33] ICA LDA FERET Lu et al. [34] PPBTF SVM FERET Zheng & Lu [35] LGBP, MLBP, LBP SVMAC CAS-PEAL, FERET, BCMI Berbar [36] DCT, GLCM, DWT SVM AT@T, Face94, UMIST, FERET Andreu et al. [37] LBP+PCA SVM FERET, AR Our Method MMIP+CSVM SVM FRGC 2.0, FERET, LFW, UND To address real-world facial gender classification problem, Tapia & P´erez [26] 137 present mutual information for feature selection and fusion to improve gender 138 classification. They achieve promising results on LFW and FERET databases 139 by adopting the SVM classifier. Shan [7] proposes gender recognition on real- 140 world facial images in which AdaBoost is utilized to select the discriminative 141 LBP features. Finally, gender classification results are acquired by applying 142 the SVM classifier with the boosted LBP features. Gallagher & Chen [27] 143 propose contextual features and adopt Gaussian Maximum Likelihood (GML) 144 for gender classification. They perform real-world gender classification on the 145 Groups database and obtain 74.1% accuracy. Rai & Khanna [28] propose Gabor 146 based (2D)2PCA and adopt the SVM classifier for gender classification. They 147 perform the experiments on LFW and FERET databases with promising results 148 7 achieved. Mery & Bowyer [29] present random patch for feature extraction and 149 use the sparse adaptive classification to classify the gender on FERET and 150 Groups databases. 151 Furthermore, some attempts perform gender classification using different 152 feature extraction methods. Hadid et al. [30] propose 13 variants of local binary 153 for gender classification. They use the fusion of three features including LBP, 154 LPQ and BSIF. Finally, they adopt the SVM classifier and obtain an accuracy 155 of 89.85% on the Groups database. Moeini et al. [31] propose a feature fusion 156 method to extract features from both texture and depth images for gender 157 classification. Their method is evaluated on FERET and LFW databases by 158 adopting the SVM classifier. Recently, Han et al. [32] present a novel method 159 for demographic estimation (i.e., age, gender and ethnicity). Accordingly, they 160 use BIF and adopt the SVM classifier for gender classification on the LFW 161 database. 162 Table 1 also shows that a number of research studies that rely on LBP 163 operator are proposed in the literature. For example, Jain & Huang [33] use 164 ICA to represent the image and the SVM classifier for gender classification, and 165 achieve 96% classification accuracy on the FERET database. Shan [7] proposes 166 discriminative LBPH bins for gender classification. The selected LBPH bins 167 provide a compact facial representation and achieve 94.81% accuracy. Lu et 168 al. [34] propose PPBTF, and use AdaBoost and SVM classifiers for gender 169 classification with 92.71% accuracy on the FERET database achieved. Zheng & 170 Lu [35] use LGBP, MLBP and LBP to extract gender features, and achieve more 171 than 95% accuracy on multi-view facial images from the CAS-PEAL dataset and 172 frontal images from FERET and BCMI datasets. Berbar [36] uses DCT, GLCM 173 and DWT for feature extraction and the SVM classifier for gender classification. 174 Their results show that merging of features extracted from DCT and GLCM 175 degrades the classification accuracy on AT@T, FERET, UMIST, and Faces94 176 datasets, while using 2D-DWT, an improvement in accuracy (ranging between 177 96.18% and 99.6%) is observed for all datasets except FERET (92%). Andreu 178 et al. [37] conduct a comprehensive experimental study of gender classification 179 8 methods from neutral to distorted facial images. They compare local and global 180 approaches for classifying the gender by using gray-level information, PCA, 181 and LBP features as well as 1-NN, SVM, and PCA+SVM classifiers. Finally, 182 experiments on the FERET database achieve 94.06% accuracy. 183 It is worth noting that our study differs from the above-mentioned meth- 184 ods for facial gender classification by the following aspects: 1) conducting the 185 multi-spatial multi-order based texture operator to represent the gender infor- 186 mation; and 2) developing a feature vector selection scheme for investigating the 187 discriminative facial subregions and pattern orders. As a result, the discrimina- 188 tive characteristics are obtained by the proposed MMIP matrix and the CSVM 189 based feature vector selection scheme, leading to less redundant information for 190 improving the classification accuracy. 191 3. Methodology 192 In this section, we present the MMIP distribution matrix to extract discrim- 193 inative gender characteristics and the CSVM-based feature vectors elimination 194 and ranking approach to measure the importance of facial subregions and textu- 195 ral pattern orders for gender classification. Specifically, we elaborate its design 196 considerations and the implementation algorithms. Table 2 summarizes the 197 major notations used in this paper. 198 3.1. Feature Vector Extraction 199 We first introduce the MMP descriptor and then propose the MMIP matrix 200 to emphasize the gender discriminative information. 201 3.1.1. Multi-spatial Multi-order Pattern 202 Given a textural image D, we propose a new texture operator, MMP distri- 203 bution matrix MMP(D) = [Mp,q]P ×Q, to represent the intensity change with 204 high-order derivative at different subregions of D, where P and Q represent 205 the number of horizontal and vertical subregions of D, respectively. That is, 206 9 Table 2: Major notations. Notation Description D A textural image (D ⊆R2) MMP(D) MMP matrix of D Rp,q The p-th row and q-th column subregion of D Mp,q An element of MMP(D) (p = 1, . . . , P; q = 1, . . . , Q) MDP i(Rp,q) The i-th order MDP in Rp,q (i = 1, . . . , N) f i(Rp,q) The corresponding PDF of MDP i(Rp,q) Zp,q 0 An arbitrary pixel of Rp,q MDP i(Zp,q 0 ) The i-th order MDP of Zp,q 0 MDP i θ(Zp,q 0 ) The i-th order MDP in θ direction of Zp,q 0 θ = 2(k−1)π M A direction between a certain neighbor and Zp,q 0 (k = 1, . . . , M 2 ) Zp,q j The j-th neighbor of Zp,q 0 (j = 1, . . . , M) Ii θ(Zp,q j ) The i-th order derivative in θ direction of Zp,q j Zp,q l,j The l-th neighbor of Zp,q j (l = 1, . . . , M) F(·, ·) The encoding function for MDP MMIP(D) MMIP matrix of D Np,q An element of MMIP(D) MIP i(Rp,q) The i-th order MIP in Rp,q gi(Rp,q) The corresponding PDF of MIP i(Rp,q) G(·, ·) The encoding function for MIP Xt, yt The t-th training instance and class label (t = 1, . . . , T) sign(·), sum(·) The sign and summation function w, b The weight vector and bias term E(·) The transformation function that transforms a vector to a matrix Φ, K(·, ·) The mapping and kernel function ∥· ∥, ∥· ∥F The L2-norm and Frobenius norm εt, αt The slack variable and coefficient of Xt C, σ The regularization and width parameter W 2(α) The inverse-square of the classification margin 10 D = [Rp,q]P ×Q, where Rp,q represents the p-th row and q-th column subregion 207 of D. Each Mp,q in MMP(D) is defined as 208 Mp,q = [f 1(Rp,q), . . . , f N(Rp,q)], (1) where N is the number of derivative pattern order, f i(Rp,q) (i = 1, . . . , N) is a 209 feature vector that represents the probability density function (PDF) extract- 210 ed from the i-th order multi-order derivative pattern (MDP), MDP i(Rp,q), in 211 subregion Rp,q, which is defined as 212 MDP i(Rp,q) = {MDP i(Zp,q 0 )|Zp,q 0 ∈Rp,q}, (2) where Zp,q 0 ∈Rp,q is an arbitrary point of Rp,q, and MDP i(Zp,q 0 ) is the i-th 213 order MDP of Zp,q 0 , which can be computed by 214 MDP i(Zp,q 0 ) = {MDP i θ(Zp,q 0 )|θ = 2(k−1)π M , k = 1, . . . , M 2 }, (3) where MDP i θ(Zp,q 0 ) is the i-th order MDP in θ direction of Zp,q 0 , θ = 0, . . . , (M−2)π M 215 are the directions of all neighbors of Zp,q 0 , M (i.e., 2, 4, 8, 16, …) is the number 216 of the neighbors. MDP i θ(Zp,q 0 ) can be recursively computed by 217 MDP i θ(Zp,q 0 ) = {F(Ii−1 θ (Zp,q 0 ), Ii−1 θ (Zp,q 1 )), . . . , F(Ii−1 θ (Zp,q 0 ), Ii−1 θ (Zp,q M ))}, (4) where F(·, ·) that encodes a certain order gradient transitions into binary pat- 218 terns is defined as 219 F(u, v) =      0, if uv > 0 1, if uv ≤0 (5) and Ii−1 θ (Zp,q j ) that represents the (i −1)-th order derivative in θ direction of 220 Zp,q j is recursively defined as 221 Ii−1 θ (Zp,q 0 ) = Ii−2 θ (Zp,q 0 ) −Ii−2 θ (Zp,q j ), (6) Ii−1 θ (Zp,q j ) = Ii−2 θ (Zp,q j ) −Ii−2 θ (Zp,q l,j ). (7) In particular, the first order derivative I1 θ(Zp,q j ) is defined as 222 I1 θ(Zp,q 0 ) = I(Zp,q 0 ) −I(Zp,q j ), (8) I1 θ(Zp,q j ) = I(Zp,q j ) −I(Zp,q l,j ). (9) 11 where I(Zp,q 0 ) represents the gray value of Zp,q 0 . 223 In summary, MDP i θ(Zp,q 0 ) is a M-bit binary sequence that describes the 224 gradient trend changes of (i −1)-th order directional derivatives of Zp,q 0 while 225 MDP i(Zp,q 0 ) is a M 2 2 -bit binary sequence that describes (i −1)-th order deriva- 226 tives of Zp,q 0 . Therefore, MMP(D) actually encodes all the points in D with all 227 corresponding neighbors and directions, which is computationally time-consuming 228 for non-trivial images. To address this problem and further efficiently capture 229 the gender discriminative information, based on MMP matrix, we propose M- 230 MIP matrix as described below. 231 3.1.2. Multi-spatial Multi-order Interlaced Pattern 232 To develop a fast version of MMP and suppress the face information with 233 gender information emerging, we only maintain the encodes on the most relevant 234 neighbor for a given direction, i.e., discarding the MDP encodes on the other 235 (M −1)-neighbors with less relevant information (Figure 2). In this way, we 236 propose the MMIP distribution matrix, MMIP(D) = [Np,q]P ×Q, as the gender 237 textural descriptor. Each Np,q in MMIP(D) is defined as 238 Np,q = [g1(Rp,q), . . . , gN(Rp,q)], (10) where gi(Rp,q) (i = 1, . . . , N) is a feature vector that represents the PDF ex- 239 tracted from the i-th order multi-order interlaced pattern (MIP), MIP i(Rp,q), 240 in subregion Rp,q, which is defined as 241 MIP i(Rp,q) = {MIP i(Zp,q 0 )|Zp,q 0 ∈Rp,q}, (11) where MIP i(Zp,q 0 ) is the i-th order MIP of Zp,q 0 (Figure 2), which is defined as 242 MIP i(Zp,q 0 ) = {G(Ii−1 (M−2)π M (Zp,q 0 ), Ii−1 (M−2)π M (Zp,q 1 )), . . . , G(Ii−1 0 (Zp,q 0 ), Ii−1 0 (Zp,q M 2 )), G(Ii−1 2π M (Zp,q 0 ), Ii−1 2π M (Zp,q M 2 +1)), . . . , G(Ii−1 0 (Zp,q 0 ), Ii−1 0 (Zp,q M ))}, (12) where G(·, ·) is the encoding function for MIP, which is defined as 243 G(u, v) =      1, if u ≥v 0, if u < v (13) 12 Figure 2: Illustration of the coding process of MIP i(Zp,q 0 ) (highlight in red). In summary, in each direction, only the derivatives for the center point and 244 its neighbor point in that particular direction will be calculated. This will dra- 245 matically decrease the length of the pixel representing code produced by MIP 246 compared to the MDP operator. In this way, MIP keeps only the more impor- 247 tant information hence makes the process much faster. It produces an M-bit 248 representation of each point in D, which makes the operator M 2 times faster 249 than MDP. Moreover, compared to LBP, MIP contains a more detailed descrip- 250 tion by calculating the high-order derivative directional variations, while LBP 251 provides first-order derivative information and is incapable of describing more 252 detailed information. Based on MIP, we want to select the most discrimina- 253 tive feature vectors that encode the facial textural information inseparably for 254 better gender classification performance, leading to a new feature vector selec- 255 tion problem. Therefore, we propose CSVM based feature selection method to 256 learn the discriminative facial subregions with textural pattern orders for gender 257 classification as discussed below. 258 13 3.2. Feature Vector Selection 259 To investigate the discriminative facial subregions with textural pattern or- 260 ders for gender classification, we extend the traditional feature selection tech- 261 niques to the feature vector selection paradigm. The input training data com- 262 pose MMIP distribution matrices, e.g., X1 = MMIP1(D), . . . , XT = MMIPT (D) 263 extracted from T facial images. In the following, we first propose the CSVM 264 model and then discuss the CSVM-based feature vector selection technique. 265 3.2.1. Chain-type Support Vector Machine 266 For each input data Xt (t = 1, . . . , T), the corresponding feature dimension 267 is Xt = MMIPt(D) ∈RP D×QN, where D = 2M (or D = 2 ˜ M ( ˜ M < M)) is the 268 feature dimension of (or downsampling) MIP on each subregion, i.e., gi t(Rp,q) ∈ 269 RD. Since the feature representation of MIP, gi t(Rp,q), in each Xt is inseparable 270 for gender classification, with an abuse of notation, we rearrange the elements of 271 Xt and denote it by Xt = [g1 t (R1,1), . . . , gN t (R1,1), . . . , g1 t (RP,Q), . . . , gN t (RP,Q)]T ∈ 272 RS×D, where S = PQN. Let yt ∈{−1, +1} be the class label of Xt. 273 Since all the elements of an arbitrary row of Xt will integrally represent the 274 gender information, we call them chain-type feature vectors. Hence, figuring out 275 the most reliable classification boundary and investigating the most indispens- 276 able feature vectors for gender classification are necessary for the new learning 277 paradigm. In the literature, since SVM-based feature selection and classification 278 techniques are widely used with a strong interpretability, we extend the SVM 279 technique to chain-type SVM (CSVM) for selecting the chain-type feature 280 vectors. The propose CSVM aims to separate the training data with largest 281 margin and capture the best generalization ability of the new hyper-plane (or 282 decision function) 283 h(X) = sign(sum(E(w) ⊗X) + b), (14) where sign(·) is the sign function that determines whether the predicted classi- 284 fication comes out positive (+1) or negative (−1), ⊗represents the Kronecker 285 product of two matrices with the same size, b ∈R is the bias term, sum(·) is 286 14 the summation function that sums all the elements of a matrix, w ∈RS is the 287 weight vector, and E(·) is a transformation function that transforms a vector 288 to a matrix. In this paper, E(w) = [w, . . . , w] | {z } D ∈RS×D. 289 Suppose all the data are linearly separable, and the primary aim of the 290 hyper-plane is to maximize the margin (or distance) of the closest positive and 291 negative training data. To conduct the maximum margin or optimal hyper- 292 plane, we must classify the matrices of the training set correctly with largest 293 classification margin and minimize the norm of the weight vector simultaneously. 294 For the non-linearly separable data, we use the mapping function Φ that maps 295 the training matrices into a high dimensional space H, where the new hyper- 296 plane with maximal margin is constructed. We propose the following quadratic 297 optimization problem as the CSVM model 298 w ∗= argminw{ 1 2∥w∥2 + C ∑T t=1 εt} s.t. yt(sum(E(w) ⊗Φ(Xt)) + b) ≥1 −εt, εt ≥0, t = 1, . . . , T, (15) where Φ(Xt) ≜[Φ(x t,1), . . . , Φ(x t,D)] (X = [x t,1, . . . , x t,D], x t,d ∈RS, d = 299 1, . . . , D), ∥· ∥is the L2-norm, εt is the slack variable of Xt, and C is a reg- 300 ularization parameter. As a result, the optimal hyper-plane becomes h(X) = 301 sign(sum(E(w ∗) ⊗Φ(X))+b). To solve Eq. (15), we use the dual optimization 302 technique [38] to transform it into its dual form 303 α∗= argmaxα{∑T t=1 αt −1 2 ∑T t=1 ∑T r=1 αtαrytyrK(Xt, Xr)} s.t. ∑T t=1 αtyt = 0, 0 ≤αt ≤C, t = 1, . . . , T, (16) where αt is the corresponding coefficient of Xt (if αt ̸= 0, the corresponding 304 matrix is a support matrix), K(Xt, Xr) = exp{−∥Xt−Xr∥2 F 2σ2 } is the kernel func- 305 tion, σ is the width parameter and ∥·∥F is the Frobenius norm. We can further 306 obtain b∗based on the solution α∗of Eq. (16), leading to the new hyper-plane 307 h(X) = sign(∑T t=1 α∗ t ytK(Xt, X) + b∗). (17) 15 3.2.2. CSVM-based Feature Vector Selection 308 Based on CSVM theory, in this paper, we propose CSVM-based feature vec- 309 tor selection algorithm for discriminative gender analysis. Since the type of 310 input data for the classifier is a feature matrix, it is equivalent to the traditional 311 SVM (i.e., the input of the training data is a feature vector) when all rows of 312 the matrix are connected as a string according to the column order. For the 313 connected feature string, every element is actually a feature vector that repre- 314 sents the MIP operator of the facial subregions with a certain textural pattern 315 order. Therefore, selecting different rows of the input matrix is equivalent to 316 selecting the chain-type string one feature vector by one feature vector. 317 Similar to the Mercer’s Theorem [38], the kernel function can be derived 318 by K(Xt, X) = sum(Φ(Xt) ⊗Φ(X)). Thus, the discrimination function of the 319 CSVM used in Eq. (17) is 320 h(X) = sign(∑T t=1 α∗ t ytsum(Φ(Xt) ⊗Φ(X)) + b∗) = sign(sum(W(α∗) ⊗Φ(X)) + b∗), (18) where W(α∗) = ∑T t=1 α∗ t ytΦ(Xt). Therefore, the inverse-square of the classifi- 321 cation margin is 322 W 2(α∗) = ∥W(α∗)∥2 F = ∑T t=1 ∑T r=1 α∗ t α∗ rytyrsum(Φ(Xt) ⊗Φ(Xr)) = ∑T t=1 ∑T r=1 α∗ t α∗ rytyrK(Xt, Xr), (19) therefore, to lower the computational cost and extract discriminative feature 323 vectors for discriminative gender analysis simultaneously, we need to select the 324 most important feature vectors of the training matrices. Motivated by the fea- 325 ture selection techniques [39, 40] and to obtain the best generalization ability 326 of CSVM with largest classification margin, we propose the recursive feature 327 vector elimination scheme. We use the overall compound information among 328 initial feature vectors and remove the group of feature vectors that has nega- 329 tive impact on the classification margin of CSVM (Eq. (19)), i.e., the feature 330 vectors with deteriorating gender classification performance are discarded from 331 the training matrices. 332 16 Algorithm 1 CSVM-based Feature Vector Elimination & Ranking Algorithm. Input: T training matrices, Xt ∈RS×D (t = 1, . . . , T); the class label yt ∈{−1, +1}; the kernel function, K(·, ·); the width parameter, σ; the regularization parameter, C. Output: the index set F of the optimal feature vectors; the index set G of the corresponding rankings of these feature vectors; new training matrices, Xt ∈R|F|×D (t = 1, . . . , T). 1: Initialization: F = {1, . . . , S}, G = ∅; 2: repeat 3: Give a solution α∗of Eq. (16), W(α∗) ←CSVM Training; 4: Compute the inverse-square of the classification margin, W 2(α∗), using Eq. (19); 5: for each s ∈F do 6: Compute the inverse-square of the classification margin, W 2 −s(α∗ −s); 7: end for 8: Order these inverse-square of the classification margin, W 2(α∗), {W 2 −s(α∗ −s)}s∈F; 9: if W 2(α∗) is the smallest value of these (|F| + 1) items then 10: F = F; 11: {Xt}T t=1 = {Xt}T t=1; 12: else if W 2(α∗) is not the smallest value of these (|F| + 1) items, while W 2 −s(α∗ −s) is the smallest value of them then 13: F = F −{s}; 14: {Xt}T t=1 = {X−s t }T t=1; 15: end if 16: until F no longer changes. 17: for each s ∈F do 18: Compute the inverse-square of the classification margin, W 2 −s(α∗ −s); 19: end for 20: Arrange these values, {W 2 −s(α∗ −s)}s∈F, in an ascending order: W 2 −s1(α∗ −s1), . . . , W 2 −s|F|(α∗ −s|F|); 21: Update G = {1, . . . , |F|} for F = {s1, . . . , s|F|}. 17 We further determinate the ranking order of the selected feature vectors 333 according to their different contributions to the classification margin. This 334 method is actually a greedy strategy to obtain the optimal feature vectors as 335 the criterion tries to acquire optimal classification margin in each iteration. 336 Algorithm 1 presents the detailed implementation of the method. Note that X−s t 337 represents the matrix Xt of with s-th row removed, α∗ −s is the mapping solution 338 when all the training data {Xt}T t=1 are replaced by {X−s t }T t=1, {s1, . . . , s|F|} is 339 a certain permutation of F with W 2 −s1(α∗ −s1) ≤. . . ≤W 2 −s|F|(α∗ −s|F|). 340 4. Experiments 341 We conduct experiments of gender recognition on four famous datasets to 342 validate the efficiency of the proposed method. In the experiments we assume 343 that the face images have been detected and normalized as discussed in [41]. 344 Specifically, the experimental setup is presented in Section 4.1. The discrimi- 345 native facial subregions with textural pattern orders are investigated on FRGC 346 2.0 dataset in Section 4.2. We report the experimental results comparing four 347 different descriptors and classifiers on FRGC 2.0 and FERET datasets in Sec- 348 tion 4.3. Finally, the comparison results with seven state-of-the-art baselines 349 are discussed in Section 4.4. 350 4.1. Experimental Setup 351 We use four datasets to evaluate our algorithm, i.e., FRGC 2.0 [19], FERET 352 [20], LFW [21], and UND [22]. In the following, we summarize the characteristics 353 of these four datasets. 354 • The FRGC 2.0 is one of the most comprehensive datasets publicly available 355 for face analysis. In this paper, we select 458 unique front individuals from 356 the dataset, including 262 male and 196 female objects. All of them are 357 normalized with the entire face and cropped to 128×144 images so that 358 each image contains little or no hair information. 359 18 • The FERET dataset is one of the most widely used database in facial 360 gender classification studies. It consists of 14,126 gray-scale images rep- 361 resenting 994 individuals (i.e., 591 male and 403 female objects). These 362 images contain variations in lighting, facial expressions, pose angle, and 363 aging effects. In this work, we collect 1,000 mug-shot face images with 364 500 male faces and rest 500 female faces. Meanwhile, 512×768 pixels face 365 images are cropped and normalized to 80×80 pixels based on the true 366 positions of two eyes and a mouth. 367 • The LFW dataset contains more than 13,000 facial images collected from 368 the web, including 5,749 people. The faces have a large range of variation 369 include lighting, expression, pose, race, gender, and background. Here, 370 we only use the regular frontal facial images (i.e., the fa partition) in this 371 database, containing 550 male and 450 female objects. 372 • There are 10,700 individuals, i.e., 5,646 male and 5,054 female objects, in 373 the UND dataset. In the experiments, we use 5,000 male and 5,000 female 374 images for gender classification. 375 For a fair comparison, no other form of pre-processing is applied. The pro- 376 posed MMIP matrix is compared with four different feature descriptors, i.e., 377 LBP [7], LDP [8], IDP [9], and DWT [10]. Meanwhile, we compare our method 378 with four widely-used classifiers, i.e., k-NN [42], Na¨ıve Bayes [43], AdaBoost 379 [44], and SVM [45]. In addition, to compute spatially enhanced histograms, 380 the images are divided into non-overlapping blocks of size 4 × 4 resulting in 16 381 blocks. 382 To evaluate gender classification performance, stratified five-fold cross vali- 383 dation is used for the datasets. The five-fold cross validation scheme randomly 384 divides the data into five non-overlapping subsets and at a time, only one subset 385 (20% of the data) is used as a test while other subsets (80% of the data) are 386 used to train a classifier. This procedure is repeated five times so that each 387 subset is used once as a test set. In addition, in order to highlight the perfor- 388 mance of different approaches, we assume the training and test datasets have 389 19 the independent constraint condition (except for [35]), which means that the 390 same individual does not appear in both training and test set. Finally, the av- 391 erages and standard variations of five different classification rates are reported. 392 The parameters of SVMs are determined by five-fold cross validation for the 393 Gaussian kernel function. The modified code for CSVM is based on LibSVM 394 provided by [45]. The experiments are conducted on a computer with 4.00 GB 395 RAM and Inter(R) Core (TM) i7-3770 with 3.40 GHz CPU. 396 4.2. Discriminative Feature Analysis 397 As a case study, in this section, we intend to find the most indispensable 398 feature vectors for gender classification on FRGC 2.0 dataset. In order to explore 399 the significant differences between the female and male groups by highlighting 400 meaningful facial components (i.e., forehead, eyes, nose, cheek, mouth, chin, or 401 cross areas), we have designed two cases of experiments, i.e., symmetrically or 402 asymmetrically segmenting the facial regions. 403 4.2.1. Discriminative Feature Analysis through Symmetrical Division 404 Every face sample is cropped to 128×144 and divided into 16 blocks of 32×36 405 pixels symmetrically. Four orders of MIP feature (i.e., first-order, second-order, 406 third-order, and fourth-order MIP) are extracted. Totally we have 64×256 = 407 16,384 dimensions for each sample, using histogram down sampling, the dimen- 408 sion of each sample is reduced to 64×32 = 2,048. The grid search technique 409 is used to figure out the best parameter value (C, σ) of SVM (Gaussian kernel 410 is utilized in the paper). We set the kernel parameter σ = 0.1, 0.2, . . . , 1 and 411 penalty parameter C = 1, 10, . . . , 104. In grid search process, pairs of (C, σ) are 412 tested, and the best parameter combination (i.e., C = 102, σ = 0.6) with average 413 classification accuracy 95.58% is achieved. Meanwhile, the number of eliminated 414 feature vectors is 26 and hence 38 feature vectors are optimally preserved. 415 Figure 3 shows the ranking list of these eliminated and selected feature 416 vectors, which indicates that the selected feature vectors are also approximately 417 symmetrical. The CSVM based feature selection method not only increases 418 20 Figure 3: The ranking list of the eliminated and selected feature vectors. The left part is the input face image and subregions divided in a symmetrical way; the right parts are: (a) sixteen subregions with four pattern orders of MIPs; (b) the selected and eliminated pattern orders in different subregions (the yellow parts represent the remaining feature vectors, e.g., 2-3(4) represents the second region with third-order MIP fourthly selected; the red parts represent eliminat- ed feature vectors, e.g., 2-2(8) represents the second region with second-order MIP eighthly eliminated); (c) the optimal feature vectors combination; (d) the representation of selected subregions and pattern orders. the accuracy but also decreases the dimension of the feature space enormously 419 from 64×32 = 2,048 to 38×32 = 1,216 on FRGC 2.0 dataset. To separate the 420 male and female groups with optimal classification accuracy, Figure 3 shows 421 that different subregions should be optimal combined with different textural 422 pattern orders. For instance, in the region of forehead, we need two high- 423 21 Table 3: Gender classification rate comparison without feature vector selection mechanism. Pattern order First-order MIP Second-order MIP Third-order MIP Fourth-order MIP All orders MIP Classification rate 73.80% 82.10% 59.83% 65.07% 71.62% order patterns (i.e., third-order and fourth-order MIP), while in region of cheek, 424 we need another two high-order patterns (i.e., second-order and fourth-order 425 MIP) to figure out gender differences. Finally, the combination of all these 426 feature vectors presents the largest classification margin, which validates that 427 this combination is indispensable for gender classification. Furthermore, all the 428 different importance for the SVM classifier can be quantitatively measured and 429 ranked. According to the results of ranked feature vectors, the upper subregions 430 with pattern orders combination have the highest priority to be selected for 431 discriminative analysis and gender classification. Therefore, the upper regions 432 of the face have been proved to be the most significant parts for the task of 433 gender classification. 434 We also compare our results with first-order, second-order, third-order and 435 fourth-order MIP, and all of them for gender classification. Table 3 shows that 436 compared to the first-order or second-order MIP, the gender classification accu- 437 racy decreases significantly when using all pattern orders combination. However, 438 all the gender classification rates are significantly less than our proposed method 439 as we use facial subregions with textural pattern orders to form the optimal fea- 440 ture representation for gender classification, which also supports that gender 441 differences exist in different facial subregions with different textural pattern 442 orders, and also with different priorities to be selected. 443 4.2.2. Discriminative Feature Analysis through Asymmetrical Division 444 When dividing the facial images in an asymmetrical way, we explore the 445 significant differences between female and male groups in the meaningful facial 446 components, i.e., forehead, eyes, nose, cheek, mouth, and chin. In the experi- 447 ments, the parameter combination (C = 103, σ = 0.5) with average classification 448 22 Figure 4: The ranking list of the eliminated and selected feature vectors. The left part is the input face image and subregions divided in an asymmetrical way; the right parts are: (a) sixteen subregions with four pattern orders of MIPs; (b) the selected and eliminated pattern orders in different subregions; (c) the optimal feature vectors combination; (d) the representation of selected subregions and pattern orders. accuracy 96.10% is selected. Meanwhile, the number of eliminated feature vec- 449 tors is 30 and 34 feature vectors are optimally preserved. 450 Figure 4 shows the ranking list of these eliminated and selected feature 451 vectors and suggests that the selected feature vectors are also approximately 452 asymmetrical. The CSVM based feature selection method not only increases the 453 accuracy but also decreases the dimension of the feature space enormously from 454 64×32 = 2,048 to 34×32 = 1,088 on FRGC 2.0 dataset. To separate the male 455 and female groups with optimal classification accuracy, Figure 4 shows different 456 23 Table 4: Gender classification rate comparison without feature vector selection mechanism. Pattern order First-order MIP Second-order MIP Third-order MIP Fourth-order MIP All orders MIP Classification rate 78.26% 84.61% 69.25% 72.43% 76.86% subregions should be optimal combined with different textural pattern orders. 457 For instance, in the forehead region, we need two high-order patterns (i.e., third- 458 order and fourth-order MIP), while in eyes, nose, cheek and chin region, we need 459 three high-order patterns (i.e., second-order, third-order and fourth-order MIP), 460 in the mouth region, we need another two high-order patterns (i.e., second-order 461 and third-order MIP) to figure out gender differences. 462 Finally, the combination of all these feature vectors presents the largest 463 classification margin, which validates that this combination is indispensable for 464 gender classification. Furthermore, the importance of the SVM classifier can be 465 quantitatively measured and ranked. According to the results of ranked feature 466 vectors, compared to the under subregions (i.e., mouth and chin), the upper 467 subregions (i.e., forehead, eyes, nose and cheek) with selected pattern orders 468 combination have the highest priority to be selected, which further strongly 469 supports that the upper regions of the face are the most significant parts for the 470 task of gender classification. 471 We also compare our results with first-order, second-order, third-order and 472 fourth-order MIP, and all of them for gender classification, Table 4 shows that 473 compared to the first-order or second-order MIP, the gender classification ac- 474 curacy decreases significantly when using all pattern orders combination. How- 475 ever, all the gender classification rates are significantly less than our proposed 476 method as we use facial subregions with textural pattern orders to form the 477 optimal feature representation for gender classification. 478 In summary, our proposed approach has a significant improvement for gender 479 classification by investigating gender differences in facial subregions and pattern 480 orders. On one hand, when we symmetrically divide the facial images, the se- 481 24 lected feature vectors are approximately symmetrical according to the selected 482 feature vectors. Therefore, we can further apply the proposed method to gender 483 classification even if either left or right face is corrupted. These experimental 484 results not only help determine the significant differences between female and 485 male groups, but also obtain the symmetry of the selected feature vectors while 486 reducing the classification error rate to 4.4%. One the other hand, when we 487 asymmetrically divide the facial images, the proposed method not only select- 488 s significant different subregions and pattern orders for gender classification, 489 but also explores the significant differences between female and male groups in 490 meaningful facial components, i.e., forehead, eyes, nose, cheek, mouth, and chin. 491 As a result, the gender classification accuracy has been improved from 95.58% 492 to 96.10%. 493 4.3. Comparison with Different Feature Descriptors and Classifiers 494 The results of gender characteristics analysis show that selected feature vec- 495 tors are indispensable for improving gender classification performance. For com- 496 prehensive analysis, in this section, we further compare our proposed method 497 with four widely used feature descriptors and classifiers for gender classification 498 on FRGC 2.0 and FERET databases. 499 4.3.1. Comparative Studies on FRGC 2.0 Dataset 500 The classification rates are listed in Figure 5. First, in all cases except the 501 first one, the SVM classifier gives slightly better accuracy than the AdaBoost 502 classifier, however, the AdaBoost classifier gives slightly better standard vari- 503 ations than that of the SVM classifier, one possible reason is that the SVM 504 classifier has a greater capacity to fit attributes of individual faces. Overall, the 505 best accuracy for both SVM and AdaBoost classifiers seems to be significantly 506 better than other classifiers when using the same feature descriptor. Second, the 507 SVM classifier is superior in classification performance on FRGC 2.0 dataset. 508 The classification rate of MIP+SVM shows that the feature vectors extracted 509 by MIP operator and selected by CSVM algorithm are discriminative for gender 510 25 Figure 5: The average classification accuracy and standard deviation by 5-fold cross-validation are compared with four feature descriptions and classifiers. classification. Additionally, compared to LBP, LDP, IDP and DWT operators, 511 the proposed MMIP matrix is more powerful to integrate spatial and high-order 512 information for enhancing gender’s characteristics. 513 4.3.2. Comparative Studies on FERET Dataset 514 Figure 6 lists the compared results of gender classification with different fea- 515 ture descriptors and classifiers on FERET dataset. This is similar to the above 516 compared results. Overall, the proposed method is consistently competitive 517 with the other four feature descriptors. The SVM classifier has better accuracy 518 than the other classifiers in most cases. This supports the effectiveness of the 519 26 Figure 6: The average classification accuracy and standard deviation by 5-fold cross-validation are compared with five feature descriptions and classifiers. proposed method. Moreover, the overall average accuracy on FRGC 2.0 dataset 520 is slightly better than that on FERET dataset, one likely reason is that our 521 experiments use the low resolution images in the FERET dataset. 522 In addition, we observe that given a feature extraction method, when using 523 different classifiers, the SVM classifier achieves the best classification accuracy, 524 the other classifiers perform from best to worst are AdaBoost, Na¨ıve Bayes and 525 k-NN. Given a classifier, in most cases, the MIP descriptor achieves the best 526 classification accuracy, the other descriptors performance from best to worst 527 LBP, IDP, LDP and then DWT. One likely reason is that, in our model, the 528 process of training the SVM classifier is achieved by efficient eliminating a large 529 number of feature vectors with low relevance and high redundancy. These results 530 27 Table 5: Classification performance of state-of-the-art gender classification ap- proaches on all datasets using 5-fold cross validation (the best result on each dataset is highlighted in bold). Dataset/Method Jain [33] Makinen [1] Lu [34] Zheng [35] Shan [7] Berbar [36] Andreu [37] Our Method FRGC 2.0 94.86% 91.93% 90.32% 92.58% 95.26% 94.08% 95.69% 96.10% FERET 95.67% 92.86% 94.85% 99.10% 94.81% 92.00% 94.06% 95.03% LFW 93.15% 91.33% 93.68% 94.81% 92.56% 90.49% 92.25% 94.10% UND 86.33% 84.51% 86.19% 87.28% 84.68% 83.05% 85.72% 89.42% have been discussed in Section 4.2. 531 4.4. Comparison with State-of-the-art Approaches 532 Table 5 shows the classification performance of our proposed method on all 533 datasets compared to other seven representative gender classification approach- 534 es. For example, on FRGC 2.0 and UND datasets, the classification accuracy 535 of our method consistently leads the corresponding classification accuracy of 536 other competitors. Specifically, the state-of-the-art performance is 95.69% by 537 LBP+SVM [37] on the FRGC 2.0 dataset, and the best classification accuracy 538 among the competitors is achieved by [35] (87.28%) on the UND dataset. Our 539 proposed method is relatively more apt to achieve higher classification accuracy 540 (i.e., 96.10% and 89.42%) than the best competitors on these two datasets. It is 541 worth noting that, the differences in gender classification are with independent 542 constraint conditions. However, in [35], facial images of individual people may 543 appear in both the training and test sets (i.e., people have multiple images in 544 the FERET dataset), which makes the task both easier and much less applicable 545 to the problem we are interested in, i.e., recognizing the gender of people for 546 whom we have not trained the classifier. 547 The results shown in Table 5 indicate that the performances of gender clas- 548 sification without independent constraint conditions is apparently higher than 549 that with independent constraint conditions. When using the method proposed 550 by [35], the classification rate with independent constraint condition is reduced 551 by about 7% compared to that without the independent constraint condition. 552 28 On the FERET dataset, the best performance is obtained by [35], however, 553 they use a smaller dataset without independent constraint, which will actu- 554 ally make the problem easier to handle than other methods. As mentioned, 555 the classification rate with independent constraint conditions is significantly re- 556 duced when using the method proposed by [35], this is also true on the LFW 557 dataset (Table 5). Our proposed method is comparable with the method pro- 558 posed by [33], where they also use more training images and less test images. 559 For comparison with other methods, our proposed method achieves best classi- 560 fication accuracy with independent constraint for front images. Therefore, with 561 considering independent constraint, our proposed method has obtained better 562 classification performance than those previously published methods for gender 563 classification on FERET and LFW datasets. Meanwhile, it should be empha- 564 sized that the gender classification results are improved significantly while the 565 number of input feature vectors is reduced drastically, which indicates that the 566 proposed method has important implications for real-world applications. 567 5. Conclusion 568 In this paper, we propose a novel technique for facial gender recognition 569 by investigating the discriminative subregions and pattern orders. The pro- 570 posed method consists of two important components: 1) a generalized texture 571 operator, i.e., the MMIP distribution matrix, for enhancing facial gender infor- 572 mation; and 2) a CSVM based feature vector selection algorithm for measuring 573 the importance of facial subregions and pattern orders. As a result, the re- 574 dundant and irrelevant feature vectors represented by the MMIP matrix can 575 be eliminated with the discriminative gender characteristics pinpointed for sub- 576 sequent facial gender classification. The proposed method is technically effect 577 and obtains higher classification performance with discriminative gender infor- 578 mation achieved. Compared to the classical feature descriptors, the proposed 579 MMIP matrix can significantly improve the classification accuracy. Meanwhile, 580 the proposed method achieves comparable classification performance compared 581 29 to seven state-of-the-art baselines for gender classification. As part of future 582 work, we will 1) apply the proposed method to classify gender from noisy facial 583 images; and 2) investigate discriminative gender characteristics from corrupted 584 facial images for robust gender classification. 585 Acknowledgements 586 This publication was made possible by funding from the DOD ARO grant 587 #W911NF-15-1-0510, and the NIH grants 2G12MD007595 and P01 CA214091. 588 References 589 [1] E. Makinen, R. Raisamo, Evaluation of gender classification methods with 590 automatically detected and aligned faces, IEEE Transactions on Pattern 591 Analysis and Machine Intelligence 30 (3) (2008) 541–547. 592 [2] J. Bekios-Calfa, J. M. Buenaposada, L. Baumela, Revisiting linear dis- 593 criminant techniques in gender recognition, IEEE Transactions on Pattern 594 Analysis and Machine Intelligence 33 (4) (2011) 858–864. 595 [3] H.-C. Shih, Robust gender classification using a precise patch histogram, 596 Pattern Recognition 46 (2) (2013) 519–528. 597 [4] T. F. Cootes, G. J. Edwards, C. J. Taylor, Active appearance models, IEEE 598 Transactions on Pattern Analysis and Machine Intelligence 23 (6) (2001) 599 681–685. 600 [5] L. Ballihi, B. B. Amor, M. Daoudi, A. Srivastava, D. Aboutajdine, Boost- 601 ing 3-d-geometric features for efficient face recognition and gender classi- 602 fication, IEEE Transactions on Information Forensics and Security 7 (6) 603 (2012) 1766–1779. 604 [6] Z. M. Hafed, M. D. Levine, Face recognition using the discrete cosine trans- 605 form, International Journal of Computer Vision 43 (3) (2001) 167–188. 606 30 [7] C. Shan, Learning local binary patterns for gender classification on real- 607 world face images, Pattern Recognition Letters 33 (4) (2012) 431–437. 608 [8] B. Zhang, Y. Gao, S. Zhao, J. Liu, Local derivative pattern versus local 609 binary pattern: face recognition with high-order local pattern descriptor, 610 IEEE Transactions on Image Processing 19 (2) (2010) 533–544. 611 [9] A. Shobeirinejad, Y. Gao, Gender classification using interlaced derivative 612 patterns, in: International Conference on Pattern Recognition, IEEE, 2010, 613 pp. 1509–1512. 614 [10] H. Hu, Variable lighting face recognition using discrete wavelet transform, 615 Pattern Recognition Letters 32 (13) (2011) 1526–1534. 616 [11] K. Ueki, H. Komatsu, S. Imaizumi, K. Kaneko, N. Sekine, J. Katto, 617 T. Kobayashi, A method of gender classification by integrating facial, 618 hairstyle, and clothing images, in: International Conference on Pattern 619 Recognition, IEEE, 2004, pp. 446–449. 620 [12] A. Lapedriza, M. J. Maryn-Jimenez, J. Vitria, Gender recognition in non 621 controlled environments, in: International Conference on Pattern Recogni- 622 tion, IEEE, 2006, pp. 834–837. 623 [13] X.-C. Lian, B.-L. Lu, Gender classification by combining facial and hair in- 624 formation, in: International Conference on Neural Information Processing, 625 Springer, 2008, pp. 647–654. 626 [14] B. Li, X.-C. Lian, B.-L. Lu, Gender classification by combining clothing, 627 hair and facial component classifiers, Neurocomputing 76 (1) (2012) 18–27. 628 [15] L. Lu, Z. Xu, P. Shi, Gender classification of facial images based on mul- 629 tiple facial regions, in: WRI World Congress on Computer Science and 630 Information Engineering, IEEE, 2009, pp. 48–52. 631 [16] A. Hasnat, S. Haider, D. Bhattacharjee, M. Nasipuri, A proposed system 632 for gender classification using lower part of face image, in: International 633 Conference on Information Processing, IEEE, 2015, pp. 581–585. 634 31 [17] J. Merkow, B. Jou, M. Savvides, An exploration of gender identification us- 635 ing only the periocular region, in: International Conference on Biometrics: 636 Theory Applications and Systems, IEEE, 2010, pp. 1–5. 637 [18] ¨O. ¨Ozbudak, M. Kirci, Y. C¸akir, E. O. G¨une¸s, Effects of the facial and 638 racial features on gender classification, in: Mediterranean Electrotechnical 639 Conference, IEEE, 2010, pp. 26–29. 640 [19] P. J. Phillips, P. J. Flynn, T. Scruggs, K. W. Bowyer, J. Chang, K. Hoff- 641 man, J. Marques, J. Min, W. Worek, Overview of the face recognition 642 grand challenge, in: IEEE Conference on Computer Vision and Pattern 643 Recognition, IEEE, 2005, pp. 947–954. 644 [20] P. J. Phillips, H. Moon, S. A. Rizvi, P. J. Rauss, The feret evaluation 645 methodology for face-recognition algorithms, IEEE Transactions on Pat- 646 tern Analysis and Machine Intelligence 22 (10) (2000) 1090–1104. 647 [21] E. Learned-Miller, G. B. Huang, A. RoyChowdhury, H. Li, G. Hua, Labeled 648 faces in the wild: A survey, in: Advances in Face Detection and Facial 649 Image Analysis, Springer, 2016, pp. 189–248. 650 [22] L. A. Alexandre, Gender recognition: A multiscale decision fusion ap- 651 proach, Pattern Recognition Letters 31 (11) (2010) 1422–1427. 652 [23] B. Moghaddam, M.-H. Yang, Learning gender with support faces, IEEE 653 Transactions on Pattern Analysis and Machine Intelligence 24 (5) (2002) 654 707–711. 655 [24] S. Baluja, H. A. Rowley, et al., Boosting sex identification performance, 656 International Journal of computer vision 71 (1) (2007) 111–119. 657 [25] Z. Yang, M. Li, H. Ai, An experimental study on automatic face gender 658 classification, in: International Conference on Pattern Recognition, IEEE, 659 2006, pp. 1099–1102. 660 32 [26] J. E. Tapia, C. A. Perez, Gender classification based on fusion of different 661 spatial scale features selected by mutual information from histogram of 662 lbp, intensity, and shape, IEEE Transactions on Information Forensics and 663 Security 8 (3) (2013) 488–499. 664 [27] A. C. Gallagher, T. Chen, Understanding images of groups of people, in: 665 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 666 2009, pp. 256–263. 667 [28] P. Rai, P. Khanna, A gender classification system robust to occlusion us- 668 ing gabor features based (2d)2pca, Journal of Visual Communication and 669 Image Representation 25 (5) (2014) 1118–1129. 670 [29] D. Mery, K. Bowyer, Automatic facial attribute analysis via adaptive sparse 671 representation of random patches, Pattern Recognition Letters 68 (2015) 672 260–269. 673 [30] A. Hadid, J. Ylioinas, M. Bengherabi, M. Ghahramani, A. Taleb-Ahmed, 674 Gender and texture classification: A comparative analysis using 13 variants 675 of local binary patterns, Pattern Recognition Letters 68 (2015) 231–238. 676 [31] A. Moeini, K. Faez, H. Moeini, Real-world gender classification via local 677 gabor binary pattern and three-dimensional face reconstruction by generic 678 elastic model, IET Image Processing 9 (8) (2015) 690–698. 679 [32] H. Han, C. Otto, X. Liu, A. K. Jain, Demographic estimation from face 680 images: Human vs. machine performance, IEEE Transactions on Pattern 681 Analysis and Machine Intelligence 37 (6) (2015) 1148–1161. 682 [33] A. Jain, J. Huang, Integrating independent components and linear dis- 683 criminant analysis for gender classification, in: International Conference 684 on Automatic Face and Gesture Recognition, IEEE, 2004, pp. 159–163. 685 [34] H. Lu, Y. Huang, Y. Chen, D. Yang, Automatic gender recognition based on 686 pixel-pattern-based texture feature, Journal of Real-Time Image Processing 687 3 (1-2) (2008) 109–116. 688 33 [35] J. Zheng, B.-L. Lu, A support vector machine classifier with automatic con- 689 fidence and its application to gender classification, Neurocomputing 74 (11) 690 (2011) 1926–1935. 691 [36] M. A. Berbar, Three robust features extraction approaches for facial gender 692 classification, The Visual Computer 30 (1) (2014) 19–31. 693 [37] Y. Andreu, P. Garc´ıa-Sevilla, R. A. Mollineda, Face gender classification: A 694 statistical study when neutral and distorted faces are combined for training 695 and testing purposes, Image and Vision Computing 32 (1) (2014) 27–36. 696 [38] M. A. Hearst, S. T. Dumais, E. Osuna, J. Platt, B. Scholkopf, Support 697 vector machines, IEEE Intelligent Systems and their applications 13 (4) 698 (1998) 18–28. 699 [39] I. Guyon, J. Weston, S. Barnhill, V. Vapnik, Gene selection for cancer clas- 700 sification using support vector machines, Machine Learning 46 (1) (2002) 701 389–422. 702 [40] J. Neumann, C. Schn¨orr, G. Steidl, Combined svm-based feature selection 703 and classification, Machine Learning 61 (1) (2005) 129–150. 704 [41] S. Gutta, J. R. Huang, P. Jonathon, H. Wechsler, Mixture of experts for 705 classification of gender, ethnic origin, and pose of human faces, IEEE Trans- 706 actions on Neural Networks 11 (4) (2000) 948–960. 707 [42] F. Pernkopf, Bayesian network classifiers versus selective k-nn classifier, 708 Pattern Recognition 38 (1) (2005) 1–10. 709 [43] S.-B. Kim, K.-S. Han, H.-C. Rim, S. H. Myaeng, Some effective techniques 710 for naive bayes text classification, IEEE Transactions on Knowledge and 711 Data Engineering 18 (11) (2006) 1457–1466. 712 [44] S. Mathanker, P. Weckler, T. Bowser, N. Wang, N. Maness, Adaboost 713 classifiers for pecan defect classification, Computers and Electronics in A- 714 griculture 77 (1) (2011) 60–68. 715 34 [45] C.-C. Chang, C.-J. Lin, Libsvm: a library for support vector machines, 716 ACM Transactions on Intelligent Systems and Technology 2 (3) (2011) 27. 717 35