Human diseases such as diabetes, high blood pressure and the cerebral source disorders have effects on the retina vessels of human’s eyes. By classifying the retina vessels as two sets of arteries and veins, it can be evaluated the progress and symptoms of mentioned diseases. In this paper, a retinal blood vessel classification algorithm based on structural, directional and frequency features along with feature optimization using Tagouchi genetic algorithm is proposed. For this purpose, to classify the vessels in fundus images, at first the vessels are segmented. In this algorithm, to extract simultaneously information related to direction, diameter and dynamical behavior of the blood vessel, a novel feature based on wavelet transform using entropy contents of DWT and Directional Wavelet Entropy (DWE), Fourier transform using Fourier descriptors have been presented. Also 2-D Frequency Similarity Sectors (2DFSS) is introduced to represent and describe the variations of thickness and direction of the blood vessel. After extracting the feature vector using hybrid model of Genetic algorithm and Tagouchi strategy, the optimal features are selected. Then by employing the multi-layer neural network classifier, the vessels are recognized into arteries and veins classes. With these represented attributes, the classification is performed based on the structure and direction of vessels. Ultimately, the accuracy rate of 82.09% and precision rate of 81.58% are simultaneously obtained in problem of the retinal vessel recognition on a database consisting of 40 images.