Semantic image segmentation based on Convolutional Neural Networks (CNNs) is one of the main approaches in computer vision area. In convolutional neural network-based approaches, a pre-trained CNN which is trained on the large image classification datasets is generally used as a backend to extract features (image descriptors) from the images. Whereas, the special size of output features from CNN backends are smaller than the input images, by stacking multiple deconvolutional layers to the last layer of backend network, the dimension of output will be the same as the input image. Segmentation using local image descriptors without involving relationships between these local descriptors yield weak and uneven segmentation results. Inspired by these observations, in this research we propose Context-Aware Features (CAF) unit. CAF unit generate image-level features using local-image descriptors. This unit can be integrated into different semantic image segmentation architectures. In this study, by adding the proposed CAF unit to the Fully Convolutional Network (FCN) and DeepLab-v3-plus base architectures, the FCN-CAF and DeepLab-v3-plus-CAF architectures are proposed respectively. PASCAL VOC2012 datasets have been used to train the proposed architectures. Experimental results show that the proposed architectures have 2.7% and 1.81% accuracy improvement (mIoU) compared to the related basic architectures, respectively.