In today’s modern medicine, with the spreading use of radiological imaging devices in medical centers, the need to accurate, reliable, and portable medi­cal image analysis and understanding systems has been increasing constantly. Since usually the images are not accompanied by the required clinical anno­tation, automatic tagging and captioning systems are among the most desired applications. This research proposes an automatic structured radiology report generation system that is based on annotation methods. Extracting useful and descriptive image features to model conceptual contents of the images is one of the main challenges in this regard. Considering the ability of deep neural networks in soliciting informative and effective features as well as lower reso­urce requirements, MobileNets are employed as the main building block of the proposed system. Furthermore due to the lack of large labeled medical data for training the network and risk of over-fitting, a joint descriptor is induced from the deep features and local bina­ry patterns. Experimental results confirm the efficiency of the proposed hybrid approach with accuracy 91.4%, as compared to the end-to-end deep networks and classic annotation methods.