Fixed asset tax which is assessed based on the purpose of land use, is a very important resource in Japanese municipalities. Article 408 of the Local Tax Act shows that field survey is required to be carried out once a year to confirm the use of land. Municipal officials spend a lot of time and effort to conduct field surveys to check the use of land in order to make a proper evaluation. Therefore, in this study, we utilize three types of data, aerial photography images, shapefiles, and taxable data of registered land category owned by local governments to promote the utilization of public and private sector data. The purpose of this study is to reduce the burden of land evaluation work and improve its efficiency and accuracy by using deep learning. It is confirmed that our system is effective in supporting land evaluation work and improving the efficiency of field survey work.