This paper deals with the passive control problem for uncertain neutral systems with time-varying delays via memory state feedback. Based on the stability theory of Lypunov and the decomposing and magnifying techniques of matrix, an existence condition in terms of linear matrix inequality(LMI) and an explicit formula of gains of memory state feedback controller for the unperturbed neutral systems, which guarantee the asymptotic stability and passivity of the closed-loop systems, are shown. Furthermore, the results are extended to the case in which the underlying system is dependent on the uncertain parameters. Finally, the efficiency of the proposed memory state feedback controller is demonstrated in a simulation example.