Privacy issues are becoming more and more important in Artificial Intelligent (AI). Yet, there is a lack of systematized or standardized privacy framework that focuses on AI embedded smart toys, with conversation functionality, to address user privacy requirements. To address this issue, we develop a prototype of a Privacy Identification (PI) system for Dialogue Design (DD). We call this system a PI-DD system. To develop such a PI-DD system, our research works were separated into two parts: (1) Create phrases' database that considers the Personally Identifiable Information (PII) law which states privacy laws and information security best practices and is used in various U.S. federal, and (2) Build the dialogue rule for robot conversations. To illustrate the algorithms of the PI-DD system, we take the sample phrase of Mattel's Hello-Barbie smart toy. We present an architecture of the PI-DD algorithm at the end of this paper.