Research has shown how people anthropomorphize conversational agents (CA) and unconsciously bring their gender stereotypes into human-agent interaction. For this reason, there has been a long lasted dilemma on whether designers should design CAs that conform to or violate stereotypical expectations. Despite the urgency and importance of navigating through this dilemma, how to better design the gender identity of CA is still an open research question. In this thesis, we describe the problem space of CA identity design and argue that we can calibrate the gender effect by manipulating a metaphor we attach to the CAs. To this end, we approached these research questions from three angles: (1) developing a framework to address the ethical dilemma in CA identity design, (2) evaluating the effect of gender and metaphor in chatbot profiles, and (3) calibrating gender stereotyping through metaphor manipulation.