Large Language Models have attracted global attention due to their capabilities in understanding, knowledge synthesis, and generating contextually relevant responses, mimicking certain aspects of human reasoning. Although LLMs have demonstrated feasibility in performing autonomous driving tasks in simulated and real-world environments, little is known about their safety and ethical decision-making. To address these questions, we propose a novel framework for evaluating and interpreting the ethical decision-making mechanism of LLM-driven autonomous vehicles. Our study investigates the ethical dilemma of prioritizing saving pedestrians or passengers inspired by the Moral Machine Experiment. We used a stated preference survey to include factors of group size, age, gender, fatality risk, and pedestrian behavior to create 13,122 choice scenarios (a full factorial design) to analyze responses from advanced LLMs, including the GPT-4 series models from OpenAI and Mistral-Large from Mistral AI. Our findings reveal significant differences in the decision-making process and preferences for saving road users among these LLMs. Using a binary logit model to interpret GPT-4′s decisions, we found that the estimated number of deaths, age, and gender significantly affect the model's choices. The decision tree method was also applied to analyze LLMs’ decision-making processes, uncovering potential ethical standards and conditions considered by the models. This study provides valuable insights into ethical considerations in AI systems and thus facilitates the responsible development of AI in autonomous vehicles.