Reinforcement learning (RL) has gained wide attention, but its implementation in autonomous vehicles is still limited by insufficient sample efficiency and heavy training costs. The training efficiency of RL agents is influenced by the dimension of the state space, which can be partitioned to reduce the complexity of sampling and computation. This study proposes a hierarchical clustering-based state grouping reinforcement learning (HCSG-RL) method for the switching decision of autonomous vehicles. First, we partition the base state space into groups and generate a hierarchical tree of state space groups. Then, we train multiple sub-agents for each node in the hierarchical tree. Finally, we add these trained-well sub-model into master policy. This method allows us to fully explore all state spaces and improve the training efficiency of individual agents, which handles the 'long-tail' issue and the curse of dimensionality issue. We conduct experiments in a simulation environment and results show that the proposed method has 16-72% reward improvement compared to the tree model in different road length.