Efficient adiabatic control schemes, where one steers a quantum system along an adiabatic path ensuring minimal excitations while achieving a desired final state, that enable fast, high-fidelity operations are essential for any practical quantum computation. However, current optimization protocols are not universally tractable due to stringent requirements imposed by the microscopic systems encoding the qubit, including complex energy level structures and unwanted transitions, and generally require a trade-off between speed and fidelity of the operation. Here, we address these challenges by developing a general framework for optimal control based on the quantum metric tensor. This framework allows for fast and high-fidelity adiabatic pulses, even for a dense energy spectrum, based solely on the Hamiltonian of the system instead of the time evolution propagator and independent of the size of the underlying Hilbert space. Furthermore, our framework suppresses diabatic transitions and state-dependent crosstalk effects without the need for additional control fields. As an example, we study the adiabatic charge transfer in a double quantum dot to find optimal control pulses with improved performance. We show that for the geometric protocol, the transfer fidelities are lower bounded F>99% for ultrafast 20ns pulses, regardless of the size of the anti-crossing, while being robust against miscalibration errors and quasistatic noise.