Smartphones, or more general handhelds, which are commonly used for indoor localization purposes, are not a viable option in places where people do not carry them all the time, e.g., home and office. Alternatively using wearable devices brings along limitations in regards to power supply, processing capability and availability of sensors, which prevents the adoption of many common handheld localization solutions. In this work a distributed localization system is presented, using wearable and handheld jointly, to address these drawbacks. Using only a magnetometer, an accelerometer, and Bluetooth radio, localization is performed by means of a particle filter. In addition, a smart handoff mechanism is presented, which uses the wearable only when it is necessary, thus reduces energy consumption on the wearable without affecting the desired location accuracy. Evaluating the system with ten participants, a localization accuracy of 90.31% in an indoor environment spanning about 320 square meters was achieved.