Multi-Interpretive Multi-Objective Reinforcement Learning

Abstract

Aligning AI systems with human values is a multi-objective problem, since values such as helpfulness and harmlessness often conflict and the tradeoffs between them must be balanced. Multi-objective reinforcement learning (MORL) addresses this by learning sets of policies that represent these tradeoffs. Values, however, do not come with a single agreed meaning. Objectives such as fairness or professionalism are essentially contested, and different people, groups, and normative theories operationalize the same objective into different reward functions. Existing alignment approaches either commit to one interpretation before training, average conflicting interpretations into a single signal, or represent groups with single-objective reward models. In each case, either the plurality of interpretations or the multi-objective structure of the problem is lost.

This thesis introduces multi-interpretive multi-objective reinforcement learning (MIMORL), which treats the interpretation of the objectives as an explicit dimension of the learning problem. The underlying multi-interpretive multi-objective Markov decision process (MIMOMDP) carries a matrix-valued reward function in which each row is one interpretation of the objectives. Under the assumption that a single theory or opinion guides the interpretation of all objectives together, the preference space factors into a preference over objectives and a preference over interpretations, so that its size grows linearly rather than exponentially with the number of interpretations. Two algorithms are proposed. The outer-loop method trains one multi-objective sub-solver per interpretation and combines their solutions through an interpretation weight, while the inner-loop method conditions a single network on both preference vectors.

Both methods are evaluated in a modified firefighters environment, in which two defensible interpretations of the values professionalism and proximity induce different optimal behaviour, and are compared against specialist, scalarized, and flat multi-objective baselines as well as exactly computed reference solution sets. Both MIMORL variants recover the reference sets on the pure interpretations and on all evaluated mixtures between them, matching the strongest baseline while remaining steerable across interpretations at inference time, with a factored formulation that is theoretically more efficient than the baselines as the number of interpretations grows. A single trained agent can therefore be deployed and steered to different interpretations, or to positions between them, without committing to one reading of its values in advance, which constitutes a step towards pluralistic AI alignment.