This paper discusses the approach for architecture design space optimization of aeronautical systems investigated in the DEFAINE project. In system architecture optimization problems, hierarchical relations between design variables may exist. This means that the quantity and type of some variables are dependent on the value of other variables. To address this challenge, a nested optimization strategy is proposed, where an outer loop deals with the independent design variables and an inner loop with the dependent ones. As the characteristics of the dependent variables may become known only during workflow execution, a dynamic workflow formulation approach is developed to introduce these variables into the MDAO workflow, automatically, during execution. The proposed methodology is implemented using a suite of technologies provided by partners of the DEFAINE consortium to enable formulation and execution of collaborative MDAO problems. This implementation is then applied to the design and optimization of the structural layout of a UAV aileron. The goal is to minimize the aileron mass, subjected to failure criteria constraints. In this use case, hierarchical relations between variables are present. The aileron skin is discretised into material zones, each one defined by a (dependent) design variable. The number of zones, hence the number of design variables, is not known a priori as it depends on the number and position of ribs and spars(both independent variables). A preliminary implementation of the proposed approach proved effective in dealing with the hierarchical mixed-integer design space. Trade off studies comparing different aileron architectures could be efficiently set up and executed. Some limitations in the implemented workflow management technology, however prevent performing a full nested architecture optimization. Therefore, to demonstrate the effectiveness of the proposed nested approach, a parallel study was conducted on the same aileron, using a less generic and manual implementation of the nested MDAO workflow. This second study successfully minimized the mass and cost of the aileron structure, thereby proving the merit of the proposed nested approach to address system architecture optimization.