Temporary railway timetables are implemented during periods of disrupted operations, such as planned track closures, and operate under constrained conditions. Temporary timetables lack quantitative methods to assess their overall performance and systematic comparison. This paper introduces an approach to quantify the effectiveness of temporary railway timetables. Timetable effectiveness is defined as the ability of a temporary timetable operating under constrained conditions to maintain operable, stable, and robust train operations under deterministic and stochastic variability. Based on this definition, a multi-criteria evaluation framework is proposed that integrates four performance indices: capacity utilization, stability, robustness, and operability. Existing timetable attributes are reformulated to reflect constrained operating conditions. In addition, a new timetable attribute, operability, is introduced to capture the executability of infeasible timetables. The framework aggregates the indices into a single Temporary Timetable Effectiveness (𝑇𝑇𝐸) indicator. The proposed methodology is demonstrated through a real-world case study of a planned track closure on the Dutch railway network. Multiple temporary timetables are quantified and compared using microscopic simulation. The results show that the framework is capable of distinguishing both marginal and substantial differences between alternative timetable designs and making trade-offs between performance indices explicit. The proposed framework provides a systematic quantitative assessment and comparison of temporary railway timetables.