Mental Workload (MWL) is a construct widely used in HCI to assess the cognitive demand users must exert to perform a task. Research in human factors, however, has suggested several issues regarding its definitions, scales, and applications. This paper, first, introduces debates surrounding the MWL concept and its most popular measure, the NASA-TLX. We present a systematic review of CHI papers involving MWL and highlight severe issues in its application. Finally, through a validation experiment, we assess the convergent validity and sensitivity of two MWL instruments—NASA-TLX and MRQ. Our findings reveal disagreements in the definitions of MWL and severe drawbacks in NASA-TLX and its applications. Our validation study also presents evidence for a lack of convergent validity and sensitivity of MWL subjective scales in HCI tasks. Our findings recommend caution when employing NASA-TLX in user studies and highlight the need for an MWL definition that is agreed upon within the HCI community.