Overcoming the Barriers to Large Language Model (LLM) Adoption

Overcoming the Barriers to Large Language Model (LLM) Adoption: A study on Organisations’ Perceived Risks of LLMs

Kooij, Matthew (TU Delft Technology, Policy and Management)

Zhauniarovich, Y. (mentor)
Klenk, M.B.O.T. (mentor)
van der Voort, H.G. (graduation committee)

Delft University of Technology

Management of Technology (MoT)

2024-05-27

Generative AI is expected to have a significant impact on how business is done, especially for knowledge-intensive domains. Professional usage amongst knowledge workers is already widespread and many of them believe that LLM use for work will make them more efficient, help them generate ideas, and improve the quality of their work. Yet, the technology comes with numerous potential risks, including job displacement, threats to data privacy, unreliability, cybersecurity risks, and non-compliance with new AI regulation. The decision to adopt LLMs for knowledge work is thus a significant one as organisations must carefully weigh up the benefits and drawbacks that these models present.

Using a qualitative research approach, this study was aimed at exploring employee and organisational perceptions on the benefits and risks of LLM adoption within the Dutch financial sector. This study was conducted in partnership with a global Professional Services Firm and used their existing network of people and clients to conduct interviews with experts who act as advisors to top management in the decision-making process of new technology adoption such as LLMs. 18 semi-structured interviews were done to collect qualitative data. The research question to be answered is:

- How do employees’ and organisations’ perceived benefits and risks of Large Language Models (LLMs) influence financial organisations’ LLM adoption plans?

Exactly half of participants (9/18) said that they use LLMs in their own work while the other half stated that they do not due to a lack of perceived benefits. Microsoft was found to be a significant player in the current adoption of LLMs at Dutch financial institutions with 57% of all LLMs used by participants being owned by the software vendor. The most common LLM use cases were literary and creative in nature and included preparing presentation slides (16%), text generation (12%), email composition (12%), and structuring documents (12%).

The targeted capabilities that organisations would like to achieve with their future adoption plans include helping programmers write better code, analysing help desk conversations, assisting employees via LLM chatbots, and analysing emails to predict customer questions. Employees were most excited about the potential efficiency and productivity gains that LLMs offer for their work (19%), how LLM usage could free up more time for focused, interesting, and fun work (11%), repetitive tasks becoming automated by LLM (8%), the new opportunities that LLMs present (8%) such as new business models, and improvements to customer service (5%). General overlap can be seen between the targeted capabilities of future LLMs to be adopted and employee expectations.

Instead of restricting LLM usage, it is recommended that organisations find ways to incorporate them into their employee workflows by providing clear policies and guidelines. It was found that this approach will be most beneficial to creative and literary workflows like improving writing/grammar/translation, information search, structuring documents, and text generation. To achieve this integration, organisations should write and implement clear usage policies. This will ensure that the benefits of allowing LLM usage at work are enjoyed while mitigating the most important risks.

Generative AI
Large Language Models (LLMs)
Perceived risk
Technology Adoption

http://resolver.tudelft.nl/uuid:2182d7af-9ef6-470f-9bde-a3f8962d171c

Student theses

master thesis

© 2024 Matthew Kooij