This dissertation focuses on the research problem of divergence between the theoretical promise of big data use for public policymaking and the empirical support for that promise. To address this problem the dissertation asks three sequential research questions: Firstly, why does this divergence exist. Secondly, how can it be improved. And lastly, how to design and carry out research capable of these improvements. These questions form the theoretical core of the dissertation and effectively propose a research approach to studying big data use in public policymaking. However, merits of a research approach can only be demonstrated by research that adopts it, which his why this dissertation also researches a more practical problem: The difficulty of measuring and evaluating ‘social investment’ policies. The dissertation asks “Can social media data be used to operationalize and measure social investment?”, which involves a set of research sub-questions focusing on how can requisite information be extracted, what such extraction implies for policymaking, whether the requisite information is present in Twitter data, and whether it changes between a period of normalcy and a period of crisis. Answering this research question in a more design-oriented proof-of-concept way, but while utilizing the proposed research approach, is relevant both for the research problem of operationalizing social investment as well as demonstrating the merits and pitfalls of the research approach proposed as a solution to the primary research problem.

Despite great potential, high hopes and big promises, the actual impact of big data on the public sector is not always as transformative as the literature would suggest. In this paper, we ascribe this predicament to an overly strong emphasis the current literature places on technical-rational factors at the expense of political decision-making factors. We express these two different emphases as two archetypical narratives and use those to illustrate that some political decision-making factors should be taken seriously by critiquing some of the core ‘techno-optimist’ tenets from a more ‘policy-pessimist’ angle. In the conclusion we have these two narratives meet ‘eye-to-eye’, facilitating a more systematized interrogation of big data promises and shortcomings in further research, paying appropriate attention to both technical-rational and political decision-making factors. We finish by offering a realist rejoinder of these two narratives, allowing for more context-specific scrutiny and balancing both technical-rational and political decision-making concerns, resulting in more realistic expectations about using big data for policymaking in practice.