Climate change is a heated discussion topic in public arenas such as social media. Both texts and visuals play key roles in the debate, as they can complement, contradict, or reinforce each other in nuanced ways. It is therefore urgently needed to study the messages as multimodal objects to better understand the polarized debate about climate change impacts and policies. Multimodal representation models such as CLIP are known to be able to transfer knowledge across domains and modalities, enabling the investigation of textual and visual semantics together. Yet they are not directly able to distinguish the nuances between supporting and sceptic climate change stances. This paper explores a simple but effective strategy combining modality fusion and domain-knowledge enhancing to prepare CLIP-based models with knowledge of climate change stances. A multimodal Dutch Twitter dataset is collected and experimented with the proposed strategy, which increased the macro-average F1 score across stances from 51% to 86%. The outcomes can be applied in both data science and public policy studies, to better analyse how the combined use of texts and visuals generates meanings during debates, in the context of climate change and beyond.

The influence of visualizations on decision-making about controversial policy issues is increasingly recognized in the political and policy sciences. In this paper, we explore how combinations of visuals and text on Twitter (now X) lead to the formation of networks of actors sharing similar textual and visual framings about a policy issue in an online setting, which we conceptualize as Multimodal Online Discourse Coalitions (MODCs). MODCs struggle over the meaning of contested policy issues. We examine multiple MODCs in 2018 in the context of the regulatory decisions in that year about CRISPR-Cas gene editing technology in the USA, Mercosur, and the EU. Based on an SNA and a qualitative visual and discursive analysis in three languages on Twitter in 2018 (covering in total ~ 427 k Tweets), we show that MODCs in English and Spanish focused on technocratic aspects of CRISPR-Cas, resembling the regulatory decisions in the USA and Mercosur. In Europe, next to technocratic MODCs, an MODC in French formed around ethical/normative framings of the consequences of CRISPR-Cas applications, using visuals of embryos to represent “GMO babies.” These visuals were emotional triggers in their framing of CRISPR technology. The ethical/normative framing reflected the argument brought to the CJEU by a group of French actors involved in the court case which categorized CRISPR-Cas as a GMO technology in the EU. These results suggest that the French MODC and their visualization was of influence on the EU decision-making process; however, more research is needed to verify the role of this online debate in the decision-making process.