Enabling the Real-Time Integration of Big and Open Linked Data (BOLD) for decision-making

Abstract

Research shows that CIOs of both public and private organizations are searching for new ways to support better outcomes of their data-driven decision-making. As linking external data with internal company data can result in interesting new insights and value, CIOs are increasingly developing IT strategies that involve the real-time use of external data. One of the innovations that is being adopted in real-time to enable better informed decision-making is external Big and Open Linked Data (BOLD). Combining internal company data silos with external BOLD in real-time helps in the creation of these real-time insights, such as new relationships, patterns, or more context to existing data that was not yet useful. These new insights can be used for better decision-making on a variety of problems and opportunities, such as strengthening customer relationship, better pricing, building a better product market fit, expanding products and services, and managing risk. However, the adoption of external BOLD addresses several adoption barriers that hinder the real-time integration of external BOLD with internal company data. The presence of adoption barriers show that there is a lack of alignment across the different levels of an organization and between the organizations concerned with the data integration. A common practice to align a new strategy and its implementation across the organization is Enterprise Architecture (EA). Countless Enterprise Architecture Frameworks (EAFs) are in existence, but none of them seems to focus on aligning an organization with its business ecosystem. This study shows that the foundation of EA is not capable of satisfying the external mindset of organizations and that an extended approach is required to align a new strategy between organizations. Therefore, this thesis answered the following main research question:
How can organizations enable the real-time integration of external Big and Open Linked Data with internal company data for decision-making?
This thesis used a Design Science Approach to develop an extended EAF that enables the real-time integration of BOLD for decision-making. Throughout this thesis five sub-research questions are answered to derive to the answer of the main research question. First, two systematic literature reviews are conducted to explicate the research problem by identifying the adoption barriers of BOLD and the limitations of the foundation of EA. Secondly, explorative expert interviews and qualitative data analysis are conducted to define the requirements for an extended EAF that deals with the adoption barriers and enables the real-time integration of external BOLD with internal company data. Thirdly, several systems engineering techniques (Morphological chart, Best-of-class chart, and brainstorm sessions with creative methods) are used to transform the requirements into an extended EAF. Lastly, the extended EAF is evaluated on quality and suitability through a ‘Plan Evaluation’, which resulted in potential improvements of the design. The findings are synthesized for answering the main research question.

To explicate the misalignment across organizations and between organizations, the different barriers that hinder the real-time adoption of BOLD are identified. This resulted in a conceptual model of adoption barriers of BOLD related to information quality (data absence, data quality, a lack of meta-data), task complexity (task handling issues, lack of standards and different sources), technology (Lack of supporting infrastructure, legacy systems, data fragmentation), use & participation (Lack of knowledge, implementation issues, constraints, lack of acceptance), legislations (privacy, security issues, and permits and license issues) and governance (Policy issues, Scalability Issues and a Lack of Data Governance). If organizations want to use external BOLD for decision-making in real-time, no adoption barriers can be present. This study shows that the foundation of EA can align a new IT strategy and its implementation across the organization, but not of aligning a new strategy between organizations and its ecosystem. This means the practice can mitigate the adoption barriers that arise across an organization can be overcome, but not the adoption barriers that arise between organizations.

This thesis shows that a general transformation from a system of record to a system of engagement is required to adapt to this external mindset, which is key part of the paradigm shift that comes with an organization’s digital transformation. Modern organizations should embrace the system of engagement, including the company’s ecosystem as part of their Enterprise Architecture Framework. Hence, this thesis presents the ‘Ecosystem Architecture Framework’ that facilitates this transformation and has incorporated the required capabilities to mitigate the adoption barriers that arise between an organization and its ecosystem. The Ecosystem Architecture Framework provides guidance for organizations to implement an IT strategy that involves the real-time integration of external BOLD on an institutional, technological and transactional level. First, the framework provides a proactive approach that initially deals with the potential adoption barriers. Secondly, the framework enables collaboration between the data users and BOLD providers by establishing strategic alliances with mutual incentives to mitigate the adoption barriers. Thirdly, the framework allows organizations to involve all stakeholders concerned with the BOLD integration in the design of the system. By developing a shared architecture vision and architectural requirements alignment between the organization and ecosystem is created. Fourthly, the framework enables organizations to initially link their architectures through a direct real-time connection. This study demonstrates that organizations need to invest in a hybrid integration platform (e.g. an API Management Platform) that facilitates that organizations can easily integrate and share integration services (e.g. APIs), different applications can be linked together, collaboration between applications is possible, and orchestration of the applications by a single interface. Fifthly, the framework looks beyond the technical implementation and ensures that an inter-organizational model developed to make the architecture working. The inter-organizational model consists of shared artifacts that make sure the same way of working is implemented within the business ecosystem. This can be achieved by developing uniform standards for data, meta-data, technical information about integration services; A common data model that ensures one data model, unified schema and semantics are used within the business ecosystem; An independent API policy that ensures that each external data provider can be internally linked to the architecture by conforming to this policy; and institutions between the organizations that enforce the same way of working to enable sustainability of the data. Lastly, the framework ensures that the architecture keeps running by adapting to changes of the environment. Therefore, it is important that organizations develop a governance mechanism that ensures the system is scalable, maintainable and sustainable. It is important that the inter-organizational model and shared artifacts of the ecosystem are managed and adapt to changes of the environment to keep the architecture running.

The outcomes of this research have both scientific and societal relevance. The former is achieved by adding the following contributions to science: new adoption barriers of BOLD; a critical assessment of the foundation of EA and EAFs; the required capabilities to mitigate the adoption barriers; and an approach that can be used to enable the real-time integration of BOLD with internal company data. The latter is achieved by providing a validated approach that can help to increase the success rate of data-driven strategies, which is relevant for national governments and organizations that want the data-driven strategies to succeed. The thesis also has some limitations. Firstly, the Ecosystem Architecture Framework is only validated trough a plan evaluation, which is a ‘design on paper’. Secondly, the framework is dependent on the TOGAF-ADM and can only be used as design principles without TOGAF-ADM. Also, the report has become very lengthy, as the research covers many different research methods.

For future research, it is recommended to also validate the design through a ‘product- and process evaluation’ that investigate the implementation issues and effects in a real-world context. Secondly, it is recommended to do future research on additional capabilities to create an EAF that can be used independently. Thirdly, an in-depth study on the specific capabilities is recommended to investigate the different design patterns for incorporating the capabilities. Fourthly, it would be interesting to do research on engaging the Ecosystem Architecture Framework in different industries with different organizations part of different ecosystems and evaluate the differences.