Tech-leading organizations are embracing the forthcoming artificial intelligence revolution. Intelligent systems are replacing and cooperating with traditional software components. Thus, the same development processes and standards in software engineering ought to be complied in artificial intelligence systems. This study aims to understand the processes by which artificial intelligence-based systems are developed and how state-of-the-art lifecycle models fit the current needs of the industry. We conducted an exploratory case study at ING, a global bank with a strong European base. We interviewed 17 people with different roles and from different departments within the organization. We have found that the following stages have been overlooked by previous lifecycle models: data collection, feasibility study, documentation, model monitoring, and model risk assessment. Our work shows that the real challenges of applying Machine Learning go much beyond sophisticated learning algorithms – more focus is needed on the entire lifecycle. In particular, regardless of the existing development tools for Machine Learning, we observe that they are still not meeting the particularities of this field.

In 2014, a Microsoft study investigated the sort of questions that data science applied to software engineering should answer. This resulted in 145 questions that developers considered relevant for data scientists to answer, thus providing a research agenda to the community. Fast forward to five years, no further studies investigated whether the questions from the software engineers at Microsoft hold for other software companies, including software-intensive companies with different primary focus (to which we refer as software-defined enterprises). Furthermore, it is not evident that the problems identified five years ago are still applicable, given the technological advances in software engineering. This paper presents a study at ING, a software-defined enterprise in banking in which over 15,000 IT staff provides in-house software solutions. This paper presents a comprehensive guide of questions for data scientists selected from the previous study at Microsoft along with our current work at ING. We replicated the original Microsoft study at ING, looking for questions that impact both software companies and software-defined enterprises and continue to impact software engineering. We also add new questions that emerged from differences in the context of the two companies and the five years gap in between. Our results show that software engineering questions for data scientists in the software-defined enterprise are largely similar to the software company, albeit with exceptions. We hope that the software engineering research community builds on the new list of questions to create a useful body of knowledge.

The appeal of delivering new features faster has led many software projects to adopt rapid releases. However, it is not well understood what the effects of this practice are. This paper presents an exploratory case study of rapid releases at ING, a large banking company that develops software solutions in-house, to characterize rapid releases. Since 2011, ING has shifted to a rapid release model. This switch has resulted in a mixed environment of 611 teams releasing relatively fast and slow. We followed a mixed-methods approach in which we conducted a survey with 461 participants and corroborated their perceptions with 2 years of code quality data and 1 year of release delay data. Our research shows that: rapid releases are more commonly delayed than their non-rapid counterparts, however, rapid releases have shorter delays; rapid releases can be beneficial in terms of reviewing and user-perceived quality; rapidly released software tends to have a higher code churn, a higher test coverage and a lower average complexity; challenges in rapid releases are related to managing dependencies and certain code aspects, e.g. design debt.

The development of Cloud Infra-Services has shifted over the past decade in the direction of a software code development process, also known as infrastructure as code (IaC). Contemporary continuous delivery settings in industry require fast feedback. As a consequence, companies need insight in time spent, especially in the development of such services. We examine a series of 28 Cloud Infra-Services within ING, and explore which factors affect their overall time to market and development time. An initial perception among several stakeholders in the Cloud Infra-Service development process, that Cloud Infra-Services within ING take longer than those in peer companies, is not confirmed by our benchmark. Development team members identified the time to internal market of services to be affected negatively by the portal where consumers can order a service and the Orchestration Workflows and by team dynamics. This perception is supported by additional metrics. We propose that promising ways to reduce lead time include reducing the complexity of the ING environment, by treating Cloud Infra-Services like regular software deliveries and by reducing the dependencies between teams in terms of tooling and collaboration.

Background: During the period of one year, ING developed an approach for software analytics within an environment of a large number of software engineering teams working in a Continuous Delivery as a Service setting. Goal: Our objective is to examine what factors helped and hindered the implementation of software analytics in such an environment, in order to improve future software analytics activities. Method: We analyzed artifacts delivered by the software analytics project, and performed semi-structured interviews with 15 stakeholders. Results: We identified 16 factors that helped the implementation of software analytics, and 20 factors that hindered the project. Conclusions: Upfront defining and communicating the aims, standardization of data at an early stage, build efficient visualizations, and an empirical approach help companies to improve software analytics projects.

Based on the large amounts spent by software companies to develop new and existing software systems, we argue that an evidence-based approach that focuses on a software portfolio as a whole should be in place to support decision-making. We developed EBSPM as an evidence-based, practical model to support software companies to actively steer at optimization of their software delivery portfolio. We evaluated the model in case studies and surveys in industry, to demonstrate its strengths and limitations in practice. This lead to the following results:
• We analyzed - from a portfolio point of view - the characteristics of best performers and worst performers, in a dataset of 352 software projects, resulting in 7 success factors and 9 failure factors.
• We found that a release process that performs above average on cost and duration, satisfies stakeholders through fast response and direct value, even when the reliability and availability of the actual system are weak.
• A statistical, evidence-based pricing approach for software engineering, as a single instrument, can be used in the subject companies to create cost transparency and performance management.
• We found significant differences between the EBSPM-repository and an ISBSG-subset. Practitioners and researchers alike should be cautious when drawing conclusions from a single repository.
• We found that a focus on shortening overall project duration and improving communication and team collaboration on intermediate progress is likely to have a positive impact on stakeholder satisfaction and perceived value.
Based on the findings, we conclude that it is wise for software companies to collect and analyze their own historic software portfolio data because cross-company large differences in performance are found. We obtained a better understanding of the differences and equalities between effort and cost of software deliveries. Additionally, we studied the effects of pricing of software deliveries, giving us a better insight into ways to support decision-making. Based on the results of ongoing research, we expect that automation of the measurement and analysis process, based on statistics to calculate strong relationships, is a direction in which the analysis of software portfolio (software analytics) is the to develop strongly in the coming years.

Context: In this paper we present a multiple case study on the insights of software organizations into stakeholder satisfaction and (perceived) value of their software projects. Our study is based on the notion that quantifying and qualifying project size, cost, duration, defects, and estimation accuracy needs to be done in relation with stakeholder satisfaction and perceived value. Objectives: We contrast project metrics such as cost, duration, number of defects and estimation accuracy with stakeholder satisfaction and perceived value. Method: In order to find out whether our approach is practically feasible in an industrial setting, we performed two case studies; one in a Belgian telecom company and the other in a Dutch software company. Results: In this study we evaluate 22 software projects that were delivered during one release in the Belgian telecom company, and 4 additional large software releases (representing an extension of 174% in project size) that were delivered in a Dutch software company. Eighty-three (83) key stakeholders of two companies provide stakeholder satisfaction and perceived value measurements in 133 completed surveys. Conclusions: We conclude that a focus on shortening overall project duration, and improving communication and team collaboration on intermediate progress is likely to have a positive impact on stakeholder satisfaction and perceived value. Our study does not provide any evidence that steering on costs helped to improve these. As an answer to our research question - how do stakeholder satisfaction and perceived value relate to cost, duration, defects, size and estimation accuracy of software projects? - we found five take-away-messages.

Research Repository used to develop the Evidence-Based Software Portfolio Management approach. Holding data of approx. 500 finalized software projects from 4 different companies in The Netherlands and Belgium.

Context: The research literature on software development projects usually assumes that effort is a good proxy for cost. Practice, however, suggests that there are circumstances in which costs and effort should be distinguished. Objectives: We determine similar-ities and differences between size, effort, cost, duration, and num-ber of defects of software projects. Method: We compare two es-tablished repositories (ISBSG and EBSPM) comprising almost 700 projects from industry. Results: We demonstrate a (log)-linear relation between cost on the one hand, and size, duration and number of defects on the other. This justifies conducting linear regression for cost. We establish that ISBSG is substantially differ-ent from EBSPM, in terms of cost (cheaper) and duration (faster), and the relation between cost and effort. We show that while in ISBSG effort is the most important cost factor, this is not the case in other repositories, such as EBSPM in which size is the dominant factor. Conclusion: Practitioners and researchers alike should be cautious when drawing conclusions from a single repository.

ING Bank, a large Netherlands-based internationally operating bank, implemented a fully automated continuous delivery pipeline for its software engineering activities in more than 300 teams, that perform more than 2500 deployments to production each month on more than 750 different applications. Our objective is to examine how strong metrics for agile (Scrum) DevOps teams can be set in an iterative fashion. We perform an exploratory case study that focuses on the classification based on predictive power of software metrics, in which we analyze log data derived from two initial sources within this pipeline. We analyzed a subset of 16 metrics from 59 squads. We identified two lagging metrics and assessed four leading metrics to be strong.

Context: In this paper we describe and evaluate a tool for Evidence-Based Software Portfolio Management (EBSPM) that we developed over time in close cooperation with software practitioners from The Netherlands and Belgium. Objectives: The goal of the EBSPM-tool is to measure, analyze, and benchmark the performance of interconnected sets of software projects in terms of size, cost, duration, and number of defects, in order to support innovation of a company's software delivery capability. The tool supports building and maintaining a research repository of finalized software projects from different companies, business domains, and delivery approaches. Method: The tool consists of two parts. First, a Research Repository, at this moment holding data of for now 490 finalized software projects, from four different companies. Second, a Performance Dashboard, built from a so-called Cost Duration Matrix. Results: We evaluated the tool by describing its use in two practical applications in case studies in industry. Conclusions: We show that the EBSPM-tool can be used successfully in an industrial context, especially regarding its benchmarking and visualization purposes.

Context: In this paper we present an exploratory study on the insights of organizations into the perceived value of their software projects. Our study is based on the notion that quantifying and qualifying project size, cost, duration and defects needs to be done in relation with stakeholder satisfaction and perceived value. Objectives: We expect that bringing perceived value into the equation will help in increasing the impact such organizations deliver. Method: In order to find out whether our approach is practically feasible in an industrial setting, we performed an exploratory study in a Belgian telecom company. Results: In this study we evaluate 22 software projects that were delivered during one release. Fiftythree (53) key stakeholders provide stakeholder satisfaction and perceived value measurements in 103 completed surveys. Conclusions: We conclude that a focus on shortening overall project duration, and improving communication on intermediate progress improved stakeholder satisfaction and perceived value. Our study does not provide any evidence that steering on costs helped to improve these.

In this paper we explore opportunities, challenges, and obstacles that Functional Size Measurement (FSM) experts assume to be in automatically derived functional size, directly from the software project code itself. We designed a structured survey, that was answered by 336 FSM specialists. A majority of the respondents consider FSM to be an important tool for decision making. No indications are found for any perceived impact of agile methodology on the difficulty of applying FSM. Respondents overall think of automated FSM as important, but also difficult to realize. 54% of the respondents think that automated FSM will help measurement specialists, while 44% thinks that it will help decision makers too. The most preferred FSM method for automation is COSMIC (25%), followed by IFPUG (21%) and Nesma (16%). Respondents perceive automated FSM to be most suitable for baselining, benchmarking, and maintenance and legacy purposes.

In this paper, we attempt to understand what contributes to a successful process for managing legacy system evolution. We provide an analysis of a number of key performance indicators such as cost, duration, and defects. By normalizing through function points, we furthermore compare to a larger benchmark. To do so we performed a mixed, retrospective case study on a series of nine software releases and eight single once-only releases, all performing on a single, legacy software system, in a West-European telecom company. We interviewed eleven stakeholders that were closely involved in the subject software releases. As a result, we listed a number of observations from the quantitative and qualitative analysis. We found that a release process that performs above average on cost and duration satisfies stakeholders through fast response and direct value, even when the reliability and availability of the actual system is weak.

We study whether an assumed positively skewed distribution of effort data prevents software estimators to learn over time; leading to increasing differences between planned and actual effort and a deteriorating (worsening) trend on productivity. We analyze data of 25 software releases of one application, collected over a period of six years in a public sector institution in The Netherlands. We statistically test for distribution, trend on differences between planned versus actual effort over time, and productivity of software portfolios. The key contributions of this paper are that we show that a proposed assumption that assumes any relation between a positively skewed distribution of effort data and a deteriorating productivity is not applicable to the subject dataset. We find that the effort data is to be characterized as positively skewed distributed, and we do see a shift over time from under-estimation to over-estimation. We do not find evidence for a deteriorating productivity; on the contrary productivity improves over time, indicating that estimators in the subject organization did learn.

A medium-sized west-European telecom company experienced a worsening trend in performance, indicating that the organization did not learn from history, in combination with much time and energy spent on preparation and review of project proposals. In order to create more transparency in the supplier proposal pro-cess a pilot was started on Functional Size Measurement pricing (FSM-pricing). In this paper we evaluate the implementation of FSM-pricing in the software engineering domain of the company, as an instrument useful in the context of software management and supplier proposal pricing. We analyzed 77 finalized software engineering projects, covering 14 million Euro project cost and a project portfolio size of more than 5,000 function points. We found that a statistical, evidence-based pricing approach for software engineering, as a single instrument (without a connec-tion with expert judgment), can be used in the subject companies to create cost transparency and performance management of software project portfolios.

What can we learn from historic data that is collected in three software companies that on a daily basis had to cope with highly complex project portfolios? In this paper we analyze a large dataset, containing 352 finalized software engineering projects, with the goal to discover what factors affect software project performance, and what actions can be taken to increase project performance when building a software project portfolio. The software projects were classified in four quadrants of a Cost/Duration matrix: analysis was performed on factors that were strongly related to two of those quadrants, Good Practices and Bad Practices. A ranking was performed on the factors based on statistical significance. The paper results in an inventory of 'what factors should be embraced when building a project portfolio?' (Success Factors), and 'what factors should be avoided when doing so?' (Failure Factors). The major contribution of this paper is that it analyzes characteristics of best performers and worst performers in the dataset of software projects, resulting in 7 Success Factors (a.o. steady heartbeat, a fixed, experienced team, agile (Scrum), and release-based), and 9 Failure Factors (a.o. once-only project, dependencies with other systems, technology driven, and rules- and regulations driven).

In this research we aimed to identify distinguishing factors in software releases. For this purpose we analyzed the metrics of 26 software projects. These projects were release-based deliveries from two stable, experienced development teams. During the measurement period both teams transformed from a plan-driven delivery model (waterfall) to an agile approach (Scrum). Overall, we observed that these small release-based projects differ largely from non-release-based projects. Our research indicates that a combination of release-based working, a fixed and experienced development team, and a steady heartbeat contribute to performances that can be characterized as best-practice. The main contribution of this paper is that we found five success factors (all reducing development complexity) that result in best-of-class performance for small software releases.