Reference study of IFC software support: The GeoBIM benchmark 2019—Part I

Industry Foundation Classes (IFC), the buildingSMART open standard for BIM, is underused with respect to its promising potential, since, according to the experience of practitioners and researchers working with BIM, issues in the standard’s implementation and use prevent its effective use. Nevertheless, a systematic investigation of these issues has never been carried out, and there is thus insufficient evidence for tackling the problems. The GeoBIM benchmark project is aimed at finding such evidence by involving external volunteers, reporting on various aspects of the behavior of tools (geometry, semantics, georeferencing, functionalities), analyzed and described in this article. Interestingly, different IFC software programs with the same standardized data sets yield inconsistent results, with few detectable common patterns, and significant issues are found in their support of the standard, probably due to the very high complexity of the standard data model. A companion article (Part II) describes the results of the benchmark related to CityGML, the counterpart of IFC within geoinformation.


Introduction
In the Architecture, Engineering and Construction (AEC) fields, as well as in further disciplines, interoperability is of increasing importance, in order to enable re-use and exchange of data and information, including in the strictly related asset and facility management field.Furthermore, it is essential for reciprocal integration of data having different nature.One of the current research topics is, for example, the integration of building information models (BIMs) with 3D city models, for supporting several use cases, e.g.building permits issuing, 3D cadastre, complex assets and facility management.International standards are conceived as a solution for fostering such interoperability.The most popular standard data models, considered for obtaining such integration, are usually the OGC CityGML for 3D city models and buildingSMART Industry Foundation Classes (IFC) for BIMs.
However, although a part of the building designers and professionals, supporting openBIM are IFC enthusiasts, many others working with BIMs still seldom use IFC as a first choice for exchanging their models, and only export them to IFC when required explicitly by law, when trying to integrate with software from different vendors, or when no other solutions are available, as emerged from several personal communications.This often happens because they are aware of some limitations in the IFC format (as for example documented more than ten years ago by Pazlar and Turk (2008)), although they might acknowledge its potential as an open exchange format.
In practice, limited support for IFC might not been considered as a major issue, since some very widespread software, such as the Revit format, is often used as de facto standard for BIMs exchange among designers, as well as for integrating the BIM model with further systems (Baik et al. (2015); Kensek (2014); Petersen et al. (2018); Lv (2018); Kamari et al. (2018); Papadopoulos et al. (2017); Lamartina and Sa (2019); Aksamija (2018); Farid Mohajer and Aksamija (2019) are only few examples), while the actual use of BIMs by authorities, besides visualization, is still limited, for the moment.This solves the most immediate interoperability issues, even though the Revit format is a proprietary binary format, which hinders re-usability in an open way and across time.
A more specific study of the limitations of IFC was part of the GeoBIM benchmark project1 (see Section 2) and is reported in this paper.Within the project, the approach to the study of the support for the two standards involved in the GeoBIM integration (IFC and CityGML) was conceived in parallel, also with the aim of understanding if one of the two offered more effective solutions that could be possibly borrowed by the other one in future developments.However, the final outcomes of the two different tasks are very specific for each standard and deserve to be presented and discussed separately, considering the specificities of each case.For these reasons, this paper, focusing on the results about Task 1, the benchmark task related to the support for IFC, is written in tandem with Noardo et al. (2020), which describes Task 3 covering the support for CityGML.In order to allow each paper to be read on its own, the two papers share some information (i.e.Section 2 explaining the general context and motivation of the study about the GeoBIM benchmark set-up; Section 3.1 covering the initial part of the methodology about the entire GeoBIM benchmark set-up, and Section 3.3 concerning some similarities in the methodology).

The GeoBIM needs and the concept of this study -COMMON WITH TASK3 PAPER
Two increasingly developed, studied and used 3D information systems have revealed their potential in the related fields: • 3D city models, which are used to represent city objects in three dimensions and advance previous 2D maps and other cartographic products, in order to support city analysis and management, city planning, navigation, and so on (e.g.Biljecki et al. (2015); Kumar et al. (2017); Egusquiza et al. (2018); Jakubiec and Reinhart (2013); Liang et al. (2014); Bartie et al. (2010); Peters et al. (2015); Nguyen and Pearce (2012)); • Building Information Models (BIM), which are used in the architecture, engineering and construction fields (AEC) to design and manage buildings, infrastructure and other construction works, and which also have features useful to project and asset management (e.g.Petri et al. (2017); Haddock (2018); Azhar (2011) ).
Several international standards exist to rule the representation of the built environment in a shared way, to foster interoperability and cross border analysis and, consequently, actions, or to reuse tools, analysis methods and data themselves for research and possibly government.Some example of international standards are: the European Directive aiming at an Infrastructure for spatial information in Europe (IN-SPIRE)2 , aimed at the representation of cross border pieces of land in Europe, for common environmental analysis; the Land and Infrastructure (LandInfra) 3 , by the Open Geospatial Consortium (OGC), aimed at land and civil engineering infrastructure facilities representation; and the green building data model (gbXML) 4 , aimed at the representation of buildings for energy analysis.
Nonetheless, the two dominant reference open standards for those two models are CityGML5 , by the OGC, focusing on urban-scale representation of the built environment, and the Industry Foundation Classes (ISO, 2013)6 , by buildingSMART, aimed at the very detailed representation of buildings and other construction works for design and construction objectives, first, but also intended to enable project management throughout the process, and asset and facility management in a following phase.Those standards are both intended to be very comprehensive and are therefore very wide and articulated.They both use complex data models allowing for a wide variety of models using object-oriented representations, even if that comes at a cost of slower and more inconsistent implementations.
Due to the overlapping interests in both fields (meeting in the building-level representation), increasing attention is being paid to 3D city model-BIM integration (GeoBIM), where the exchange of information between geospatial (3D city models) and BIM sources enables the reciprocal enrichment of the two kinds of information with advantages for both fields, e.g.automatic updates of 3D city models with high-levelof-detail features, automatic representation of BIM in their context, automated tests of the design, and so on (Liu et al., 2017;Fosu et al., 2015;Aleksandrov et al., 2019;Kumar et al., 2019;Niu et al., 2019;Noardo et al., 2019a;Arroyo Ohori et al., 2018;Kang and Hong, 2015;Stouffs et al., 2018;Lim et al., 2019).
The GeoBIM subject can be divided into several sub-issues.
1. First, the harmonization of data themselves, which have to concretely fit together, with similar (or harmonizable) features (e.g.accuracy, kind of geometry, amount of detail, kind of semantics, georeferencing).
2. Second, the interoperability is a fundamental key in the integration.It is important to note here, that before enabling the interoperability among different formats (e.g.GIS formats and BIM formats), which is the object of the theme of point three below, the interoperability GIS-to-GIS and BIM-to-BIM itself is essential.That means that the formats of data have to be understood and correctly interpreted uniquely by both any person and any supporting software.Moreover, an interoperable dataset is supposed to remain altogether unchanged when going through a potentially infinite number of imports and exports by software tools, possibly converting it to their specific native formats and exporting it back.For this, it is desirable to rely on open standards.
3. Third, the effective conversion among different formats, i.e. transforming one dataset in a (standardised) format to another one in compliance with the end format specifications and features.
4. Fourth, the procedures employing 3D city models and the ones based on BIM should be changed in order to obtain better advantages by the use of both, integrated, since those systems enable processes which are usually more complex than just the simple representations.
The many challenges implied by the points above are still far from being solved, and one of the essential initial steps is actually to outline such challenges more sharply.
In particular, the second point (interoperability and involved standards) is often considered to be solved by standardization organizations.It is desirable to rely on open standards for this, because of the welldocumented specifications of open standards enable longer-term support, as well as their genericity with respect to different software vendors, as opposed to closed point-to-point solutions that merely connect one proprietary system to another (and might be discontinued or stop working at any moment).However, our previous experiences suggest that the support for open standards in software is often lacking.
The researchers promoting this study (as users of data, advocates of open standards and developers of tools adopting such standards) have noticed, over their research and work activities, how the use of those standards in data and their implementations in software were not always straightforward and not completely consistent with the standard specifications either.Many tools, when managing standardized data, do not adequately support features or functionalities as they do when the data is held in the native formats of the software.In addition, software tools have limitations with respect to the potential representation (geometry, semantics, georeferencing) of data structured following these standards, or can generate errors and erroneous representations by misinterpreting them.
The standards themselves are partly at fault here, since they often leave some details undefined, with a high degree of freedom and various possible interpretations.They allow high complexity in the organization and storage of the objects, which does not work effectively towards universal understanding, unique implementations and consistent modelling of data.This is probably due to the fact that such standards often originate as amalgamations of existing mechanisms and compromises between the various stakeholders involved.These experiences have been informally confirmed through exchanges within the scientific community and especially with the world of practitioners, who are supposed to work with (and have the most to gain from) those standardized data models and formats.However, more formal evidence on the state of implementation of these open standards and what problems could be connected to the standard themselves have not been compiled so far.
For this reason, the GeoBIM benchmark project7,8 was proposed and funded in 2019 by the International Society for Photogrammetry and Remote Sensing (ISPRS)9 and the European Association for Spatial Data Research (EuroSDR)10 .The aim of the benchmark was to get a better picture of the state of software support for the two open standards (IFC and CityGML) and the conversions between them, in order to formulate recommendations for further development of the standards and the software that implements them.In addition, we tested two known major technical issues related to GeoBIM integration and which are known to be solved only partially in practice: the ability of tools and methods to georeference IFC and the conversion procedures between IFC and CityGML.
The relevant outcomes regarding the buildingSMART IFC standard are the subject of this paper.

Industry Foundation Classes and knotty points
The buildingSMART Industry Foundation Classes (IFC) standard 11 is an open standard data model for Building Information Modelling (BIM) to be shared and exchanged through software applications, domains and use cases, within the Architecture Engineering and Construction (AEC) and Facility Management (FM) fields.It includes classes for describing both physical and abstract concepts (e.g.cost, schedule, etc.) concerning AEC-FM for buildings, mainly (recent versions are extending it for including infrastructures and other kinds of constructions)12 .It has also been adapted as the ISO 16739 international standard (ISO, 2013).
The standard includes relevant constructs for a wide variety of disciplines, use cases and processes associated to the construction domain, most prominently the semantic description and geometric representation of typical construction elements and their relationships.
The IFC is structured in a deeply hierarchical data model, furthermore organized in several and deep and complex meronymic (part-of) trees too.The spatial composition (Site/Building/Storey/Space/Zone) is one more kind of aggregation, different from the element (meronymical) composition one (e.g. a stair and the assembled elements in it).Moreover, nesting is a slightly different kind of element composition, representing the products which are specifically designed as complementary by manufacturers.Finally, subtraction relationships are also part of the IFC model, representing openings by means of the voiding mechanism.A great number of further relationships are added to this complexity (e.g. to associate materials, geometric representation or other property information and so on).
An additional complexity to the semantic model is given by the possibility to store the same kind of object by means of several entities.For example the layers within a compound wall object can be represented by means of an associated IfcMaterialLayerSet, but also as a more generic decomposition where every wall layer is modelled as a distinct IfcBuildingElementPart.
A great number of attributes can be associated with entities (and inherited by the parent-ones), both directly or through property sets.
All this semantic complexity is intended to represent faithfully the buildings as functional to the standard designed scope.However, the implementation and use of such theoretically precise model is difficult and can result in inaccuracies or under-use of it, besides hindering interoperability for leaving too high freedom in filling the information in and in choosing the kind of representation to be used.
Additional terms which can be used in IFC, are defined within the buildingSMART Data Dictionaries (bsDD) and are modelled according to the International Framework for Dictionaries (IFD) 13 .It is based on the standard ISO 12006-3.
The IFC current versions are: IFC2x3, which was released in 2005 (with the latest corrigendum in 2007) and the IFC4.1 from 2018.At the time of writing, the most implemented and used version is still (by far) IFC2x3.For this reason, both versions were considered in this study.
A third part of the standards is the Information Delivery Manual (IDM), which defines the workflow and the information exchange specifications and requirements in the processes involved in the construction life cycle.
In an IDM, a set of Model View Definitions (MVD) can be defined for identifying the portion of the IFC model which is needed for a particular information exchange in the IDM to be fulfilled.This can define a use-case oriented part of the wide IFC model, to be implemented in software.
Interesting to note is that this mechanism is in a way opposite of CityGML Application Domain Extensions (ADEs) (Biljecki et al., 2018).In an IFC MVD a subset of definitions is selected from the monolithic schema, instead of an addition.On the other hand, for property set definitions, the usage of MVD and ADE are analogous, as both nominate additional attributes.
IFC derives many aspects from ISO (2014), informally known as STEP.The majority of geometry definitions are derived from ISO 10303-42 and the typical exchange formats are based on STEP Physical File (SPF, ISO 10303-21) and an XML encoding (ISO 10303-28).
Parametric modelling is usually employed in BIM and IFC, which makes it possible to encode many kinds of geometries.This includes Boolean operations and complex sweeps, for example the sweep of an arbitrary profile along a curve while constraining the normal vector.Also explicit geometries are supported in the form of Boundary Representations and (added in IFC4) efficient support for triangulated meshes.The implementation of the former type of geometry is that supporting the full stack of geometric procedures in IFC is a major implementation effort and due to implementation choices can sometimes lead to different results in importing applications.The complexity can, therefore, have consequences on interoperability and the way different pieces of software read and re-export the same geometry.

IFC geometries
The particular issues described in this section guided the methodological choices of providing a specific set of IFC geometries among the benchmark datasets, in order to point out specific related behaviour (Figure 1 is an example of uncontrolled geometries transformations within some of the tested software).
IFC is a complex standard and the fidelity of both importers and exporters is still evolving.Implementations are based on varying paradigms and geometric modelling principles: Boundary Representation, Polyhedra or Triangle meshes.Some implementations use so called toolkits that offer complete support for STEP and EXPRESS (the modelling language in which the IFC schema is specified; with an embedded complex Turing-complete constraint language to specify WHERE rules and FUNCTIONS).Other implementations use custom in-house developments and, for that reason, typically have less support for the automatic validation of IFC models based on the WHERE rules.The IFC schema is evolving as well14 mostly aiming at the improvement of reliability, reuse of industry-standard best-practices and the reduction of implementation effort.
Given these considerations, the permutations of IFC definitions (some of which assessed by this set of analytical geometries) span a three-dimensional matrix along the axes (Figure 2) that specify: 1. whether the situation is known to be exported by authoring tools 2. whether the situation is handled successfully by importing applications and 3. whether that situation is valid according to the IFC schema.
As such we can summarize this state in a tuple of three elements.For example, for a situation that is exported by authoring tools, successfully imported, but not valid according to the schema, the tuple would be < Y, Y, N >.
From these states we want to highlight three that are meaningful: • < N, Y /N, Y /N > is a situation that is never exported and can hence be removed from the schema to reduce the implementation effort of a fully compliant application.
• < Y, Y, N > are situations that are successfully imported, but not valid according to the schema, schema constraints may be loosened to actually allow such situations.
• < Y, N, Y /N > is the biggest problem for practitioners, i.e. geometries that are exported, but not successfully imported (whether valid or not).
An additional concern is that while validity according to the schema is fairly constant and universal, import success depends on the application used, and with many different tools used by practitioners, the diverging results of importers may cause misconceptions and disputes.Also note that validity is not as universal as it might seem at first glance.The IFC schema defines a point equality tolerance (IfcGe-ometricRepresentationContext.Precision) that defines "the tolerance under which two given points are Figure 2: Three-dimensional matrix representing the IFC geometries conditions: can be exported / can be imported / is valid according to the IFC standard.
still assumed to be identical".How this is handled in geometry kernels is implementation dependent.Additionally there are WHERE rules that specify that an extrusion length needs to be positive (IfcEx-trudedAreaSolid.Depth is of type IfcPositiveLengthMeasure which has a rule WR1 : SELF > 0.) but at the same time EXPRESS does not specify restrictions on the floating point number type, so an infinitesimal value can be supplied that is greater than zero, but when mapped to the number type in the importing application cannot be distinguished from it.Or an extrusion depth can be supplied that is below the model tolerance and therefore may fail in applications that adhere to this tolerance.
A significant set of geometries with a variation of such three features was added as part of the tested IFC datasets, as described in Section 3.2.1.

IFC georeferencing
Proper georeferencing of an IFC file facilitates a link between the (local Cartesian) coordinates inside an IFC model with their corresponding real-world coordinates in a geodetic reference system, and thus to place the model of a single building or construction within its context and environment.There are several options to store georeferencing information in IFC, with varying level of detail.These options range from basic address information over the specification of the geolocation of a reference point to the definition of an offset between the project coordinate system and the global origin of a coordinate reference system and the corresponding rotation of the XY-Plane.Below some of the georeferencing options are described and classified into Levels of Georeferencing (LoGeoRefxx) as proposed by Clemen and G örne (2019).This classification scheme is not officially defined in the IFC standard but can help practitioners to quickly assess which georeferencing information is available in the IFC file.Common for the presented options are that coordinates for a reference point (which is usually the origin of the local Cartesian system where the model is designed) are stored and sometimes complemented with a direction of the axis in the local system.
LoGeoRef 20 The coordinates of the reference point are defined with respect to the world geodetic system WGS84.In particular the attributes RefLatitude, RefLongitude and RefElevation of the IfcSite entity are specified accordingly.
LoGeoRef 30 The coordinates of the reference point is the stored in IfcCartesianPoint referenced within IfcSite, often defining the projected coordinates of the model reference point.Any rotation regarding project (or probably global) north could be stored in IfcDirection as an attribute of IfcSite.This is an ad-hoc solution used by several tools, and there is no formal way to state the details of the projected coordinate system.
LoGeoRef 40 The more official IFC-way to store the reference point is to apply the georef in the project's representation context.Necessary attributes for location (WorldCoordinateSystem) and the direction (TrueNorth) are stored within IfcGeometricRepresentationContext.
The options above are provided by both the IFC version 2x3 and IFC version 4 standard, while the option below is only supported since IFC version 4.
LoGeoRef 50 In this option the coordinates of the reference point are stored in IfcMapConversion using the attributes Eastings, Northings and OrthogonalHeight for global elevation.The rotation for the XY-plane will be stored using the attributes XAxisAbscissa and XAxisOrdinate.The coordinate reference system (CRS) used is defined in IfcProjectedCRS where all common CRS are supported.
For clarification which CRS is used the attribute Name of IfcProjectedCRS should be filled with the proper EPSG code.
The georeferencing of BIM has not been a priority for designers and software developers.Therefore the georeferencing information is not regularly stored and used, and not always read and exported in a completely consistent way.Moreover, the available tools to georeference BIMs are not perfectly optimized to acquire and store the correct information in the foreseen attributes and entities in the IFC files.For this reason, the IFC georeferencing, which is fundamental for GeoBIM integration, was investigated within the benchmark, from two main points of view.
On the one hand, the ability of software to correctly interpret the georeferencing information provided within the datasets was checked, with the focus on the georeferencing level LoGeoRef20 and LoGeo-Ref40.Since LoGeoRef40 is more accurate, allowing the storage of Cartesian projected coordinates, we asked participants to check if this one could be read and interpreted by software correctly, within the Task 1 (see Section 3.1).
On the other hand, it is quite unexplored if and how the tools can edit the georeferencing information of IFC models.It is known that there are several options (see e.g.Clemen and G örne (2019)) but normally there is no or only little control of how georeferencing information is stored in the final model.For this reason, we tested this issue separately, within the benchmark Task 2 (see Section 3.1).

The GeoBIM benchmark general set-up -COMMON WITH TASK3 PAPER
The benchmark was intended as a way to combine the expertise of many people with different skills, coming from several fields and interests, in order to describe the present ability of current software tools to use (i.e.read, visualize, import, manage, analyse, export) CityGML and IFC models and to understand their performance while doing so, both in terms of information management functionalities, and possible information loss.Moreover, since the big dimension of such standardised datasets often generate difficulties in their computational management, the ability to handle large datasets was a further part of the tests.
In particular, the four topics investigated in the benchmark are: T ask1 What is the support for IFC within BIM (and other) software?15 T ask2 What options for geo-referencing BIM data are available?
T ask3 What is the support for CityGML within GIS (and other) tools?
T ask4 What options for conversion (software and procedural) (both IFC to CityGML and CityGML to IFC) are available?
For this purpose, a set of representative IFC and CityGML datasets were provided (Noardo et al., 2019b) and used by external, voluntary, participants in the software they would like to test in order to check the support in it for the considered open standard (Noardo et al., 2019a).
Full details about the tested software and a full list of participants can be found in the respective pages of the benchmark website 16 .The significant number of participants, balance in skills, fields of work, levels of confidence about the tested software (asked them to be declared) offered the possibility to limit the bias in the results.
The participants described the behaviour of the tested tools following detailed instructions and delivered the results in a common template with specific questions, provided as online forms.In the end, they delivered both their observations and the model as re-exported back to the original standardised format (CityGML or IFC).
In order to cover the widest part of the list of software potentially supporting the investigated standards, we completed the testing ourselves, by searching the online documentation of both the standards and the potential software.
In the final phase of the project, the team coordinating the study analysed the participants' observations, descriptions and delivered further documentation (screenshots, log files, related documents and web pages).From this review, an assessment of the performances and functionalities of the tested tools was derived.Moreover, the delivered models were validated and analysed using available tools, when possible, and/or through manual inspection (Section 3.3).This approach allowed the inquiry about the level of interoperability given by the standard and its software implementation, by comparing the results of the export with the imported model features.
It is important to notice that the test results are not intended to substitute the official documentation of each software.Moreover, there were no expertise nor skill requirements to participate in the benchmark tests.Therefore, some information could be wrong or inaccurate, due to little experience with the tested software or the managed topics.The declared level of expertise is intended to lower this possible bias.Moreover, the benchmark team and the authors tried to double check the answers (at least the most unexpected ones) as much as possible, but the answers reported in the data were generally not changed from the original ones.The eventual discrepancies between the best potential software performances and what was tested could anyway be showing a low level of user-friendliness of tools (and thus a degree of difficulty in achieving the correct result).

The provided IFC datasets
We have selected IFC datasets allowing the testing of the main issues regarding the support of IFC (Table 1): one dataset was the architectural IFC model of a small two-floor building, intended to test the support for IFC features in the most reliable way (Myran.ifc).The second one was a large and complex architectural IFC model of a big tower in Rotterdam (UpTown.ifc),intended to test the software's performance; and the last one was the architectural IFC model of a medium-size building in IFC4, which was chosen to test also the support for IFC4 files and eventual discrepancies with the management of the other datasets, which were provided in IFC version 2x3 (which our previous experience suggests is better supported).All the IFC files where originally modelled in Autodesk Revit and thereafter exported as IFC files.
Moreover, since performing tests of IFC geometries could be tricky in complex files combining many geometry types in an uncontrolled manner, we chose to monitor this aspect better by means of a set of geometries generated on purpose by means of IfcOpenShell to test the support of software for specific kinds of geometry modelling (more detail in Section 3.2.1).Those are also provided in both IFC2x3 and IFC4.

IFC geometry sets
The geometries used in the BIM models can have a huge numbers of variations and fully checking them and their consistency and correctness is still an unsolved task.
Moreover, IFC admits a number of geometry types that can be useful to modellers, but they are sometimes not supported and can be interpreted in different ways by software.On the other hand, IFC puts validity constraints on certain geometries, but some software implements workarounds aiming at reading those invalid geometries too, which are often undocumented, and there is therefore often little possibility to keep track of these solutions.
For these reasons, a specific set of geometries (Figure 3) was provided among the benchmark datasets in order to test the specific cases, described in Table 2.The structure of the results template guiding the test about such datasets is as follows, for every geometry in the set.First there is a general question whether an object is displayed in that slot after import.Then what follows are questions about the positioning relative to the Z = 0 plane (mostly to distinguish how the negative extrusion depth is handled), the rendering of curved surfaces and what shape is displayed (to identify some specific situations such as the non-normalized extrusion direction).

Answers analysis about the support for IFC-SIMILAR to the TASK3 PAPER one
The methodology for analysing the results about the support of software for IFC (Task 1) and CityGML (Task 3) are very similar, since they were also conceived to test similar issues concerning interoperability and the ability of software to keep files consistent with themselves after import-export phases.
The initial part of results analysis (Section 4.2) is qualitative, providing the description of software support and functionality based on the delivered answers.
The complete answers and documents delivered in the online templates 18were double checked for correctness and consistency with respect to the asked questions.However, due to the nature of the initiative, we trusted the delivered information about the software, double checking it with new tests only in cases of inconsistent answers in different tests about the same software, or eventually, unexpected answers.In these cases, we also considered the level of expertise of the participant to assess if further checks were actually needed.
The delivered answers in the templates were critically assessed, cross-checking them with the different tests about the same software and the attached screenshots.A score about each aspect considered for the assessment of general support and software functionalities is assigned, as: 1-full support; 0.5-partial support; 0-no support.Those are synthesized in a table (Table 4), from where it is also easier to deduce possible patterns across many issues for a single software package or across many software packages for a single issue.
The definition of software groups are getting increasingly fuzzy, since the functionalities of all of them are continuously being extended and now tend to overlap with each other.However, in the tables, and more generally, in the analysis, in order to help the detection of possible patterns, the tested software are classified considering the criteria that usually guide the choices made by users, based on their different needs for specific tasks: • GIS -people expect GIS to combine different kind of geodata and layers and make analysis on them, structured in a database, in a holistic system; • 'Extended' 3D viewers are likely software that were originally developed for visualising the 3D semantic models, including georeferencing, and query them.They were (sometimes later) extended with new functions for applying symbology or making simple analysis.
• Extract Transform and Load (ETL) software, and conversion software, are expected to apply some defined transformations or computations to data; • 3D modelling tools have good support for geometry editing, but is not originally intended to manage georeferenced data nor semantics; • Analysis software are intended specifically for few kinds of very specific analysis (e.g.energy analysis); • BIM software, are intended to design buildings or infrastructures according to the BIM method.
The investigated issues, reflected in the different sections of the provided templates, regarded mainly the support of the software for the two standards (how the software read and visualise the datasets) and the functionalities allowed by the software with standardised datasets (what is it possible to do with such data).In particular, the test about the support was intended to test: how is the georeferencing information in the files read and managed (Section 4.2.1);how are the semantics read, interpreted and kept after the import (Section 4.2.2); and how is the geometry after the import (Section 4.2.3).
The last questions were related to the kind of functionalities (Section 4.3) that are offered by the software for the management of IFC data: • What kind of visualization is enabled (3D, 2D, with textures, with specific themes); • What kind of editing is possible (attributes, geometry, georeferencing); • What kind of query (query the single object to read the attributes, selection by conditions on attributes, spatial query, computation of new attributes); • What analysis are allowed.This topic is more complex, since very different analysis can be possible.Therefore we summarized it by defining two analysis types: 'Type 1' is any kind of analysis regarding the model itself (like geometric or semantic validation), and 'Type 2' are the simulations and analysis about the performances of the represented object (e.g. a building) with respect to external factors, in the city or environment (e.g.shadow, noise, energy, etc.).
• Final issue: Is it possible to export back to IFC?
Moreover, the support for each of the delivered datasets were noted, given the specific features: IFC2x3 building model (Myran.ifc),IFC4 building model (Savigliano.ifc),very heavy model (UpTown.ifc).
This first parts provide a reference about the tools themselves for people intending to use standardised information.In addition, the most challenging tasks and most frequent issues for the management of standards were supposed to be pointed out.
A second, more quantitative, part of the analysis considers the delivered models exported back to IFC from the tested software (Sections 4.4 and 4.4.1).The exported models of UpTown.ifc were visually inspected but could not be completely analysed since the used tools (such as the NIST IFC analyser) crashed without producing any useful result, probably given the big size of these datasets.For this reason the results regarding UpTown are not described in the paper.The models can be though downloaded together with the other data 19 .
The numbers and types of features of such files were calculated and compared to the same features in the initial datasets that were provided for the test.The NIST IFC analyser20 , calculating the number of IFC entities of each type in the dataset, was used for this check.Moreover, manual inspection of the files within 3D viewers (BIM Vision, RDF IfcViewer) was used to check the apparent problems (e.g.missing elements) and further changes (e.g.grouping in storeys, transformation of building elements in IfcBuildingElementProxy, change to some kind of element to others and so on).The inspection of the text format was also helpful to check some elements (e.g. the more formal ones, documenting organization, applications, IFC version and so on).
This allowed us to assess the level of interoperability that the connected standards-tools can actually reach in the different cases: i.e. can the data be imported and re-exported without any change?
A further assessment (Section 4.4.3) was intended to evaluate the software and hardware connected performance.The times declared by the testers were compared for each datasets to see if their computational weight could affect their management within software.
Given the complexity of measuring software performance to the closest second, this was not requested of the users.Instead, they were asked to provide an approximate timing value for each test, according to a classification that was proposed following the way they could affect the perception or the work of a user, as explained in the following list: • It is almost immediate (good!) • Less than a minute (ok, I will wait) • 1-5 minutes (I can wait, if it is not urgent) • 5-20 minutes (in the meantime I do other things) • 20 minutes-1 hour (I cannot rely on it for frequent tasks) • more than 1 hour (I launch my process and go home, definitely ineffective for regular work) Other options included reporting if the software crashed or if the task was not possible with the software provided, and participants were also asked to provide information about the specification of the machine, as this may impact overall performance of the software.Due to their diverse levels of size and complexity, timing results are summarised for the individual datasets.
4 Results: support of software for Industry Foundation Classes

Tested software against support for Industry Foundation Classes
For the benchmark we tested 31 different software packages (Table 3), trying to cover all the possible solutions to manage IFC, although a huge number of software packages and other tools exist, considering that the BIM-related industry is quite developed and still growing.In the table, they are organised based on the kind of software and divided into: open source, proprietary and freeware (but not open source) software.Moreover, the levels of expertise of participants making the tests (from L1 the least to L4 the most expert) are also reported.
Some pieces of software were tested several times, with different levels of expertise: 2 tests with eveBIM (level 1, beginner and level 4, developer); 2 tests with FZK viewer (levels 1 and 2); 3 tests with Safe Software FME (both expert); 2 tests with FreeCAD (levels 1 and 2); 3 tests with ArchiCAD, levels 1 and 2; and many of tests about Revit, as expected, since very popular software (levels 1, 2 and 3).
In addition, more (unsuccessful) tools were considered, for example within the Tekla suite, Tekla Structure is supposed to be the only tool supporting IFC (and finding the trial versions of others was anyway tricky); Autodesk Fusion, which is however not supposed to support IFC; bimspot21 crashed when trying to import any of the datasets.
Other software, found on internet or initially proposed by some of the participants were also not able to support IFC (e.g.ACCA Solarius and Solarius PV, iTown, BimView, bimvie).There were also issues with finding the trial versions of other software, such as BuildingReconstruction, OpenDesignAlliance IFC SDK, only allowing some of the tools to be tried, or were not reading IFC correctly (e.g.other Bentley software, ACCA EdiLus).
However, the most popular tools were covered and tested thoroughly.
Table 3: Software tools tested against support for IFC within the benchmark Task 1.

Software support for IFC
In Table 4, the initial, more qualitative, analysis of the participants' reports22 is synthesized.The result is discussed below focusing on the themes georeferencing, semantics and geometry

Load of IFC data: Georeferencing information
It is possible to notice that in most of cases, georeferencing information, and elevation, are not read correctly and the models are moved to a local system having the origin in (0,0,0), using different units of measure (mainly metres and millimetres).In some cases it is possible to change this.For example in eveBIM, by default, it would load the model with a local coordinate system; it is however possible change the settings to have it use the model's coordinate system.In other cases, the CRS is misinterpreted, for example Bentley Microstation+Terrasolid moved all the models to 0,0,0 in the CRS EPSG:3152.In other software, some georeferencing information is reported in very high detail, including datum, ellipsoid and so on.However, this information is not correct.
With some other software, like Revit or ArchiCAD different tests give different answers, therefore it is possible that it depends on import settings.
To summarize, only 30% of the tools correctly use the georeferencing information with proper global coordinates (as provided by LoGeoRef30 or 40).
Very few changes are reported about the tests with the IFC4 dataset, which does not use the extended IFC4 georeferencing features anyway.
Orientation of the model is instead kept correctly in most of the cases (90%), although it was difficult to assess if the added True North information was read.

Load of IFC data: Focus on Semantics
From the results related to semantics, it is noticed that the functionalities are better than the ones regarding the georeferencing management, with general good support, with some exceptions regarding entities: • entities are partially recognized properly in Bentley Microstation+TerraSolid, where for the Myran model, only IfcWall and IfcWallStandardCase were translated; • for ACCA Edificius a partial interpretation of entities is reported: "For example opening, door, and curtain wall are detected by the software.IFCBeam and IFCCovering are loaded as an IFC Proxy Object.It also has vertical/horizontal envelopes which are actually not IFC classifications" ; • again, for Revit, one of the participants states: "some classes that are different in IFC are the same in Revit, following the standard settings; it is possible to set own Revit Categories (therefore set them to the correct IFC Class name)" ; • in ArchiCAD, they report "in some cases Some classes/layers are missing, most notably are the roofs and windows".
For many tools it was tricky to check the relationships, including hierarchies and, in other cases, some issues are reported as following: • in FME, hierarchy and relationships are not kept directly, but by means of parent ids; • for Revit, the loss of many relationships is reported, probably due to the need to separate elements while importing; • in BIM Vision some relationships are reported to be possibly missing; • in FreeCAD, the support is also reported to be partial: "spatial aggregation and containment is properly retained (although in debug mode, aggregations with more than 10 objects are skipped).Type objects (IfcRelDefinedByType) seem not to be considered.IfcRelConnectsPathElements (connection between walls) is ignored.IfcRelVoidsElement/IfcRelFillsElement could not be checked because due to import settings, openings where not imported as objects in their own right" ; • In Autodesk civil 3D, only storeys aggregation is kept; • in FZK Viewer, although the relationships are managed rather well, sometimes hierarchical relationships and aggregations are treated in the same way, which is however more an inaccuracy than a serious issue.
Also the results regarding attributes show some issues: • in the FZK Viewer sometimes they are different from the reference ones, although generally wellmanaged and interpreted; • Bentley Microstation+TerraSolid can read consistent attributes for the Uptown dataset but cannot for the Myran one; • in ACCA PriMus-IFC, it is only possible to find many "IfcRelConnectsPathElements" relations; • DDS-CAD is supposed to read attributes correctly, however, the function did not work, therefore it was not possible to check; • Tekla structures does not imports all the entities, although the present ones are consistent • in Revit, some of the attributes were consistent with the IFC ones, but most of them were missing; finally, in ArchiCAD, some objects are not recognized correctly in terms of attributes and semantics.
Approximately, 80% of the tools manage semantics in a satisfactory way, the 3D viewers group especially.However, many inaccuracies are still there to reach a good interoperability.In particular, it is noticeable how Revit reports inconsistencies with respect to the imported original data, although being the software where all the datasets were modelled and from where they were exported.

Load of IFC data: Focus on Geometry
The software tools in this study are considered to support the geometry if there are no apparent errors; that is no detailed geometric study was conducted.Specific reported features include: • FZK Viewer, generally works well, except with the Myran model, whose geometrical representation looks a bit distorted in one of the tests, while the geometry is not visualized at all in the other one; Moreover, with the Savigliano (IFC4) file, some elements (some roof pitches) are not visualised; • eveBIM detected wrong normals, although we do not know if they were wrong in the beginning or were changed during import, and with respect to IFC4 file, the IfcAdvancedBrep is not implemented so the geometry is not shown in this case; • about Bentley MicroStation +TerraSolid, some slight changes in dimensions are reported; • BIMServer seems to read geometry correctly, except that the roof at the top of the tower is missing, and possibly windows.
• again, about Revit some issues are reported, like some walls intersecting with the floors and beams that do not join correctly, subtracting the volumes; mistakenly display of subtraction solids inside the families of doors and windows, which are used to pierce the walls; often the stratigraphy of the walls is not correct in the corners and intersections; • in Vectorwork a scale can be applied to the data, therefore it is necessary to check that it is correctly set.
In addition, Table 4 reports whether the software is able to load and work with the different datasets.
Issues include: failures most likely connected to the computational requirements in the case of Uptown.ifc; to the IFC version in case of Savigliano.ifc; and maybe to possible geometry issues in the case of Myran.ifc, which is not read correctly in a few cases.
For example, the version 0.18 of FreeCAD is not able to import the Myran model, and one of the tests with FZK viewer fails with Myran as well.In addition, import errors are reported in Table 5: Errors in the imports for some of the software tools.

IFC geometries interpretation
The dataset is generated for both the IFC2X3 schema as well as the IFC4 schema, with the IFC4 geometries being a subset of the ones supplied for IFC2X3.For that reason one would expect similar import success ratios for the IFC2X3 and IFC4.This is not the case, as shown in Figures 4 and 5.A possible explanation is that applications supporting IFC4 are newer or better and therefore have higher success rates.
As mentioned in the discussion in earlier paragraphs, one of the main concerns of the authors regarding the state of IFC support in importing tools is the divergence in how situations are handled.On the one ,WHPYLVLELOLW\UDWLRDPRQJUHVSRQGHQWVIRU,)&; ,WHPYLVLELOLW\UDWLRDPRQJUHVSRQGHQWVIRU,)& Figure 5: Item visibility for IFC4.The labels on the horizontal axes refers to the column and row of the geometry in Figure 3 & ' ' ' ' (" Figure 6: Consistency among respondents, defined as the pair-wise equality of all combinations of answers (1.0 in case all respondents answer the same, 0.0 when all different, although that's generally due to the nature of the questions).Consistency is averaged over the three follow-up questions (regarding position, surface shading and shape type) and excludes answers of the respondents that indicated the shape is not displayed.
hand this can be explained as evolving support for more complicated geometry types or types that are not considered native to the domain of the importing tool.On the other hand, what can be observed in practice is that some tools import invalid situations without notifying the user.In some cases these are due to tools not performing checks on the entity instance attribute values or on the resulting geometries.For example, in the case of the negative extrusion depth, when the IFC parser does not validate WHERE rules automatically, detecting the negative extrusion depth requires an explicit statement in the program code that may have been omitted accidentally.In other cases there is a deliberate effort to "heal" certain invalid situations.For example the IfcOpenShell software library provides functionality to detect self-intersections in face boundaries definitions and discard all but the largest cycle of edges 23 .While on the one hand this is implemented with good intentions, this behaviour does deviate from strict standard compliance and results in different flavours or dialects of IFC being supported among applications, which in the long run will hurt interoperability.The divergence of importing applications is displayed in Figure 6.
The least consistency among the IFC2X3 results is seen in D4, C1, C5 (various profiles with invalid extrusion direction).Naively one would expect consistent results here given that this is a violation of a schema constraint, but this shows the state of implementation in the industry that support for these constraints in parser libraries cannot be assumed, especially not the complicated ones as in this case (the IfcDotProduct function).D3, D5, D2 are also inconsistent which stems from the question on whether fillet radii are incorporated and whether the surfaces are shaded smooth or faceted.Interesting to highlight is that there is quite some consistency in F4 and F5 as virtually no application took the StartParam and EndParam of IfcSweptDiskSolid into account.

Using IFC data
BIM tools are usually very complex and making a very detailed report about them is difficult.However, the general functionalities regarding visualization, editing, query, analysis and export possibilities are tested (Table 4).All the software offer visualisation functions, mainly only 3D, usually also 2D within the BIM and 3D modelling software groups.Those same two groups are the ones mainly allowing editing of geometry or attributes.Usually, query functionalities are partial, within most of the tools: • they are foreseen within GIS software; • 3D viewers usually have partial query abilities, for example BIM Vision provides general pre-defined ways to view the information, listed by structure, by types, by groups (system, zones or other) and by layers.For each, properties, location, classification and relations can be shown at the object level; also FZK Viewer offers mainly pre-defined queries; • Lexocad can only query based on object type (e.g.wall, slab, wall standard, . . .); • in Autodesk civil 3D, as well as in Allplan and AutoCAD Architecture it is only possible to select properties which are in the native software format; • in Vectorworks, query tools are present, however query is possible if the project is connected to a database; object selection and information retrieval is possible; finally, ArchiCAD presents a "find-and-select" tool, but issues are reported.
Looking at analysis, very few software packages allow it (only 20% for the analysis concerning the model itself, Type 1, and 30% analysing the designed building performances, Type 2), which is very little for something which is supposed to be an operational tool.Since many different kinds of analysis can be within the two categories (Type 1 and 2), and the tested tools offered very various features, only the general support for the two types are reported in Table 4.In some cases, the support is partial, for example, some Type 1 analysis are sometimes foreseen but not working, as it is the case of Solibri Anywhere, or only in specific cases, like Vectorworks, working only with NURBS.With the Type 2 analysis the pattern is similar, with some tools giving issues, for example, in Allplan, Revit and ArchiCAD, the Type 2 analysis sometimes do not work with the datasets, or with IFC files in general, although available.It is quite difficult to give an exhaustive overview of this, though.
80% of the software can export to IFC and only 30% of them allow some customization about the IFC version and Model View Definition to be used.
As additional notes and warnings to the export operations, the participants reported: • for FME, there were a number of warnings related to inappropriate geometry types; • for FreeCAD, the default settings yielded an error regarding colors (to be checked in later versions and reported for fixing if necessary).By changing the settings to use the IFCOpenShell serializer, the error did not appear; • for ACCA PriMus, it is possible to add additional properties; • in Revit it is possible to customize the export by means of manual entity mapping; • similarly in Vectorworks, a layer mapping pre-process was needed to export the file.Each layer had to be selected separately and assigned to the correct "vector story name".
No sharp connection between the IFC certification and the software support and functionalities is verifiable.

Writing of IFC2x3 files: analysis of the exported Myran.ifc models
The models which were exported from the tested pieces of software24 were analysed by means of the NIST IFC analyser, in order to formally check them.The tool counts the number of entities, relationships and properties and summarizes it in reports.From comparing the summaries, counting the features in the exported files, with the ones regarding the original datasets, we can have an idea about the happened changes and we can point out which are the entities that tools struggle more with.
Some of the delivered Myran models were re-exported by the tools without errors, but apparently the export failed, since the building is not represented anymore: this is the case of one of the ArchiCAD tests, where only the signboard of the building is stored; the test with Tekla structure, where only one beam is there; and one of the models exported by Revit, which is empty.In addition, the models exported by FZKViewer could not be read by BIM Vision nor NIST IFC analyser to be analysed and inspected, and RDF IfcViewer only shows the IfcSite geometry, as terrain, and other two random (wrong) objects in one of the FZKViewer -exported cases, and a completely wrong building in the other one (Figure 7).Also, the model exported by FreeCAD gives some problems when trying to load it into viewers: RDF IfcViewer can show it only very far and small, and it can be hardly handled in such software; BIM Vision is not able to visualise it; Revit gives a warning ("IFC:only 2 points in polyloop #702627, expected ¿=3"), but then shows it quite correctly, although missing the colours, usually indicating the entities (Figure 8).There, it is possible to see that some elements are missing (e.g. the beams in the exterior stairs).
In Allplan (Figure 9), the difference that could be noticed with respect to the original is the lack of the external stair and, in the grouping, the IfcSite is stored separately than the three other storeys, which is actually a better storage, but anyway different from the original benchmark-provided file.In AutoCAD Architecture, the three storeys are collapsed in a single one, big parts of objects belonging to the side building are missing (e.g.walls, doors), the roofs' colour changed for some reason, probably due to the loss of some properties, and some of the walls that were cut at the roof level, are not, so that they protrude above the roof (Figure 10).In the model exported by Archline XP (Figure 12), the windows and doors seem to be disappeared, even if they rest below the most exterior layers; probably the reason was some issues in the use of IfcOpenings to subtract the volumes corresponding to doors and windows in all the concerned walls and coverings.The roof slab is also missing.
The two Revit ones, the two ArchiCAD ones, the ACCA PriMus one, the ACCA usBIM.viewerone (only showing something weird, similar to a duplication in the wireframe representing the site), in eveBIM, Simplebim, FME and Bricscad look similar to the original dataset, and one could assume they are consistent.
However, a closer look into the kind and number of entities included in the models, by means of the NIST IFC analyser analysis (Table 6), shows they are actually not.
The only two tools leaving the model completely unchanged (same number of each feature) are: ACCA Primus, which is a software supporting calculation of materials for costs assessment, and ACCA us-BIM.viewer.And the only entity which is unchanged across all the models is IfcUnitAssignment.Together with IfcBuilding, which is 1 for all the models.
The section regarding general information (blue-green), since it is quite easy for the software to keep track of such entities, interpreting and writing them back in a correct way: the number of IfcBuildings remain the same (one).Moreover, for example, eveBIM and Simplebim add one IfcApplication to the file, probably intended to document that the file passed also through such tools.Therefore it is not a mistake in the writing of the file, but a wise update of the file metadata.
The same applies for IfcOwnerHistory, IfcOrganization, IfcPerson, IfcPersonAndOrganization.However, the syntax used to add the information in these two files is not the same.Although both are not wrong, this flexibility in the writing of the new file would for sure be a difficult condition for the interpretation of the file by third applications.
In the section representing entities concerning quantities (yellow), which are not present in the original file, almost all the models remain the same, except for two of the ArchiCAD tests (which report different numbers, though), and the model exported by Tekla structure.Also the units representation (orange) is kept quite consistent, and the inconsistencies could be due to the different software settings for export (both for the original file and for the tested ones), not necessarily meaning inconsistency in the model.
IfcDerviedUnit and IfcDerivedUnitElement entities are the least kept ones by the re-exported models, which lose them completely in 7 out of 16 cases.
The most interesting part of this analysis is however concerning the IFC entities contained in the 'Shared Building Elements' and 'Product extension' parts of the IFC structure (white).
Models exported by the same software by different participants (e.g. the two coming from Revit, the two coming from ArchiCAD) present very similar results in the number of entities, although not exactly the same even.The same happens with AutoCAD Architecture and AutoCAD civil, which we can suppose using similar algorithms.Moreover, it is not possible to find any balance in the lost/gained entities considering neither the groups of entities being subclasses and superclasses, or that can be considered as alternative representation of the same object (e.g.IfcWall, IfcWallStandardCase, IfcWallType or IfcStair, IfcStairFlight, IfcStairFlight-Type), nor the specific entities and the IfcBuildingElementProxy and IfcBuildingElementProxyType.
It is also curious how some additional elements appear even if the similar ones are kept consistently.For example, in the case of 'stairs' entities, in both ArchiCAD tests and in Bricscad, although the IfcStairFlight entities are still in the same number, other 4 IfcStairFlightType objects (same number of occurrences of IfcStairFlights) are added (only three in one of such cases, for some reason).
In the models exported by Revit, 19 more IfcStair entities are added.
Similar reasoning is possible when considering IfcWall, IfcWallStandardCase and IfcWallType, although Table 6: Differences in the NIST analysis results between the Myran models exported by the tools and the provided one.
in this case in (only) one of the ArchiCAD models and in the Allplan case the balance is kept, at least, with IfcWallStandardCases becoming IfcWalls in the ArchiCAD case and IfcWallType becoming IfcWalls in Allplan.Such change in the choice of an entity whose meaning is similar, although losing the optimization advantages for which different entities were proposed in the data model, would keep the model semantically consistent.However, it is not possible to verify a consistent behavior with respect to such choice in software.
The models exported by FreeCAD lose parts of object (as we could see from the visual inspection, for example part of the external stairs are no more represented), as well as Allplan (e.g.again the external stair).Moreover, Revit-exported files, FreeCAD, Allplan, AutoCAD Architecture, AutoCAD Civil lose all the IfcMemberType entities.
But looking at the final balance in the number of entities, we can see that their number raises.Therefore, assuming that no objects or parts of the model were not exported at all, we can just report on a major number of entities describing the building elements, that means that probably some of the objects were split into several ones.
The increase in the number of entities is also confirmed by the number of geometric objects, whose numbers vary substantially, losing many instances of several geometric entities, while some of them are more than doubled (especially the most simple ones, such as IfcCartesianPoint, IfcFace or IfcPolyLoop).
Again, a pattern is difficult to be found.Similarly, for the parts concerning the other groups of entities, it is difficult to find similarities across the software or the same entities.
The increase in the number of entities is also reflected in the file sizes, that change from the original 27 MB to a maximum 94 MB for both the Revit models.
In addition, the dimensions decrease to 24 MB in the models exported by eveBIM, SimpleBIM, Bricscad.This could mean that either those ones are better optimized than the original dataset or that something went lost in the conversions.In both cases it does not depict the best situation for interoperability, even if it maybe not too bad per se.

Writing of IFC4 files: analysis of the exported Savigliano.ifc models
Some similar considerations can be done for the exported Savigliano models.
The models exported by BIMserver, eveBIM and SimpleBIM look good.
Again, the models exported by FZKViewers can't be opened in BIM Vision, nor it is shown in the RDF IfcViewer or analysed by the NIST IFC analyzer.
In the model exported by Allplan (Figure 13), the upper part representing the roofs is missing, and all the windows as well.Moreover some of the external walls and some slabs are coloured in a different way (blue), but it is not very clear what is changed.The model exported by AutoCAD Architecture (Figure 14) looks very similar to this last one: the top part of the building is almost completely missing, even a bigger part than in the Allplan model.Moreover, the windows are missing and, in this case, the openings in the walls, where the windows and doors would be supposed to be inserted, are missing too, presenting plain filled walls.Even the ramp leading to the underground parking is filled and appears as the continuation of the site.Moreover, other elements are missing, such as the windows in the stair towers, closing elements.In addition, the colour of some elements, especially slabs are missing, so that probably some property was lost.
The AutoCAD civil one is very similar to the AutoCAD Architecture one, but in addition, a slab and and a door are completely displaced and the site colour changes (Figure 15).The ramps to access the underground parking is however well represented and free.One possible explanation is that, being the software intended to represent civil works, those ramps can be better interpreted with respect to the same in AutoCAD Architecture, although probably being based on similar algorithms.The model exported by ArchiCAD also misses the top part and windows.Moreover, the site around the building is missing (Figure 16).An apparent error is a sort of beam (classified however as wall) that goes from one of the ground slabs into the ramp going to the parking.When looking at the NIST analysis (Table 7), it is again possible to notice several more inconsistencies, except for the model exported by BIMServer, which leaves the model exactly unchanged.
Simplebim comes second, considering that all the others perform very bad.In this case there are no differences in the number of building elements, and very little difference among the geometric entities, where many IfcDirections, 25% of IfcCartesianPoints and some other ones are lost.In this case the main differences are found in properties, materials and relationships.
As in the case of Myran, it is very difficult to find patterns in the loss/gain of entities.In the model exported by Bricscad it is possible to see 718 IfcWalls lost and 718 IfcWallStandardCases gained, which were probably resulting from the conversion of these ones.The numbers do not correspond perfectly, but similar behaviour can be seen also in ArchiCAD and Allplan, while AutoCAD Architecture loses 35 IfcWalls and gain 23 IfcWallTypes.The most general entity being IfcWall, among them, this is a weird result though, since the opposite direction of the conversion would be expected to be the safest one for the sake of consistency.
Very little other patterns can be found.In this cases many entities are lost, contrary to the Myran case, maybe due to the little support for some IFC4 new entities.We could also see from the visual inspection that many objects were lost and this is even more clear by the entities count, where we can see that many are lost completely.The tendencies are generally not very clear, even if it is possible to notice that the added entities are mainly IfcPropertySingleValue, IfcFace, IfcPolyLoop, but the behaviour varies case to case, and it is necessary to look at the Table 7 to check the details.

Writing of IFC files: analysis of the exported IFCgeometries.ifc and IFC4geometries.ifc models
The differences in the NIST analysis results between the IFCgeometries.ifcmodels exported by the tools and the provided one is shown in Table 8 and those for IFC4geometries.ifc in Table 9.
Table 7: Differences in the NIST analysis results between the Savigliano.ifcmodels exported by the tools and the provided one.
Table 8: Differences in the NIST analysis results between the IFCgeometries.ifcmodels exported by the tools and the provided one.
Table 9: Differences in the NIST analysis results between the IFC4geometries.ifcmodels exported by the tools and the provided one.
Looking at the two tables, there is no relevant difference between the test with the dataset in IFC v.2x3 and the one in IFC v.4.
In the case of these datasets, the original files consisted only of geometries, without any other specific semantic information.Therefore, when it is present in the exported files (e.g.Materials, Styles and property sets), it means that the software which exported them had attached such information based on its default settings.
Few tools are able to re-export the data leaving them completely consistent with the imported ones, namely: ACCA Primus, ACCA usBIM.viewer,ArchiCAD.eveBIM can also be included in the list of the good ones, since some elements are added regarding IfcPropertySets and metadata, but no elements are lost.Simplebim follows, with only few entities missing.
Many geometries resulting from parametric modelling are completely lost, as it is possible to see in correspondence of the lines in light green in Table 8 and Table 9 (e.g.IfcRevolvedAreaSolid, IfcCirclePro-fileDef, IfcIShapeProfileDef, IfcSweptDiskSolid) as well as some of the entities which are in the other green rows in the tables, such as IfcBooleanResults.Instead, it is possible to observe that the number of other more generic entities increase a lot, e.g.: IfcCartesianPoint, IfcFace, IfcFaceOuterBound, IfcPoly-Loop, and so on.Apparently, the parametric geometries tend to be simplified in the export, by most of the software, in favour of a more explicit representation.It is also visible in the analysis of the other exported models, but in these datasets it appears very clearly.

Software performances with IFC
A total of 43 different reports were returned, for 33 different software packages.In particular, multiple results were submitted for FreeCAD (2 sets), FZKViewer (2 sets), Autodesk Revit (6 sets, multiple versions), Vectorworks Designer (2 sets), Archicad (3 sets).These offer the opportunity for timing comparisons to investigate the impact of hardware on software performance.
Myran dataset Table 10 gives a summary of the success rates returned for the tests on the Myran data (in the rows, the number of tools for each option).Table 11 gives the count of the different timing values for the successful tests.Note that in some cases users reported results for some of the tests but did not report results for all of the tests ("No result reported").Additionally, some users typed in comments such as "no error" instead of giving specific timing.These are included in the "No result reported" count.For the Myran dataset, none of the tasks took more than 5 minutes to execute, and the large majority of the software packages tested could zoom and pan the data immediately.Data loading times of less than a minute were also dominant (65% in total).28 out of the 43 participants reported successful IFC export, with 86% of these exports being executed in less than a minute.

Uptown dataset
Table 12 gives a summary of the success rates returned for the tests on the Uptown data.Table 13 gives the count of the different timing values for the successful tests.As with the Myran data, in some cases users reported results for some of the tests but did not report results for all of the tests ("No result reported").Additionally, some users typed in comments such as no error instead of giving specific timing.These are included in the "No result reported" count.These results demonstrate clearly the impact of the larger dataset on the tests carried out, with only 47% of the software packages able to carry out the basic timed test that involved opening the dataset in the software, and 41% reporting that the software crashed.As above, in some cases users reported results for some of the tests but did not report results for all of the tests ("No result reported").Additionally, some users typed in comments such as "no error" instead of giving specific timing.These are included in the "No result reported" count.These results demonstrate clearly the impact of the more complex dataset on the tests carried out, with only 58% of the software packages able to carry out the basic timed test that involved opening the dataset in the software, and 35% reporting that the software crashed.For the Savigliano dataset, in contrast to the Myran dataset, 9 of the successful attempts to open the data (36%) took more 5 or more to execute, although once open zooming and panning the data was immediate, and the large majority of the software packages tested could zoom and pan the data immediately.Data loading times of less than a minute were also dominant (65% in total).16 out of the 43 participants reported successful IFC export, with all of these taking less than 5 minutes to execute.
Multiple Tests on Same Software Packages As noted above, the crowdsourcing approach taken in this project resulted in multiple participants testing the same software, providing the opportunity for comparison.For both FreeCAD and FZK viewer, one out of the two respondents reported that they were unable to open the Myran dataset.Vectorworks results are similarly consistent, with the only minor difference being that one participant reported that the Myran data opened immediately and another that it took less than one minute (both on machines with 16GB RAM).
Identical timing results are reported for the zoom/pan/rotate/query tasks for the remaining three packages (Autodesk, ArchiCAD), and as noted above, the large majority of software packages managed these functions in near immediate time.As different analytical tasks were selected by participants these are not comparable.
Differences do, however, arise when execution time for the remaining functionality is considered.Table 16 summarises the remaining timing results for these packages for the Myran dataset, which was chosen as this proved the least problematic of the three datasets in terms of successful execution of tasks.
For Revit, there is an apparent performance improvement when opening Myran data in the 2020 version of the software, and an improvement of export  the linked object query make these difficult to compare.For ArchiCAD, all users report an IFC export time for Myran of less than a minute, but two report that it was not possible to query linked information.
Examining the hardware used, Result 3 for ArchiCAD -file opening time of 1-5 minutes -was obtained on a machine with 8GB RAM, whereas the others were tested on machines with 16GB RAM (all Windows 10 machines, all with dedicated graphics cards).

Discussion
What is apparent while analysing the results of the export, is that it is very difficult to find a pattern or a sensible explanation about what happened in the export phases, since there are very few cases in which the IFC entities are completely kept / lost / either kept or lost in all the exported models.The most frequent scheme is: some models keep all -some models lose all -some models keep part.The very high complexity of both the semantic data model and the geometry management is for sure related to the issue.
Similarly, the investigation about software functionalities suggests a general difficulty in reading semantics correctly, very little support for georeferencing and quite good support for general geometry, as visualized, which is not unambiguously read though, when looking at the IFC geometries datasets results.But no clear pattern is found, that could make it possible to better understand the specific issues of IFC interoperability.A clear result is just that very little interoperability is actually reached, since there are very few tools able to read the standardised datasets correctly and even fewer that are able to export them consistently.The ability to uniquely interpret the models and to leave them consistent through the import-export phases is absolutely essential for interoperability and what it enables (data exchange, data re-use and so on).At this stage it is not possible to fully trust standardised models though, even just for simple file exchange.
Results were reported for 33 software packages -highlighting the wide range of tools available to users.
Different versions of Revit also demonstrated that Autodesk is making improvements to the interoperability of the software -Revit 2019 gave results for all data, but Revit 2018 could not handle two of the three datasets (Uptown and Savigliano).The Myran data could be opened by 40 participants, the Savigliano data by 25 and the Uptown data by 20.Twelve participants who reported that they were not able to open the Uptown data also reported the same issue with Savigliano.Ten participants reported problems with Uptown but no problems with Savigliano and 4 with Savigliano but not with Uptown, indicating that the size of the dataset is-overall-more problematic than the complexity.
Regarding performance, the impact of the size of the dataset on overall performance is marked with 55% of the software packages that managed to open the Uptown dataset reporting a time of 5 minutes or more.While it is not possible to say which software package is fastest -the approach to timing used general timing categories rather than requiring the user to undertake the onerous task of time measurement and performance will also depend both on the hardware being used, we can report that none of the software packages managed to carry out all the visualisation tasks in under a minute for the Uptown dataset, although 9 packages achieved this with the Myran dataset and 9 with the Savigliano dataset.For IFC export, 24 software packages managed to export the data in 1 minute or less for Myran, 14 for Savigliano but only two for Uptown.
It is difficult to comment on the overall results for the timing of the analytical tasks, as this would vary very much depending on what functionality the user tested/were available in the software, and participants may not have tested all available analytical functionality in their chosen software package.Further, more detailed analysis is required to better understand the range of options available to users in this category.
The set of available functionalities of software to work with IFC models is moreover very limited.The analysis about the model itself (Type 1) are essential to make sure a model is valid and suitable to support further analysis.On the other hand, tools are necessary to use IFC models for the operational intentions it was conceived for.However, not many tools are now able to work with IFC effectively.
The fact that the inquiry is based on voluntary and completely open contribution can be considered both a strong point and a limitation of this work, since it is essential to cover the investigated object in the most thorough way: as many software packages as possible, with many experts involved, but also with the inclusion of less expert users to test also user-friendliness.However, the limit could be the incompleteness of the resulting tools review.This one was limited by the ex-post integration of tests about the software not considered.Moreover, the tests reporting suspicious results according to the promoting team experiences (for both too good or too bad performances), were double-checked with new tests or asking for clarifications.A further issue could be the inexperience of some testers, reporting about tools behaviour in an inaccurate way.To lower this eventuality, it was checked that all the delivered answers were described with sufficient care, whatever the level of expertise could be.Once verified this, eventual conflicting answers with respect to the tools actual potential, could indicate a deficiency in the suitability of the tool to be used by any inexpert user (which would be anyhow necessary for the models to be used in practice).The involvement of a large part of the community (twice the number of participants delivering the results initially registered to the initiative) is also important to perceive the relevance of the topic, although dealing with a somehow hidden issue among the high-level standardization and academic communities.
The insight and data gained with this study can be the base for reasoning about improvements to the IFC model, the common behaviour of the models and the most urgent issues to be tackled by software, for example, the suitable implementation of georeferencing.
The georeferencing issue is even more articulated.First, regardless of the country and its level of advancement in implementing the BIM idea in ongoing projects, the little use by designers and architects of BIM models with the right georeference is similar.Almost all designers expect a model in their local coordinate system, despite the fact that the point cloud obtained for this purpose has the correct coordinates, consistent with the national system.To make it worse, in most cases, "long" coordinates are truncated due to software problems with "long" coordinates.Therefore, in addition to developing effective methods for storing georeferencing in IFC files, it seems necessary to provide education to designers and architects and a clear guideline about how to deal with it.
Semantics would also have advantages from simpler and clearer rules about how to structure entities, priorities and limits.For implementations, it is for sure preferable to rely on a simpler model than a very complex and expressive but inconsistent one.

Conclusion
The described study in this paper, part of the GeoBIM benchmark project, was intended to point out and provide evidence about the support and issues of available software for standardised information in IFC.Interoperability is essential for a number of use cases, and even for simply exchanging and re-using data.Standards are supposed to be enabling such interoperability and standardization is the essential premise to the development of any integration, including the GeoBIM one.Industry Foundation Classes, by buildingSMART, is the official reference open standard for BIM management and storage.However, a number of issues are informally reported, preventing the effective use of IFC in datasets, without the issues being clear to users, who usually rely on proprietary spread formats (such as the Revit native format), using them as de-facto standards.Although being aware of some data or functionality loss, no systematic proof was available to be the base of future improvements in implementations, in data modelling and in the standard itself.
The results show how difficult it is for software tools to read, manage and export the IFC datasets consistently, with full disadvantage of interoperability, first, and of effective usability of datasets in use cases too.
Possible bias in the results could be given by the little eventual expertise of participants making the tests, or by the initial inaccuracies in the provided datasets, coming from practice.However, such datasets were validated and improved as much as possible for the purpose of the benchmark, in order to limit their influence on the quality and reliability of results.Therefore, if any of such chances happened, although the great efforts in controlling them, it would reflect additional drawbacks of the standard itself, for the little clarity about its use for the modelling of actual datasets and the difficulty in implementation, which could produce little intuitive tools.
This study shows the drawbacks of the current IFC standard and implementation, and the related difficulties, also due to the challenges to which it is intended to respond (e.g.representation of the information regarding a vast and complex field, flexibility to multiple needs).The outcomes are of great importance to outline them officially, and to be the base for future research in the field and development of concrete solutions, such as the addition of constraints and specific guidelines, more simple ways to store geometry, better selection of useful semantics, and so on.
Considering the results of this study as evidence, future work should be aiming at the solution of the outlined issues.
For example, the study of clear guidelines about the use of such complex semantics, fixing priorities and criteria for their use.Moreover, specific kinds of geometries should be selected for specific cases, adding constraints to guarantee that software can import, read, use and re-export them without any change.In addition, when considering the performances related to the computational requirements, we can easily understand that the reduction of data size is urgent.
buildingSMART promisingly began working towards solving such issues, trying to work for less complex models offering more straightforward choices, easier to implement (see the "Ten principles for the future IFC" by buildingSMART 25 ).Thanks to this study, it was possible to gain a higher awareness of the specific issues to be tackled in future research in order to foster an improved adoption of the IFC standard.

Figure 1 :
Figure 1: Example of unusual geometries behaviour in some of the tested software.

Figure 3 :
Figure 3: Reference schema of the modeled IFC geometries.

Figure 4 :
Figure4: Item visibility for IFC2X3.The labels on the horizontal axes refers to the column and row of the geometry in Figure3

Figure 16 :
Figure 16: Savigliano model exported by ArchiCAD, where it is possible to notice the beam invading the ramp

Table 1 :
Summary of the descriptions, features and aims of the provided IFC data for the GeoBIM benchmark 2019

Table 2 :
Description of the IFC geometries sets (in IFC 2x3 and 4) provided as part of the IFC datasets

Table 4 :
Synthesis table of the delivered tests regarding support for IFC (benchmark Task 1).The colour scale from green to red is assigned according to the scores from 1-full support to 0-no support.Some tools work partially, for example, FME uses RefLatitude RefLongitude in IfcSite, although more accurate coordinate are assigned through an IfcLocalPlacement (LoGeoRef30) or the project's representation context (LoGeoRef40).

Table 5 :
ifc: Error 776: Boolean Operation -clipping plane generation for Cdgis-Model geometry failed; Geometry face -Invalid outer loop; Geometry polyline -Polyline contains collinear points, points removed; Geometry polylines -Double point removed eveBIM with Savigliano.ifc: some geometries can't be generated.Error message is: "Warn -IFCProduct (#565376): error IfcRepresentationItem (#564617) Error -this representation Item IfcAdvancedBRep (#564922) is not implemented" (this will be a further development of the software) RFEM Many elements were not imported because they were said to be "not relevant for structural analysis".

Table 10 :
Success rates for the Myran dataset

Table 11 :
Timing results for the Myran dataset

Table 12 :
Success rates for the Uptown dataset

Table 13 :
Timing results for the Uptown datasetFor the Uptown dataset, in contrast to the Myran dataset, 9 of the 20 successful attempts to open the data (55%) took more 5 or more minutes to execute, with 2 of the participants reporting test times of between 20 minutes and 1 hour, and 3 of over one hour.This clearly shows the impact of a larger sized dataset.Interestingly, once open, the vast majority of the testers reported sub-minute execution times for zooming and panning.Only 15 out of the 43 participants reported successful IFC export, with more than 50% of them taking over 20 minutes.Savigliano dataset Table14gives a summary of the success rates returned for the tests on the Savigliano data.Table15gives the count of the different timing values for the successful tests.

Table 14 :
Success rates for the Savigliano dataset

Table 15 :
Timing results for the Savigliano dataset time from 1-5 minutes in the 2018 version, through to "almost immediate" in the 2019 and 2020 versions of the software.In terms of hardware, all tests were carried out on Windows 10 machines.However, the machine used for Revit 2020 had 64GB of RAM, with Result 1 obtained with 16GB RAM, and Result 2, 4 and 5 with 8GB RAM.Contradictory results in

Table 16 :
Inconsistent timing results that were submitted for the Myran dataset