Software selection¶

Several commercial Off-the-Shelf (COTS) components will be required by the software solution to be developed. A basic set of components can be identified and selected at this early stage, namely a Database Management System and a Geospatial Data Server. This is still a preliminary and incomplete selection, since the functional requirements have not been fully specified and approved.

The project involves the exchange of statistical data between European Union Member States and EU institutions, such as Eurostat. The SDMX standards and content guidelines will be adopted where applicable. The Open Source Software for SDMX developed by EuroStat will be used ‘by default’ where required and applicable. Eurostat’s SDMX tools are thus excluded from the selection process applied to COTS components, except with regard to the portability constraint (see Constraints).

In addition, a software portfolio is selected to support the project’s Project Management Infrastructure. The functional requirements for these components are generic and well known for open technology development (see [HOSG09] or [Foge09]) and an integrated solution can be proposed.

The following generic technical goals for open-technology development direct the selection process:

Goals

Flexibility.

A component that can be used in a variety of ways tends to have more potential users, some of whom may aid the project (e.g., via bug reports and development time).
Portability.

A component that can be used on more platforms tends to have more potential users.
Modularity.

A component that is modular (e.g., with clearly-defined sub-components and perhaps support for a “plug-in” architecture) is more flexible, as well as being easier to review for correctness.
Use of Open Standards.

Where possible, avoid depending on interfaces that are controlled by a single vendor.
Reuse and collaborate with existing OTD (Open Technology Development) projects.

A project should focus on building new software, not re-implementing OTD projects that already exist. […]
Avoid non-OTD dependencies.

Depend only on widely-used OTD platforms, libraries, and development tools. If a component depends on a non-OTD component, and that component then needs to be changed, it may be difficult to make the change or have that change incorporated. Similarly, it may be difficult to get support for unusual libraries and development tools. […] If a proprietary component must be depended on, isolate it through plug-ins or an interface defined by an open standard. […]

Where there are alternative approaches, a simple analysis of alternatives should be performed, and discussed among the project so that key issues or alternatives are not overlooked.

Source: [SWLH11]

Process¶

The selection process follows the common GCS process:

General COTS Selection Process

Define the evaluation criteria based on stakeholders’ requirements and constraints.
Search for COTS products.
Filter the search results based on a set of ‘must-have’ requirements [“apply constraints”]. This results in defining a short list of most promising COTS candidates which are to be evaluated in more detail.
Evaluate COTS candidates on the short list [“apply factors”]
Analyse the evaluation data […] and select the COTS product that has the best fitness with the criteria. […]

After Step 5, the selected COTS product is usually customised (a.k.a. tailored) as needed in order to reduce the mismatches it has with the requirements. A COTS product can be customised in different ways, such as using add-ons, adjusting parameters, etc.

Source: [MoRE07]

For each type of selected components, the rationale for the adopted criteria and a brief analysis of alternatives are presented in the respective sections.

The candidate set of components is trimmed and ranked using the following strategies:

Strategies to evaluate COTS products

Keystone identification, which starts by identifying a key requirement (e.g. vendor location or type of technology), and then searching for products that satisfy this keystone requirement. This allows quick elimination of a large number of products that do not satisfy the key requirement.
Progressive filtering, which starts with a large number of COTS, and then progressively defines discriminating criteria through successive iterations of product evaluation cycles, where in each cycle ‘less fit’ products are eliminated.

This strategy requires running steps 1 to 4 in the GCS process iteratively until a small number of most promising COTS products is identified from which one or more can be selected for integration into the system.
Puzzle assembly, which assumes that a COTS-based system requires fitting various components together like pieces of a puzzle. This implies that a product that ‘fits’ in isolation might not be acceptable when combined with other products. Therefore, this strategy suggests considering the requirements of each product while simultaneously remembering the requirements of other products in the puzzle.

Source: [MoRE07]

Constraints¶

Major constraints¶

The keystone identification strategy requires the definition of constraints. A constraint serves to limit the alternatives under consideration [East99]. Three major constraints are identified:

Licence type

Licence compatibility is a non-functional project requirement: the objective is to facilitate the sharing, reuse and future improvement of the solution to be developed, and, if required, of any existing components it may incorporate.

Following the principle of early determination of distribution policy [TrCo10], the software solution to be developed should be distributed under an OSI-approved Open Source licence, namely the European Union Public Licence v.1.1 [EUPLv1.1].

With regard to some of the COTS components, EUPLv1.1 compatibility is also a puzzle assembly constraint: EUPLv1.1 is a copyleft licence and all the Open Source Software for SDMX developed by EuroStat is distributed under the EUPLv1.1 licence.

Two alternatives are possible:

To evaluate only COTS software under OSI-approved Open Source licences that are EUPL-compatible licence (either directly or through a FOSS exception list established by the licensor).

To evaluate also COTS software under OSI-approved Open Source licences with which the EUPLv1.1 is compatible with (either directly or through the EUPLv1.1 exception list).

Most licences allow static linking with EUPLv1.1 (combining components through compilation, copying them into the target application and producing a merged object file that is a stand-alone executable), but GNU licences (LGPL, GPL) typically do not allow dynamic linking (using components at the time the application is loaded or executed) or incorporation of source code. Since a majority of open-source components is under copyleft GNU licences, the (also copyleft) EUPLv1.1 includes an exception list or downstream compatibility list, that allows mergers between EUPLed code and:

General Public License (GPL) v.2

Open Software License (OSL) v.2.1, v.3.0

Common Public License (CPL) v.1.0

Eclipse Public License (EPL) v.1.0

Cecill v.2.0

From COTS selection purposes, the above listed copyleft licences are considered EUPL-compatible (in really, it is the EUPLv1.1 licence that is compatible with them).

The final licence for distribution purposes must be confirmed in the end of the project, based of the type of components and their use in the system.

Portability

Cross-platform compatibility is a non-functional project requirement: again, the objective is to facilitate the adoption by different organizations with distinct IT infrastructures. The portability requirement is also a good ICT procurement practice that reduces the risk of vendor lock-in and facilitates the reuse of the components by different systems.

Only COTS components that are available for and supported on common proprietary and open source operating systems are taken under consideration. The platforms’ selection procedure is described in the Operating Systems section.

Acquisition Cost

Only software with zero cost of licence acquisition is included: the project’s budget contemplates infrastructure costs (i.e. for application hosting) and software development/customization but it does not allow for licence acquisition/maintenance costs.

As a result of the application of these constraints, the candidate sets are strictly based on FOSS (Free and Open Source Software) while guaranteeing compatibility with dominant proprietary and open source operating systems.

Maturity and Sustainability¶

In practice, the acquisition cost constraint does not influence the set of alternatives, because adequate components are either freely available, or have a ‘community edition’ which is free and an ‘enterprise edition’ with paid support services.

Indirectly, this situation derives from the application of a second set of constraints related to the maturity and sustainability of each of the FOSS components. Two evaluation models are combined:

the Software Sustainability Maturity Model (SSMM) [Gard13]
the Qualification and Selection of Open Source (QSOS) maturity criteria [Atos13]

Basically:

no COTS component with an SSMM level below 4 is considered acceptable;
and the following QSOS maturity criteria scores are set as constraints:

QSOS maturity criteria and minimum scores selected as constraints

Legacy : Project’s history and heritage

Age : { 0 : Less than three months }
Popularity : { 0 : Very few identified users }

Activity : Activity inside and around the project

Contributing community : { 0 : No real community nor activity (forum, mailing lists…) }
Activity on bugs : { 0 : Low reactivity in forums and mailing lists, or no mention about bugfixes in release notes }
Activity on features : { 0 : Few or no new features }
Activity on releases/versions : { 0 : Very low activity on the production or development versions (alpha, beta) }

Industrialisation : Industrialisation of the project

Services : { Existing service offerings (support, training, audit…) = 0 : No service offering identified }
Documentation : { 1 : Documentation exists but is partly obsolete or restricted to one language or to few details }
Source code modiﬁcation : { 0 : No convenient way to propose source code modiﬁcations }

Activity on releases/versions is evaluated based on each project’s website: only projects under active development are considered (at least one minor stable release in the past 12 months), only projects with stable releases are considered.

Activity on bugs and features is evaluated either directly (through the project’s version release notes, bug tracker or source forge statistics) or indirectly, through FOSS statistics gathering sites (mainly ohloh).

Contributing community activity is evaluated for the end-user and developer communities. End-user activity is evaluated directly (through the project’s forum, mailing lists, etc.) and indirectly, through community-based Q&A sites (mainly stackoverflow and sibling sites). Developer activity is also evaluated through source forge statistics (number of contributing developers, number of commits).

Documentation is evaluated directly at the project’s website. Only projects with publicly available documentation are considered. User documentation must include at least the hardware and software requirements, installation procedure and software configuration and operation. Developer documentation must include at least source code documentation. This constraint is also a basic requirement to allow the progressive filtering and functional evaluation of the candidate set.

Open Standards¶

The selection of COTS on the basis of its implementation of open standards is a basic technical interoperability requirement.

The Open Standards listed in the recent Portuguese National Regulation on Digital Interoperability [RNID12] are adopted where applicable, if more stringent requirements are not defined by European Union legislation (see [INSPIRE]) or required by the project’s functional needs.

In Portugal, compliance with the Open Standards listed in the Portuguese National Regulation on Digital Interoperability is also a legal requirement for the public administration systems development and ICT procurement.

In practice, the open standards constraint did not (significantly) influence the set of alternatives, because FOSS components typically implement the relevant open standards. However, the level of support for the applicable open standards is a factor in the ranking and selection of alternatives.

Todo

RNID open standards

Include list of RNID open standards and its application as selection criteria of each type of component.

Factors¶

A factor is a criteria that enhances or detracts from the suitability of a specific alternative for the activity under consideration [East99]. Functional requirements vary with the type of component: the rationale for the adopted criteria and a brief analysis of alternatives is presented in each respective section.

Functional criteria have precedence over non-functional criteria. However, there is an obvious correlation between the maturity and sustainability score of a given COTS component and its functionality, flexibility, modularity and integration capabilities.

Furthermore, non-functional criteria related to the maturity and sustainability of FOSS projects - based on the SSMM and the QSOS model - can be applied when a ‘best’ choice does not emerge from the application of functional criteria.

The following nine criteria are commonly used in FOSS evaluation [Berg05]: licence type, documentation, release activity, longevity, community, support, security, integration and functionality. A short description on the application of these criteria on the selection process is given below:

Licence

Licence type is one of the constraints used to limit the set of alternatives.

Licence compatibility with EUPLv1.1 is defined as allowing the distribution of the larger work under the EUPLv1.1 either through incorporation of source code, static linking or dynamic linking. Licence type can thus be used as a factor to rank alternatives, by preferring licences that allow code incorporation over those that only allow static linking, and the latter over the ones that only allow dynamic linking. Ceteris paribus, this factor can allow a decision between tied ranking alternatives.
Documentation

Documentation availability is a constraint. Documentation quality is a factor to decide between similar alternatives (e.g. the availability of on-line tutorials and other training material, or the existence of clear coding style guidelines for developers), as it clearly facilitates software use and adoption.
Release Activity

Release activity is a constraint when applied to exclude inactive projects or immature projects. Release activity is also factor to assess the maturity of similar alternatives: a planned release schedule and a public feature roadmap is valued (as is evidence that such compromises are kept).
Longevity

Project longevity is used as an indirect indicator of each project’s stability and chance of survival. This criterion is only (qualitatively) evaluated in conjunction with release activity and community activity.
Community

Activity inside and around the project is evaluated in terms of visibility (i.e. ease of access to project website, binaries and source code, documentation, discussion lists or fora, etc.) and activity of the developer community (using indicators available at ohloh, such as number of contributors, number of commits, etc.) and user community involvement, using indirect indicators such number of related papers in Google scholar (adoption by the academic community), relative number of searches recorded in Google trends (end-user interest) and number of Q&A recorded in StackOverflow sites.
Support
- Active user and developer lists.
- Active public bug tracker.
- Paid support options available.
- SaaS providers available
Security
- Evidence of Security Backporting.
- Existence of Enterprise (a.k.a. Long Term Support) Versions.
- Evidence of bug severity classification in public bug tracker.
Integration
- Modularity
- Open Standards
- Collaboration with other products (e.g. reuse of standard libraries).
- Clear identification of software requirements (dependencies), which must also comply with the evaluation criteria.
Functionality

The functionality sub-criteria depend on the component being evaluated. Functional requirements will be listed in each specific section.

Simply put, the three generic ‘no non-sense’ criteria used in the UK Open Source Software Options for Government list allow the identification of a manageable set of candidates (tipically with least than 5 COTS components) to which to analysis of alternatives is applied.

Informal maturity and deployability criteria

Scale: “If something is working, processing, or being used millions of times per day, perhaps used by airlines to process critical transactions across the globe, that is large-scale. That should give you some confidence that it will work at that scale.”
Criticality: “If we can find examples of software being used in critical functions, like healthcare — when a patient’s health depends on it — or if it’s used in real-world operations by crime agencies, or in high-security environments, then we can say that this software can be made suitable for critical systems.”
Longevity: “If a package has been working successfully for twenty, thirty years, then it can be added to the list, because then it’s not new and unknown. If it has been in operation for many years, we know the risks.”

Source: Tariq Rashid, IT Strategy & Reform, UK Government Cabinet Office

Total Cost of Ownership¶

A financial estimate of the total cost of ownership (TCO) is beyond the scope of the software evaluation and selection exercise.

Due to the project’s budgetary constraints, license acquisition cost is a binary constraint, not a factor; hardware and infrastruture requirements are similar, regardless of the specific COTS component; and operational costs, such as support and maintenance, are not readily quantifiable for an R&D project with a variable number and type of potential adopters: nevertheless, the selection process does try and minimise the solution’s TCO, through the application of the maturity and deployability criteria.

The selection of COTS, either proprietary or FOSS, is similar in that it must cover a project’s functional requirements. However, the total cost of a given solution is distributed differently.

The licence acquisition cost should be proportional to the provided functionality: thus, in non-free COTS software the least expensive version that provides the necessary and sufficient functionality is generally chosen. (The same principle applies to commercial services, such as technical support and maintenance services, in either FOSS or proprietary solutions).

Given that a budget ceiling on software licence acquisition always exists, the affordable solution may not be the most expansible one (e.g. to support development beyond a particular project’s time frame).

When selecting FOSS solutions, the need to reduce long-term costs, such as staff training or technology migration, justifies the choice of components that provide more flexibility even when they may look ‘over-sized’ in terms of the features or the complexity strictly required for a given project.

Additional References¶

European Interoperability Framework recommendations on Open Standards

Recommendation 22

When establishing European public services, public administrations should prefer open specifications, taking due account of the coverage of functional needs, maturity and market support.

[…]The level of openness of a formalised specification is an important element in determining the possibility of sharing and reusing software components implementing that specification. […] If the openness principle is applied in full:

All stakeholders have the same possibility of contributing to the development of the specification and public review is part of the decision-making process;
The specification is available for everybody to study;
Intellectual property rights related to the specification are licensed on FRAND terms [Fair, reasonable and non discriminatory] or on a royalty-free basis in a way that allows implementation in both proprietary and open source software.

Recommendation 21.

Public administrations should use a structured, transparent and objective approach to assessing and selecting formalised specifications. […]

Definitions

Formalised Specifications : Formalised specifications are either standards pursuant to EU Directive 98/34 or specifications established by ICT industry fora or consortia

Standard : As defined in European legislation (Article 1, paragraph 6, of Directive 98/34/EC), a standard is a technical specification approved by a recognised standardisation body for repeated or continuous application, with which compliance is not compulsory and which is one of the following:

international standard: a standard adopted by an international standardisation organisation and made available to the public,
European standard: a standard adopted by a European standardisation body and made available to the public,
national standard: a standard adopted by a national standardisation body and made available to the public.

Standards developing organisation : A chartered organisation tasked with producing standards and specifications, according to specific, strictly defined requirements, procedures and rules. Standards developing organisations include:

recognised standardisation bodies such as international standardisation committees such as the International Organisation for Standardisation (ISO), the three European Standard Organisations: the European Committee for Standardisation (CEN), the European Committee for Electrotechnical Standardisation (CENELEC) or the European Telecommunications Standards Institute (ETSI);
fora and consortia initiatives for standardisation such as the Organisation for the Advancement of Structured Information Standards (OASIS), the World Wide Web Consortium (W3C) or the Internet Engineering Task Force (IETF).

Source: [EIFv2.0]

QSOS maturity criteria and default scores

Legacy : Project’s history and heritage

Age : { 0 : Less than three months; 1 : Between three months and three years; 2 : More than three years }
History : { 0 : The software has many problems which can be prohibitive; 1 : No major crisis, or unknow history; 2 : Good past experience in crisis management }
Core team : { 0 : Very few identiﬁed core developers; 1 : Few active core developers; 2 : Important and identiﬁed core development team }
Popularity : { 0 : Very few identiﬁed users; 1 : Usage can be detected; 2 : Many known users and references }

Activity : Activity inside and around the project

Contributing community : { 0 : No real community nor activity (forum, mailing lists…); 1 : Community with signiﬁcant activity; 2 : Strong community with vivid activity in forums, with many contributors and supporters }
Activity on bugs : { 0 : Low reactivity in forums and mailing lists, or no mention about bugﬁxes in release notes; 1 : Existing activity but without any clearly deﬁned process or with long resolution times; 2 : Strong reactivity based on roles and task assignments }
Activity on features : { 0 : Few or no new features; 1 : Product’s evolution is led by a dedicated team or by users, but without a clearly stated process; 2 : Feature request process is industrialized, an associated roadmap is available }
Activity on releases/versions : { 0 : Very low activity on the production or development versions (alpha, beta); 1 : Activity on production or development versions (alpha, beta) with frequent minor corrective versions; 2 : Important activity with frequent corrective versions and planned major versions linked with the roadmap }

Governance : Project’s strategy

Copyright owners : { 0 : Rights are being held by a few individuals or commercial entities; 1 : Rights are uniformly held by many individuals; 2 : Rights are held by a legal entity or a foundation that the community trust (ex: FSF, Apache, ObjectWeb) }
Roadmap : { 0 : No roadmap is published; 1 : Roadmap without planning; 2 : Versioned roadmap with planning and delay measurements }
Project management : { 0 : No clear and apparent project management; 1 : Project managed by an individual or a single commercial entity; 2 : Strong independance of the core team, rights held by a recognized entity }
Distribution mode : { 0 : Dual distribution with a commercial version along with a functionally limited free one; 1 : Subparts are only available under proprietary license (core, plugins…); 2 : Completely open and free distribution }

Industrialization : Industrialization of the project

Services : Existing service offerings (support, training, audit…) { 0 : No service offering identiﬁed; 1 : Limited service offering (geographically, to a single language, to a single provider or without warranty); 2 : Rich ecosystem of services provided by multiple providers, with guaranteed results }
Documentation : { 0 : No user documentation; 1 : Documentation exists but is partly obsolete or restricted to one language or to few details; 2 : Documentation up to date, translated and possibly adapted to several target readers (enduser, sysadmin, manager…) }
Quality assurance process : { 0 : No QA process identiﬁed; 1 : Existing QA processes, but they are not formalized or equiped; 2 : QA process based on standard tools and methodologies }
Source code modiﬁcation : { 0 : No convenient way to propose source code modiﬁcations; 1 : Tools are provided to access and modify the code (eg SCM, forge… ) but are not really used by core team to develop the product; 2 : The contributing process is well deﬁned,exposed and respected, it is based on clearly deﬁned roles }

Source: [Atos13]

References

[Berg05]

van den Berg, Karin (2005). Finding Open options. An Open Source software evaluation model with a case study on Course Management Systems. Unpublished Master’s thesis, Tilburg University, The Netherlands.

[EIFv2.0]

EIF European Interoperability Framework for pan-European eGovernment Services, Version 2.0. URL: http://ec.europa.eu/isa/documents/isa_annex_ii_eif_en.pdf

[East99]

(1, 2) Eastman, J. R. (1999). Multi-criteria evaluation and GIS. Geographical information systems. 1 :493-502.

[HOSG09]

Hauge, O.; Osterlie, T.; Sorensen, C. F.; Gerea, M. (2009). An empirical study on selection of Open Source Software-Preliminary results. In Emerging Trends in Free/Libre/Open Source Software Research and Development, 2009. FLOSS‘09. ICSE Workshop on (pp. 42-47). IEEE. URL: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5071359&tag=1

[INSPIRE]

Directive 2007/2/EC of the European Parliament and of the Council of 14 March 2007 establishing an Infrastructure for Spatial Information in the European Community (INSPIRE). URL: http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CELEX:32007L0002:EN:NOT

[MoRE07]

(1, 2) Mohamed, A.; Ruhe, G.; Eberlein, A. (2007). COTS selection: past, present, and future. In Engineering of Computer-Based Systems, 2007. ECBS‘07. 14th Annual IEEE International Conference and Workshops on the (pp. 103-114). IEEE. URL: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4148924

[RNID12]

Regulamento Nacional de Interoperabilidade Digital URL: http://dre.pt/pdf1sdip/2012/11/21600/0646006465.pdf

[SWLH11]

Scott, J.; Wheeler, D.A.; Lucas, M.; Herz, J. C. (2011). Open Technology Development (OTD): Lessons Learned & Best Practices for Military Software. Sponsored by the Assistant Secretary of Defense (Networks & Information Integration) (NII) / DoD Chief Information Officer (CIO) and the Under Secretary of Defense for Acquisition, Technology, and Logistics (AT&L). Version 1.0. URL: http://dodcio.defense.gov/Portals/0/Documents/FOSS/OTD-lessons-learned-military-signed.pdf

[TrCo10]

Triaille & Coppens (2010) - JRC Open Source Software Guidelines. Joint Reasearch Center of the European Commission URL: https://joinup.ec.europa.eu/sites/default/files/OSS-guidelines-v-DIGIT-2.pdf

[Atos13]

(1, 2) Atos (2013) - Qualification and Selection of Open Source software (QSOS). Version 2.0. 2013-01-19. Appendix A: QSOS Maturity Criteria. URL: http://backend.qsos.org/download/qsos-2.0_en.pdf

[Gard13]

Gardler, Ross (2013) - Software Sustainability Maturity Model (SSMM). OSS Watch. Joint Information Systems Committee. URL: http://www.oss-watch.ac.uk/resources/ssmm

[Foge09]

Fogel, Karl (2009) - Producing Open Source Software: How to Run a Successful Free Software Project. URL: http://producingoss.com/

[EUPLv1.1]

European Union Public Licence URL: https://joinup.ec.europa.eu/software/page/eupl/licence-eupl