Quality Assurance by Extended Peer Review: Tools to Inform Debates, Dialogues & Deliberations

Schwerpunktthema - Method(olog)ische Fragen der Inter- und Transdisziplinarität – Wege zu einer praxisstützenden Interdisziplinaritätsforschung

Quality Assurance by Extended Peer Review: Tools to Inform Debates, Dialogues & Deliberations

by Ângela Guimarães Pereira & Silvio Funtowicz, European Commission - DG Joint Research Centre [1]

In this paper the challenges of trans-disciplinary practice and quality assurance by extended peer review, in terms of knowledge co-production, mediation and representation, are addressed. Spaces for the articulation of plural narratives are explored, including the opportunity to deploy new information technologies. The TIDDD (tools to inform debates, dialogues & deliberations) is introduced in the context of the GOUVERNe project.

1     The Trans-disciplinary Challenge

It is now recognised that fields of knowledge and scientific practice are, in many cases, no longer usefully divided into isolated compartments. This awareness leads to attempts to create bridges among several disciplines, and the emergence of inter-disciplinary and multi-disciplinary studies and projects. Already in the 1960's multi-disciplinary approaches were seen as an essential way to tackle practical problems, providing an impulse for the further development of systems thinking, integrated methodologies and operational research. The late 1970's academic curricula everywhere reflected the recognition that societal issues have to be approached, framed, resolved and justified from a multiplicity of perspectives, some even recognising a trans-disciplinary evolution (Nicolescu 1999). This is clear in addressing problems of sustainability and, in general, environmental governance.

There are several definitions of trans-disciplinarity but it is generally described as a specific form of inter-disciplinarity in which boundaries between and beyond disciplines are transcended and knowledge and perspectives from different scientific disciplines as well as non-scientific sources are integrated (Flinterman et al. 2001; Klein et al. 2001).

The above definition stresses the importance of integration of different scientific disciplines and non-scientific sources (and types) of knowledge. The latter is the characteristic feature that distinguishes trans-disciplinarity from other multi-disciplinary approaches. The concept originated from the increasing demand for relevance, legitimacy and applicability (which are aspects of quality) of academic research to the challenges of societal problematiques in a policy context.

Jasanoff (1996) has argued that scientific knowledge is not independent of political context but co-produced by scientists and the society within which they are embedded. The contextualisation of knowledge production, as policy relevant scientific knowledge requires a trans-disciplinary approach, both in terms of the integration of types of knowledge and of the mediation among sources of knowledge. Trans-disciplinarity is unavoidable in knowledge production, it being implicit in the context in which knowledge co-production occurs or explicit in the integration of different types and sources of knowledge.

Each discipline has developed in an established conceptual and methodological framework, with its own scales, language, narratives, knowledge representation, knowledge mediation and communication. Also scientisation has lead to the scientific internalisation of many societal issues, especially in environmental and health domains. This has occurred mainly through the creation of masses of quantitative knowledge and argumentations which, in many cases, do not help and can even worsen controversies (Sarewitz 2004), confusing framings and justifications when (urgent) action is required.

In this paper, we concentrate on the development of a tool (TIDDD or tools to inform debates, dialogues & deliberations) in order to operationalise the concept of quality assurance through an extended peer community, as a response to the need to extend the assessment of relevant knowledge to those who contribute to its co-production, outside the boundaries of disciplinary science.

2     Quality and Extended Peer Review

The assessment of the quality of the knowledge inputs to policy issues are in many ways different from those of research science, professional practice or industrial development (Funtowicz 2001). Each of those has its established means for quality-assurance of the products of the work, be they peer review, professional associations, or the market. However, for new controversial problems, the maintenance of quality depends on open dialogue between all those affected. This we call an “extended peer community“, consisting not merely of persons with some form or other of institutional accreditation, but rather of all those with a desire to participate in the resolution of the issue (Funtowicz, Ravetz 1990). Since this context of science is one involving policy, we might see this extension of peer communities as analogous to earlier extensions of the franchise in other fields, such as women's suffrage and trade union rights.

Extended peer communities are already being created, in increasing numbers, either when the authorities cannot see a way forward, or when they know that without a broad base of consensus, no policy can succeed. They are called “citizens juries“, “focus groups“, “consensus conferences“, or any one of a great variety of other names; and their forms and powers are correspondingly varied. But they all have one important element in common: they assess the quality of policy proposals, including a scientific element, on the basis of the science they master combined with their knowledge of the ways of the world. The contribution of relevant social actors in this case is not merely a matter of broader democratic participation and their verdicts all have some degree of moral force and hence political influence.

These extended peer communities will not necessarily be passive recipients of the materials provided by experts. They will also possess, or create, their own “extended facts“. These may include craft wisdom and community knowledge of places and their histories, as well as anecdotal evidence, neighbourhood surveys, investigative journalism and leaked documents. Such extended peer communities have achieved enormous new scope and power through the Internet. Activists scattered among large cities or rainforests can engage in mutual education and coordinated activity, providing themselves with the means of engagement with global vested interests on less unequal terms than previously.

Along with the regulatory, evaluative function of extended peer communities, another, even more intimately involved in the policy process, is springing up. Particularly at the local level, the discovery is being made, again and again, that people not only care about their own environment and health but can also become quite ingenious and creative in finding practical, mixed social and technological means for their improvement. In many cases, local people can imagine solutions and reformulate problems in ways that the accredited experts, with the best will in the world, do not find normal. This is most important in the phases of policy-formation, and also in the implementation and monitoring phases. Thus, in addition to extending the traditional processes of quality assessment, participants can enhance the quality of the problem solving processes themselves.

As stated earlier, trans-disciplinary practise arose as a response to the increasing complexity of scientific knowledge production, and the need to re-establish an active dialogue among a plurality of disciplines and forms of knowledge (Nicolescu 1999). This requirement now extends beyond the inter-operability of methods and techniques coming from different scientific disciplines; it is in fact a quest for quality, not (just) excellence in scientific terms, but robustness also in societal terms (Gibbons 1999). The aim of knowledge quality assurance by extended peer review is precisely to open processes and products of policy relevant science to those who can legitimately verify its relevance, fitness for purpose and applicability in societal contexts, contributing with “extended insights and knowledge“.

Trans-disciplinary practice and extended peer review face common challenges such as, for example, resistances and closure of institutional or established practice in research and policy, different conceptual and operational framings, knowledge representations and mediation. The remaining of this paper will address the issue of knowledge representation and mediation.

3     Knowledge Representation & Mediation

In trans-disciplinarity literature, the issue of knowledge representation and communication is recurrent due to a perceived need to communicate more complex and dynamic insights, exploring the use of metaphors, patterns and analogy (see e.g. Judge 1995). Knowledge representation and mediation become an issue when different sources and types of knowledge have to be “integrated" or “fit in a framework of analysis" or have to be articulated in a same decision space. It often happens that the need to deal with a diversity of knowledges originates from those who are already used to a certain type of framing and the deployment of specific tools of assessment.

Traditionally, integration meant reductionism and the conversion of different languages into one single, mainly quantitative language (e.g. Cost Benefit Analysis or other mono-criterion evaluation techniques, such as multi-attribute theory). This tendency has persisted, despite the pitfalls of knowledge loss, poor scoping and increased controversy. The recognition of multiple perspectives has encouraged the use of frameworks trying to acknowledge and to operationalise a diversity of knowledge representations. Among such frameworks are multi-criteria evaluation (see, for instance, Munda 1995), integrated assessment modelling (see, for instance, Alcamo et al. 1994), multi-scale integration (Giampietro 2003). These attempts arise from the need to make comparisons, seek for trade-offs or even become Alephs (the place from where all dimensions could be seen at the same time, according to the poet J. L. Borges (in El Aleph written 1949 - see for instance Borges 2001) regarding alternative courses of action, policy making options and foresight.

In trans-disciplinary practice and extended peer review it is often the case that the framework in which knowledge integration and assessment occurs is that of research, characterised by concepts and tools that determine in a sense the ways in which knowledge has to be represented in order to be shared. Among others Giampietro (2003) talks about different narratives depending on who initiates the process, with which purpose, perspectives and values. Hence, the main challenges posed to trans-disciplinary research, extended peer review and other attempts to integrate different knowledge sources and types are, on one hand, the creation of spaces for knowledge representation and mediation and, on the other hand, the creation of spaces for knowledge co-production.

We argue that such methodologies should ultimately provide spaces to make sense of a variety of bits of knowledge, recognising the legitimacy of, not only different types of knowledge, but also different ways of articulating them. It is not surprising then that trans-disciplinary literature points to the use of metaphors, patterns, multi-media visualisation, isomorphism, analogies, and to methods that acknowledge diversity and complexity.

A promising development of this sort of knowledge conviviality is the creation of contexts of co-production of knowledge, entailing different types of knowledge organisation for non-scientific contexts, experimented in sustainability issues (see for instance, Guimarães Pereira, O'Connor 1999). We advocate that, as in extended quality assurance, trans-disciplinary practise requires a new type of skill, enabling the creation of multiple interfaces between scientific and non-scientific knowledges.

4     Trans-disciplinarity at Work: Case Study on Groundwater Resources

The project GOUVERNe [2] aimed at the development and pilot implementation of a user-based scientifically validated process and informatics product for the improved governance of groundwater resources. The Joint Research Centre activities in this project dealt mainly with the organisation of the available knowledge about two groundwater resources case studies in Europe (see Guimarães Pereira et al. 2003a and Guimarães Pereira et al. 2003b).

The methodology deployed was based on the concept of quality assurance by extended peer review, as a normative procedure to construct the knowledge base upon which a debate about water governance options could start in both case studies. What we called the “GOUVERNe process" was strongly based on trans-disciplinary principles, combining hybrid methodologies, integrating social research methods with evaluation tools, such as multi-criteria evaluation.

4.1     The GOUVERNe Process

Knowledge scrutiny in the GOUVERNe process was strongly based on social research. That was the means to ensure that knowledge other than scientific-technical was available in the knowledge base to debate on possible futures for groundwater resources and the associated river basins of the two case studies (in France and Greece).

The involvement of relevant social actors was done from the very first framing step which ensured that, early in the process, their perspective, concerns and ways of representing the issues were accounted for. The research framing acknowledged and shared by the relevant community helps to avoid the so called Type III error, of addressing the wrong problem, and enhances the scoping phase (i.e. focuses the work of collecting relevant information). The extended involvement also means that the issues addressed are shared and are dealt with at the appropriate depth.

Clearly, if the experts involved in the process are the only “digesters“ of the available knowledge (even if the process is inclusionary), their research framing and representation will be paramount. This is why the quality check by the relevant community throughout the whole process is essential for compliance and effectiveness, and why the process of creating socially robust knowledge (Gibbons 1999) is a continuous inclusionary process of those concerned. In GOUVERNe engagement of relevant social actors was done at several steps of the process.

What emerged from the processes of knowledge scrutiny is that activities and options explored together by those concerned, had great advantages in terms of enhancing the final process of dialogue, compared with those activities structured solely by “experts“: what becomes available as knowledge base to support the ongoing dialogues is perceived as a co-produced result and issues are then more easily appropriated by all those engaged.

One of the main research issues of this process was how to articulate different values and perspectives, as well as different representations of knowledge which may be presented through alternative narratives (language, framing, scales of measurement, numerical models, etc). GOUVERNe is about knowledge integration, while trying to keep diversity which in the interpretation of the researchers was the creation of a space: where different types of knowledge articulated in different sets of semantic rules, with different codes, different scales of evaluation, etc. could be represented through several formats implying various degrees of specialisation; where no a priori “integrative methodology“ was applied as the means of sharing knowledge, the integration being made through dialogue and interactions.

This entails the effort to produce a sort of “knowledge platform“ that is accessible to all those involved and promotes conviviality of different knowledges, including tools that help with the process evaluation, capturing plurality and diversity and avoiding the pitfall of reducing them to something plausible but meaningless. This was explored through the use of Information and Communication Technologies (ICT) and in particular multi-media knowledge representation.

4.2     Building Spaces for Conviviality: The TIDDD Concept

A major development within the GOUVERNe process was the realisation, the design and prototype implementation of a new concept tool: TIDDD - Tools to Inform Debates, Dialogues & Deliberations, deploying new ICT. The main characteristics of this tool can be defined as “tools that inform and mediate processes of debate, dialogue or deliberation which involve social actors of a governance, policy or decision process“. Mediation of knowledge in this case entails organisation, communication and exchange of a plurality of sources and types of knowledge (Guimarães Pereira et al. 2003b). In the case of GOUVERNe, there was a great deal of disciplinary knowledge, such as climate, geological and hydrological, as well as socio-economical, regulatory, etc. Scenario drivers to debate about future options were devised together with the social actors. Hence, as some modelling tools were used to characterise possible futures, there had to be some work on “translating“ that information in order to use it as input for the models. TIDDD's aim is the creation of convivial contexts of exploration, and “discovery“, where representations of knowledge come from different actors in the form of consistent narratives, aided by a multiplicity of supporting materials, namely multi-media formats, metaphors, etc. In TIDDD some pieces of information were represented through different media in order to reach different people involved. TIDDD can integrate other sources and types of knowledge that may emerge during the process, which is done through the available multi-criteria evaluation tool.

Quality assurance through extended peer review of TIDDD contents and design is one of the basic principles of this tool, since its main aim is to provide socially robust knowledge in contexts of societal debates, and even scientific controversy. This is achieved through upstream engagement of the relevant community in the implementation of the knowledge base available in TIDDD, where the social actors check all developments and ensure that contents and design are suitable to start the debate on groundwater resources futures.

5     Final reflection

TIDDD-like tools are interfaces of mediation between policy spheres and other sectors of the society. This mediation is done with the help of experts, but what comes out of the GOUVERNe process is that a new class of expert is emerging, experts in creating contexts for co-production of knowledge, experts in mediation of different types of knowledge, perspectives and values, and eventually experts in making scattered non-organised pieces of relevant knowledge intelligible to the organised and sometimes poorly flexible institutions: in a sense trans-disciplinary experts.

Trans-disciplinarity practice and extended quality assurance processes are about conviviality of different knowledges. It is hoped that tools like TIDDD can help to create the spaces where co-production and integration take place. The GOUVERNe TIDDD are in fact a trans-disciplinary platform.


[1] The views expressed in this article are those of the authors and do not represent necessarily those of the European Commission.

[2] Project no. EVK1-1999-00032: a Shared Cost Action financed by DG RTD, under FP 5. GOUVERNe stands for Guidelines for the Organisation, Use and Validation of information systems for Evaluating aquifer Resources and Needs. Online available at:


The work on TIDDD was financed by the European Commission under the GOUVERNe project (EC project # EVK1-1999-00032).


Alcamo, J.; Kreileman, G.J.J.; Krol, M.S.; Zuidema, G., 1994:
Modeling the Global Society-biosphere-climate System: Part 1: Model Description and Testing. In: Alcamo J. (ed.): IMAGE 2.0: Integrated Modeling of Global Climate Change. Dordrecht, The Netherlands: Kluwer Academic Publishers

Borges, J.L., 2001:
El Aleph. Emece Editores

Flinterman, J.F.; Teclemariam-Mesbah, R.; Broerse, J.E.W.; en Bunders, J.F.G., 2001:
Transdisciplinarity: The New Challenge for Biomedical Research. In: Bulletin of Science, Technology & Society, Vol. 21. No. 4, pp. 253-266

Funtowicz, S., 2001:
Peer Review and Quality Control. In: International Encyclopaedia of the Social and Behavioural Sciences. Oxford: Elsevier, pp. 11179-11183

Funtowicz, S.O.; Ravetz, J.R., 1990:
Uncertainty and Quality in Science for Policy. Dordrecht: Kluwer Academic Press

Giampietro, M., 2003:
Multi-Scale Integrated Analysis of Agroecosystems. London: CRC Press Llc.

Gibbons, M., 1999:
Science's new social contract with society. In: Nature 402, C81-C84

Guimarães Pereira, Â.; O'Connor, M., 1999:
Information and Communication technology and the popular appropriation of sustainability problems. In: Int. J. Sustainable Development, Vol. 2, No. 3, pp. 411-424

Guimarães Pereira, Â.; Rinaudo, J.D.; Jeffrey, P.; Blasques, J.; Corral Quintana, S.; Courtois, N.; Funtowicz, S.; Petit, V., 2003a:
ICT Tools To Support Public Participation in Water Resources Governance & Planning: Experiences From The Design and Testing of a Multi-Media Platform. In: Journal of Environmental Assessment Policy and Management. Vol. 5, No. 3, pp. 395-420

Guimarães Pereira, Â.; Blasques, J.; Corral Quintana, S.; Funtowicz, S., 2003b:
TIDDD - Tools To Inform Debates Dialogues & Deliberations. The GOUVERNe Project at the JRC. European Commission, Ispra: EUR 21880 EN

Klein, J.Th.; Grossenbacher-Mansuy, W.; Häberli, R.; Bill, A.; Scholz, R.W.; en Welti, M. (eds.), 2001:
Transdisciplinarity: Joint Problem-Solving among science, technology and society. An effective way of managing complexity. Basel: Birkhauser Verlag

Jasanoff, S., 1996:
Beyond epistemology: relativism and engagement in the politics of science. Social Stud. Sci. 26 (2), pp. 393-418

Judge, A., 1995:
Envisaging the Art of Navigating Conceptual Complexity in search of software combining artistic and conceptual insights. In: Knowledge Organization (22), 1995, 1, pp 2-9

Munda, G., 1995:
Multicriteria Evaluation in a Fuzzy Environment. Berlin: Physica Verlag

Nicolescu, B., 1999:
The transdisciplinary evolution of learning. Paper presented at the symposium on Overcoming the Underdevelopment of Learning at the Annual Meeting of the American Educational Research Association. Montreal, CDN, April 19-23, 1999. Online available at: http://www.learndev.org/ (9 April 2005)

Sarewitz, D., 2004:
How Science makes Environmental Controversies Worse. In: Environmental Science & Policy, 7, pp. 385-403



European Commission - DG Joint Research Centre
Via E. Fermi, 1 TP361, I - 21020 Ispra (VA)
Internet: http://ipsc.jrc.cec.eu.int


Ângela Guimarães Pereira
E-Mail: angela.pereira∂jrc.it

Silvio Funtowicz
E-Mail: silvio.funtowicz∂jrc.it