ESTUDIOS / RESEARCH STUDIES

RESEARCH ASSESSMENT IN HUMANITIES AND SOCIAL SCIENCES IN REVIEW

Elea Giménez Toledo*

* Grupo de Investigación sobre el Libro Académico (ILIA), Instituto de Filosofía (IFS), Consejo Superior de Investigaciones Científicas (CSIC)

E-mail: elea.gimenez@cchs.csic.es | ORCID iD: http://orcid.org/0000-0001-5425-0003

LA EVALUACIÓN DE LAS HUMANIDADES Y DE LAS CIENCIAS SOCIALES EN REVISIÓN

CONTENTS

1. INTRODUCTION Top

The evaluation of research activity is always a controversial and complex task, particularly the evaluation carried out at the individual level, but also at the level of group, department or institution. Criticism towards evaluation takes place in different countries with diverse research environments (Matthews, 2015Matthews, D. (2015). REF sceptic to lead review into research assessment. Times Higher Education. https://www.timeshighereducation.com/news/ref-sceptic-lead-review-research-assessment.; Kulczycki, 2017Kulczycki, E. (2017). Assessing publications through a bibliometric indicator: The case of comprehensive evaluation of scientific units in Poland. Research Evaluation, 26 (1), 41-52. https://doi.org/10.1093/reseval/rvw023.; Alperin y Rozemblum, 2017Alperin, J.P.; Rozemblum, C. (2017). The Reinterpretation of the Visibility and Quality of New Policies to Assess Scientific Publications. Revista Interamericana de Bibliotecología, 40 (3). https://doi.org/10.17533/udea.rib.v40n3a04.) and also internationally, as the San Francisco Declaration (DORA, 2013DORA (2013). San Francisco Declaration on Research Assessment. http://crln.acrl.org/index.php/crlnews/article/view/9104/9996.) or the Leiden Manifesto (Hicks et al. 2015Hicks, D.; Wouters, P.; Waltman, L.; De Rijcke, S.; Rafols, I. (2015). The Leiden Manifesto for research metrics. Nature, 520 (7548), 429-431. https://doi.org/10.1038/520429a.) underline. Criticism stems also from specific disciplines but, doubtlessly, it is in the Humanities and the Social Sciences where the strongest tensions take place and where the scientific community does not accept, or is highly critical towards the procedures employed in evaluation (Hug & Ochsner, 2014Hug, S.; Ochsner, M. (2014). A framework to explore and develop criteria for assessing research quality in the humanities. International Journal of Education Law and Policy, 10 (1), 55-68.; Giménez-Toledo, 2016Giménez-Toledo, E. (2016). Malestar. Los investigadores ante su evaluación. Madrid: Iberoamericana Vervuert., McCulloch , 2017McCulloch, S. (2017). The importance of being REF-able: academic writing under pressure from a culture of counting. LSE Impact blog. http://blogs.lse.ac.uk/impactofsocialsciences/2017/02/09/the-importance-of-being-ref-able-academic-writing-under-pressure-from-a-culture-of-counting/.).

Most of the criticisms towards evaluation in those fields, specifically referred to publications as research results, could be summarized into two main points: evaluation procedures are scarcely adequate to research and scientific communication practices that are consubstantial to the Humanities and the Social Sciences. Two well known examples are: a) books, book chapters and national scholarly journals are more relevant in the case of the Humanities and the Social Sciences than they are in Science, Technology and Medicine, and citations are clearly different (Hicks, 2004Hicks, D. (2004). The four literatures of social science. In: Handbook of quantitative science and technology research. Dordrecht: Springer, pp. 473-496.); and b) the information sources traditionally used for the evaluation of scholarly publications and/or to which more weight is given in science policy are limited: they provide a limited coverage of the diversity of journals that are relevant for these fields, neither do they adequately cover other scientific communication channels such as books, proceedings or reports. Consequently their coverage is poor in terms of research topics, languages and local or national issues, frequent in the Humanities and the Social Sciences. This becomes particularly notorious since all studies characterizing the Humanities and, up to a certain extent, the Social Sciences, systematically conclude that in those fields the research is closer to a geographical area and is often focused on local issues; in connection to the previous point, research results are published in a variety of languages because they are relevant for that geographical area under study and that national journals are used more frequently for the same reason; and, finally, books and book chapters are much more relevant than in other fields in order to communicate research results. In fact, these are predominant in the scientific output of many Humanities fields (Ochsner et al., 2017aOchsner, M.; Hug, S.; Galleron, I. (2017a). The future of research assessment in the humanities: bottom-up assessment procedures. Palgrave Communications, 3. https://doi.org/10.1057/palcomms.2017.20.; Tanner, 2016Tanner, S. (2016). An analysis of the Arts and Humanities submitted research outputs to REF 2014 with a focus on academic books. https://academicbookfuture.files.wordpress.com/2016/11/abof_academic-books-ref2014-report_simon-tanner.pdf.; Giménez-Toledo, 2016Giménez-Toledo, E. (2016). Malestar. Los investigadores ante su evaluación. Madrid: Iberoamericana Vervuert.).

To the previously mentioned it is necessary adding a further substantive questioning. Scholarly publications are the materialized, tangible and measurable result of research. From that point of view, publications are a fundamental element for the judgment of research activity. Nevertheless, nowadays not only the generation of such scientific knowledge and its publication is required, but also its impact and influence in society. Therefore, the evaluation of scientific publications in the scholarly curriculum is insufficient and it becomes necessary addressing other facets of research activity.

All these questions are being analyzed with increasing attention. The research on communication channels and scientific and societal impact of the Humanities and the Social Sciences is growing. The main international conferences on Research Evaluation, Bibliometrics and Scientometrics include specific sections of their programs to the various advances in that type of research. This is the case of the conferences of the International Society for Informetrics and Scientometrics (ISSI)[1] or Science, Technology & Innovation Indicators (STI)[2]. Also, the evaluation of the Humanities and the Social Sciences is the main subject of research in the framework of the COST action European Network for Research Evaluation in the Social Sciences (ENRESSH)[3], of the initiative EvalHum[4] and of the international conference it organizes: Research Evaluation in the Social Sciences and the Humanities (RESSH)[5]. These initiatives, unlike more traditional research, underline the non-bibliometric aspects of these fields; although the bibliometric aspects are part of the agenda too, particularly those involving novel methodologies.

2. OBJECTIVES Top

This work has as main objective to review the most relevant trends in the evaluation of the Humanities and the Social Sciences in the international context, particularly in Europe. This review process is aimed at the identification of the characteristics and main changes occurring in evaluation practices. Although it is not possible to encompass all aspects of evaluation in a systematic and detailed way because of the very nature of the review article, it is intended that the exposition of some of the current key aspects governing evaluation processes nowadays allows understanding the substantial changes which are taking place and the sense of that evolution.

Also, this review of the state of the art aims at showing the dynamism observable in research on evaluation in these fields, and which responds to the need for better evaluation. Likewise, this article aims at serving as a base for the knowledge of the different paths that can be followed in the evaluation of the Humanities and the Social Sciences, set apart from traditional bibliometric methods and more adjusted to research practice in those fields.

3. METHODOLOGY Top

Trends and distinctive features described in this article, showing the evaluation practices in the Humanities and the Social Sciences, are a synthesis of the continued research in this field. More specifically, the contents of this article are derived from a) the review of the scientific literature published in specialized channels; b) the active participation in the COST European Network for Research Evaluation in the Social Sciences (ENRESSH CA-15137), in its working groups, conferences and in specialized publications in collaboration on this topic; c) the interaction with scholars, book and journal publishers and evaluation agencies around the evaluation of scientific output, in scholarly activities.

4. CURRENT KEYS TO THE EVALUATION OF THE HUMANITIES AND THE SOCIAL SCIENCES Top

4.1. Full data and sources of indicators for publication channels

Recent research shows the existing diversity of evaluation models present in Europe at the moment (Ochsner et al. 2017bOchsner, M.; Kulczyck, E.; Gedutis, A. (2017b). Diversity of Research Evaluation Systems in Europe. Nordic Workshop for Bibliometrics, Helsinki, Finland. http://www.helsinki.fi/astro/library/nwb22/D1-05-Ochsner.pdf.). Such diversity is defined by various variables such as centralization or decentralization, the availability of information about scientific research at the institutional, regional or national level, the greater or more limited consideration of research practices in different fields or the inclusion of one or more dimensions of research in those evaluation processes. Certainly, many of them have scholarly publications as main object of evaluation. Not only are they the most tangible and measurable output of research activity, but they can also be considered the tip of the iceberg: where there are publications, there are research, dissemination, innovation or collaboration networks.

Maybe a first distinction in the models can be made between those countries with national information systems (Current Research Information Systems) integrating the whole of the scientific output and those which do not count with them. Countries such as Norway, Finland, Denmark and the Flanders region (Belgium) have built CRIS which allow them to count with the full output of each country/region. Also, the Lattes[6] platform in Brazil can be considered an information system at the national level. Counting with full publication data allows making evaluation in context. The starting point of evaluation is the whole scientific output of a country; from that it is possible to identify publication patterns in the different fields and apply assessments and / or weighting procedures to publication channels, mainly journals and scholarly book publishers (Giménez-Toledo et al., 2016Giménez-Toledo, E.; Mañana-Rodríguez, J.; Engels, T. C. E.; Ingwersen, P.; Siverten, G.; Verleysen, F. T.; Zuccala, A. (2016). Taking scholarly books into account: current developments in five european countries. Scientometrics, 107 (2), 685-699. https://doi.org/10.1007/s11192-016-1886-5.). Categorizations or levels assigned to scientific output are developed by expert panels from information generated by research groups and publishers’ associations.

It is relevant to underline, as an starting point, in these systems all the scientific output is taken into account, not considering only a small fraction which is visible in international databases such as Web of Science (Clarivate Analytics) or Scopus (Elsevier). This is a fundamental difference. In other countries, the model is (or has been) the opposite: there are no national databases and researchers send evaluation agencies their curricula for these to be assessed, and then, two situations can occur.

The first of them, quite common until recently, gives recognition –primarily- to scientific output included in the commercial databases Web of Science and Scopus. That scientific output is limited to the selection of sources made by the producers and it is limited, essentially, to scientific journals and books from a very limited set of scholarly publishers. It is relevant to remind here the limitations produced by evaluation relying on those databases. There are innumerable studies showing the biases of those databases and the consequent difficulties for the evaluation of output in Humanities and Social Sciences. One of the most recent ones (Gingras and Khelfaoui, 2017Gingras, Y.; Khelfaoui, M. (2017). Assessing the effect of the United States’ “citation advantage” on other countries’ scientific impact as measured in the Web of Science (WoS) database. Scientometrics, 114 (2), 517-532. https://doi.org/10.1007/s11192-017-2593-6.) shows how citations play in favor of countries with a closer relationship to the United States. The geographical distribution of the journals and publishers selected shows how incomplete these sources are for the evaluation of the scientific output of a country. Just as an example: a recent study on the Dominican Republic’ scientific output in international databases reveals that they index only 49% of the output. It is to say, 51% remains invisible if other sources are not taken into account (Riggio, 2017Riggio, G. (2017). Indicadores bibliométricos de la actividad científica de la República Dominicana. Madrid: Universidad Carlos III. Tesis doctoral.). Consequently, when the decision is made to evaluate using only those information sources, not only is a large part of the research activity being undervalued, but also relevance is being downplayed on the type of research that can be more useful for that region or country, which is usually the research that remains out of the international databases.

The second refers to evaluation processes which are not backed by a CRIS but using a richer approach for the evaluation of research outputs. In fact, in recent years it is possible to note that some national evaluation exercises do assess research output counting with diverse sources of indicators. Apart from WoS or Scopus, other databases such as ERIH Plus (European Reference Index for the Humanities in its most recent version) or other international categorizations of journals; also, databases or national indexes providing information on the scientific journals’ characteristics are taken into account. Such is the case of the lists of publication channels in the BFI / BRI Bibliometric Research Indicator in Denmark[7], the Finnish system, categorized by the Julkaisuforumi[8], the Norwegian system, developed by the Norwegian Register for Scientific Journals, Series and Publishers[9], or the evaluation system of the Flanders Region (Belgium), through the Flemish Academic Bibliography for the Social Sciences and Humanities VABB-SHW[10]. Even in some cases, such as in Spain, information sources on scholarly books such Scholarly Publishers Indicators (SPI)[11] or quality labels for book series (CEA / APQ)[12] are mentioned as information sources.

Rankings, categorizations or quality labels for books and scientific journals are tools being used in almost any evaluation process. Somehow, the use of those sources confirms the need for supporting decisions made on scientific output on objective indicators. Probably, the use of the tools is the most controversial part. Indicators on publications should be used always in combination with the criterion of experts. This has always been the golden standard in bibliometrics. Nevertheless, in some occasions evaluation processes are excessively simplified and automatized, making automatic translations of the value of a scientific contribution with regards to the position of a journal or a publisher in a ranking. Indicators of publishers should provide solid and useful information to the expert, but they are not entitled to determine the final result of the evaluation. “Metrics should support, not supplant, expert judgement” is the conclusion of The Metric Tide (Wilsdon et al., 2015Wilsdon, J.; Allen, L.; Belfiore, E.; Campbell, P.; Curry, S.; Hill, S.; Jones, R.; Kain, R.; Kerridge, S.; Thelwall, M.; Tinkler, J.; Viney, I.; Hill, J.; Wouters, P.; Johnson, B. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. https://doi.org/10.13140/RG.2.1.4929.1363.), an independent report on metrics in scientific research.

The lack of CRIS prevents from counting with a detailed knowledge on the levels of output by scientific fields, co-authorship indexes o international collaboration, the relevance of the different communication channels or languages for the communication of scientific knowledge. In all, it inhibits counting with context data at the country level on the communication pattern in the different disciplines. However, in some occasions, there are no national information systems but institutional ones. Those systems, repositories or databases specifically designed for tracing of an institution’s scientific output allow counting with precise information for evaluation in a specific university or research institution (De Filippo et al., 2011De Filippo, D.; Sanz-Casado, E.; Salido, C. U.; Ardanuy, J.; Gómez-Caridad, I. (2011). El papel de las bases de datos institucionales en el análisis de la actividad científica de las universidades. Revista española de documentación científica, 34 (2), 165-189. https://doi.org/10.3989/redc.2011.2.797.; Reale et al, 2011Reale, E.; Filippo, D. De; Gómez, I.; Lepori, B.; Potì, B.; Primeri, E.; Probst, C.; Sanz-Casado, E. (2011). New uses of the institutional databases of universities: indicators of research activity. Research Evaluation, 20 (1), 47-60. https://doi.org/10.3152/095820211X12941371876300.). In example, the Spanish National Research Council (CSIC) counts with the information system ConCiencia, while a large proportion of Spanish universities work with the software Universitas XXI for the monitoring of their research activity, among other objectives.

In the evaluation processes at the national level of countries without CRIS, the different committees or expert panels have that context defined, more or less, but without the data susceptible of confirming the publication patterns and their evaluation through time. Besides, through giving priority to scientific output included in international databases, a secondary relevance is given to research closer to the territory, which can have a societal impact –and not only a scholarly one- on the closest community.

In that sense, international publication is favored (understanding ‘international’ in a restrictive sense, since it is what is selected by the database producers) and, therefore, it takes into account topics and methodologies which are interesting in an international and globalized context. Nevertheless, it is necessary to consider too the publications which are outside that framework, not only because these disseminate scientific knowledge in other topics which are relevant for the close communities but also because, in this way, the thematic, linguistic, methodological and ideological diversity present in research is being preserved (Ochsner et al., 2017bOchsner, M.; Kulczyck, E.; Gedutis, A. (2017b). Diversity of Research Evaluation Systems in Europe. Nordic Workshop for Bibliometrics, Helsinki, Finland. http://www.helsinki.fi/astro/library/nwb22/D1-05-Ochsner.pdf.; Giménez-Toledo, 2017Giménez-Toledo, E. (2017). Edición académica en español: la diversidad como desafío. Revista de Occidente, 438, 77-103. http://www.ortegaygasset.edu/publicaciones/revistadeoccidente/noviembre-2017.). For example, research questions and their corresponding answers in indexed international journals of almost any discipline are not, in most cases, the primary concern of Latin America and Spain. Vessuri et al. (2013Vessuri, H.; Guédon, J. C.; Cetto, A. M. (2013). Excellence or quality? Impact of the current competition regime on science and scientific publishing in Latin America and its implications for development. Current Sociology, 62 (5), 647-665. https://doi.org/10.1177/0011392113512839.) allude to the internationalization of research in the following terms: ‘If the science of Latin America wants to become international on a solid base, it should not consider erroneously that Northern Atlantic’s science is the totality of the world’s science’ (p.12). The various arguments in that sense point out that if non-indexed publications are not taken into account, a large part of the essential research for national or regional contexts will be lost.

Recognizing the singularities of the scientific production in Humanities and Social Sciences and, at the same time, the interest in the observation of that production in an international framework -not only in the national ones- and from a compared perspective, the creation of a database covering the publications produced in those fields in Europe has been proposed long ago (Martin et al, 2010Martin, B. R.; Tang, P.; Morgan Jones, M. (2010). Towards a bibliometric database for the social sciences and humanities-A European Scoping Project. http://sro.sussex.ac.uk/29810/ .). Although the proposal was not carried out in practice, the idea is still valid. In fact, it is part of the objectives of the COST action European Network for Research Evaluation in the Social Sciences and the Humanities (ENRESSH, 2015ENRESSH (2015). Memorandum of understanding for the implementation of the COST action European Network for Research Evaluation in the SSH CA15137. http://enressh.eu/wp-content/uploads/2016/10/CA15137-e.pdf.). This network is collaborating with the VIRTA (Puuska et al, 2017Puuska, H.M.; Pölönen, J.; Engels, T.; Sivertsen, G. (2017). Towards the integration of European research information. Spring 2017 Membership Meeting of euroCRIS, Dublin. https://dspacecris.eurocris.org/bitstream/11366/593/1/euroCRIS_MMDublin_Puuska_Towards_integration_of_European_research_information_20170531.pdf) project in a technological infrastructure which would allow the aggregation of scientific output data from different countries in a normalized and interoperable way. The feeding system is decentralized, from each of the countries or institutions. As the project advances, and more countries participate, it will be possible to create a large database of scientific output that would allow not only the development of studies on publication dynamics, collaboration, etc. in different fields but also to count with information for the evaluation of the Humanities and the Social Sciences.

4.2. Towards a more qualitative/bottom up evaluation

Content evaluation from published research has been and is still a common claim of humanists and social scientists. In some occasions they criticize the evaluation made on the publication channels, mainly journals and publishers, dissociating the content from the continent. In the light of the studies made on this issue (McCulloch, 2017McCulloch, S. (2017). The importance of being REF-able: academic writing under pressure from a culture of counting. LSE Impact blog. http://blogs.lse.ac.uk/impactofsocialsciences/2017/02/09/the-importance-of-being-ref-able-academic-writing-under-pressure-from-a-culture-of-counting/.; Giménez-Toledo, 2016Giménez-Toledo, E. (2016). Malestar. Los investigadores ante su evaluación. Madrid: Iberoamericana Vervuert.), the scholarly community feels the pressure exerted on them by evaluation, the demands for publishing more and through some specific channels and, in relation to this, the deviation from their research topics in order to publish in the journals or publishers better valued in evaluation. As Ochsner et al. (2017aOchsner, M.; Hug, S.; Galleron, I. (2017a). The future of research assessment in the humanities: bottom-up assessment procedures. Palgrave Communications, 3. https://doi.org/10.1057/palcomms.2017.20.) point out, researchers in these areas have strong reservations with regards to quantification.

There are several factors to be considered with regards to such positioning (Giménez-Toledo, 2016Giménez-Toledo, E. (2016). Malestar. Los investigadores ante su evaluación. Madrid: Iberoamericana Vervuert., p. 23 and next pages). First, there are few qualitative scientific research evaluation schemes. The last Research Excellence Framework (REF 2014Research Excellence Framework (REF) (2014). Main panel D criteria. http://www.ref.ac.uk/2014/media/ref/content/pub/panelcriteriaandworkingmethods/01_12_2D.pdf.) (Panel D) in the UK was one of them, to the point of explicitly noting in its call that the publication channel or the journals or publishers’ classifications were not going to be considered by Humanities and Social Sciences’ panels (Research Excellence Framework, 2014Research Excellence Framework (REF) (2014). Main panel D criteria. http://www.ref.ac.uk/2014/media/ref/content/pub/panelcriteriaandworkingmethods/01_12_2D.pdf.). Expert panels would be the main procedure. The methodology was applied thanks to a vast investment in the evaluation process, since it is costly in money and time. At the same time, since it is a voluntary evaluation exercise which is performed on the researchers’ output for a six-year period, it is feasible, by contrast with the periodical evaluation processes, usually with an annual periodicity, which are done at the institutional level. The REF 2014Research Excellence Framework (REF) (2014). Main panel D criteria. http://www.ref.ac.uk/2014/media/ref/content/pub/panelcriteriaandworkingmethods/01_12_2D.pdf. has been, nevertheless a controversial and criticized evaluation exercise.

Qualitative evaluation based only on experts’ criteria is not free from problems, since there are not only schools of thought, ideologies or methodological appreciations that imply positioning oneself against or in favor of a given line of research. Also, as Ochsner et al. (2017aOchsner, M.; Hug, S.; Galleron, I. (2017a). The future of research assessment in the humanities: bottom-up assessment procedures. Palgrave Communications, 3. https://doi.org/10.1057/palcomms.2017.20.) point out: “the way SSH scholars appreciate research output of colleagues is quite different from how STEM researchers do. SSH scholars are much more critical. They criticize even work they value as excellent”. The fact that evaluations were based only on the specialists’ judgment would not guarantee a better acceptance of evaluation processes. That reason, together with costs and evaluation time spans involved in evaluation processes, contributes to the fact that many evaluation models are based on a combination of experts’ criteria (the qualitative side) and indicators (the quantitative side).

Concerning the lack of acceptance of quantitative indicators by humanists and social scientists, it is relevant to establish differences between indicators and, of course, to claim the value and usefulness of some of them. In the light of some of the reluctances that they cause, it might seem as if the indicators of a publication –its prestige, internationality, rigor in the selection of manuscripts- were values completely foreign to the idea, judgment or perception that the academia has on that publication. It might seem that the judgment of experts on a given publication channel (journal, publisher, etc.) cannot be translated, by no means, into a numerical value or a category. Bibliometric indicators are not always mere measures of the output quantity or citations received. Maybe because in some occasions they have been excessively or incorrectly used, or perhaps because numeric measurements are usually foreign to Humanities, there is a reluctance concerning their use. Nevertheless, a numerical indicator or a category might represent different qualities of the research or of the publications. When an expert panel meets in order to determine the quality level of a set of journals and they resume their judgments in a set of categories (see the Finnish, Danish or Norwegian system in Giménez-Toledo et al, 2016Giménez-Toledo, E.; Mañana-Rodríguez, J.; Engels, T. C. E.; Ingwersen, P.; Siverten, G.; Verleysen, F. T.; Zuccala, A. (2016). Taking scholarly books into account: current developments in five european countries. Scientometrics, 107 (2), 685-699. https://doi.org/10.1007/s11192-016-1886-5.) a translation is taking place in practice, which implies facilitating an evaluation process taking into account the specialists’ knowledge. The same happens in a consultation to the scholarly community on the prestige of publishers, as the one involved in the creation of the indicators’ system Scholarly Publishers Indicators. Or when, for the creation of that same system, thousands of bibliographic registers are analyzed in order to determine the specialization profile of a publisher (Giménez-Toledo and Mañana-Rodríguez, 2016Giménez-Toledo, E.; Mañana-Rodríguez, J. (2016). Lo cuantitativo en las Humanidades: los casos de la evaluación y la edición científica. Índice. Revista de Estadística y Sociedad, 69, 25-27. http://www.revistaindice.com/numero69/p25.pdf.). The construction of an indicator implies the formalization of knowledge that, until that moment, has remained ‘informal’ or tacit, thus achieving its usefulness for the evaluation processes. An enlightening article by Ferrara and Bonaccorsi (2016Ferrara, A.; Bonaccorsi, A. (2016). How robust is journal rating in Humanities and Social Sciences? Evidence from a large-scale, multi-method exercise. Research Evaluation, 25 (3), 279-291. https://doi.org/10.1093/reseval/rvv048.) shows how the theoretical distance between the opinions of experts on journals and the indicators is not such, or is not so much. They compared two evaluation processes of scientific journals, one based on qualitative judgments and the other on categorizations and show, through regression analysis, that the results are in fact quite similar. This implies that it is feasible to make that translation of qualitative criteria into quantitative indicators and that the tools that include them allow to efficiently face the overloaded evaluation processes of scientific activity.

It is also necessary to distinguish between the evaluation systems and their sources with regards to the objectives of each of them. Often, evaluation is mentioned in general terms, but it is not the same carrying out an institutional evaluation process in order to improve transparency and to develop better guidelines in science policy, than to evaluate for the assignment of resources of research, or to evaluate an individual researcher for promotion. Procedures and tools would vary depending on the objectives. Sivertsen (2017aSivertsen, G. (2017a). Unique, but still best practice? The Research Excellence Framework (REF) from an international perspective. Palgrave Communications, 3. https://doi.org/10.1057/palcomms.2017.78.) establishes a clear classification of evaluation systems in countries counting with funding systems based on research results. In that classification it can be observed how the different objectives of the evaluation systems have propitiated, in some cases, the use of expert panels in evaluation and in others the use of indicators, with differences also between the types of indicators being used. Going into such diversity of objectives and procedures should make clear that generalizations, in research evaluation too, are neither positive nor constructive.

One of the key ideas concerning evaluation processes that are claimed nowadays is that the scholarly community has to participate in the design of the process or the indicators. The so-called bottom-up approach has been thought of as a good solution for the improvement of the evaluation processes (Ochsner et al, 2017aOchsner, M.; Hug, S.; Galleron, I. (2017a). The future of research assessment in the humanities: bottom-up assessment procedures. Palgrave Communications, 3. https://doi.org/10.1057/palcomms.2017.20.). The reason is that these procedures take into account the scientific research and communication practices of humanists and social scientists, which implies not only an evaluation more adjusted to reality but also, a priori, a better acceptance of the results by the academic community. It is necessary to underline ‘a priori’ because even the procedures or sources of information that are specifically created thinking in these disciplines and which involve their scholars are often controversial or opposed by researchers. The first stage of ERIH, European Reference Index for the Humanities, is an example of this, maybe particularly notorious because of the European scope of the project. This project became ERIH Plus (2017ERIH Plus revision (2017). https://dbh.nsd.uib.no/publiseringskanaler/erihplus/about/erih_revision_completed.), a bibliographic information source of Humanities and Social Sciences’ scholarly journals which informs on the accomplishment of certain quality criteria that are demanded to that type of publications: external peer review, existence of an editorial committee, institutional affiliation of authors, etc. Paradoxically, the intervention of specialists in the fields – a common claim- in order to categorize the journals in ERIH in its first stage was so controversial (Journals under threat, 2009Editorial: Journals Under Threat: A Joint Response from History of Science, Technology and Medicine Editors (2009). Centaurus, 51, 1-4. https://doi.org/10.1111/j.1600-0498.2008.00140.x.) that nowadays it is not taken into account. ERIH Plus is a source of bibliographic information more than a source of information potentially useful for research evaluation.

Not only ERIH has been discussed by the academic community. Journals or publishers’ categorizations, weightings of indicators or the different facets of research activity that are taken into account (in example, the value of scientific dissemination or the management of scientific journals) are a permanent object of discussion in the national evaluation processes. It is a common feature to all countries, independently from the evaluation model in each of them. Sometimes the reaction against the evaluation systems or its resolutions is so overwhelming that it is taken to court. This is a surprising fact since nowadays evaluation is carried out with much more information, better indicators and, often, with the participation of the scientific community. Expert panels are usually part of most evaluation processes. Also, scientific associations communicate evaluation agencies their opinions and position concerning evaluation criteria. In general, the conditions for the evaluation have improved. However, maybe because of reductionist views on science –only what is published in international, indexed journals is valuable- or because of the systematic opposition to evaluation procedures or to quantitative indicators, there is still a lot to be done in order to reach evaluation models with wide acceptance.

In most cases, when proposing the integration of bottom up approaches to evaluation processes, it implies the combined use of indicators and peer review with the participation of the corresponding academic communities. Some examples of those approaches are the ones applied in the countries using the so-called Norwegian model or the one used for the creation of the prestige rankings of publishers Scholarly Publishers Indicators (SPI)[13]. In the first case, as mentioned before, the starting point is the set of the full output of a country. It is evaluated in context, since it is possible to identify patterns of scholarly work and publication in the different fields and, also, the judgment of specialists is considered. The Norwegian model counts with expert panels that validate the publication channels and establish quality levels for them.

In the case of SPI, the information on the prestige of publishers is obtained from readers and authors of the books published by those publishers (Giménez-Toledo, 2018Giménez-Toledo, E. (ed.) (2018). Estudio cualitativo de las editoriales académicas. La percepción de la comunidad científica española. Madrid: Federación del Gremio de Editores de España.). It is based on a consultation to the academic community that, in its condition of specialist, is the collective in the best position to evaluate the quality of the publishers’ title list. The consultation allowed the publication of prestige rankings of the publishers that was understood as an orientation in the evaluation processes. That approach is not the only one, since the SPI system also provides information on the specialization of the publishers’ title list, the manuscript selection process, the use of metadata, the grouping of their title list into collections or their presence in other information systems. Both, prestige indicators and the manuscript selection processes are elements explicitly mentioned by Spanish evaluation agencies, Agencia Nacional de Evaluación de la Calidad y Acreditación (ANECA, 2017ANECA (2017). Méritos evaluables para la acreditación nacional para el acceso a los cuerpos docentes universitarios. http://www.aneca.es/Programas-de-evaluacion/ACADEMIA/Criterios-de-evaluacion-noviembre-2017.) and Comisión Nacional Evaluadora de la Actividad Investigadora (Ministerio de Educación, Cultura y Deporte, 2017Ministerio de Educación, Cultura y Deporte (2017). Resolución de 23 de noviembre de 2017, de la Comisión Nacional Evaluadora de la Actividad Investigadora, por la que se publican los criterios específicos aprobados para cada uno de los campos de evaluació. Boletín Oficial del Estado, nº 292. Madrid, España. https://www.boe.es/boe/dias/2017/12/01/pdfs/BOE-A-2017-14085.pdf .) in fields of Humanities and Social Sciences.

4.3. Open Accesss and its implications for research evaluation

A key transformation in the way of making science and communicating research results is taking place because of the principles of open science. From the postulation stating that society as a whole should benefit from and have access to the results of scientific research developed with public funding, new academic practices emerge:

• The open publication of results, in the form of articles, books and other types of documents (reports, working documents, etc.).

• The open publication of research data in order to facilitate their use by other researchers as well as with regards to the reproducibility of research.

• The evaluation of research activity from open sources (citations from Google Scholar, alternative metrics from sources such as Twitter, Facebook, Research Gate, Academia, etc.) by contrast with closed databases.

• The assessment or academic impact (open citations) by contrast with non-academic impact which can also be established from open sources (Ràfols et al., 2017Ràfols, I.; Robinson-García, N.; Leeuwen, T. van (2017). How to make altmetrics useful in societal impact assessments: shifting from citation to interaction approaches. Impact of Social Sciences Blog. http://blogs.lse.ac.uk/impactofsocialsciences/2017/03/23/how-to-make-altmetrics-useful-in-societal-impact-assessments-shifting-from-citation-to-interaction-approaches/ .).

All these are topics to which a great deal of attention is being given in research and that have entered the scientific policy agendas, particularly with regards to open publication of results and data. The Recommendation of the European Commission concerning open access and the preservation of scientific information (European Commission, 2012European Commission (2012). Commission recommendation of 17.7.2012 on access to and preservation of scientific information. http://ec.europa.eu/research/science-society/document_library/pdf_06/recommendation-access-and-preservation-scientific-information_en.pdf.), the future creation of a platform for the publication of open results, also proposed by the European Commission (European Commission, 2017European Commission (2017). Towards a Horizon 2020 platform for open access. Nota informativa. https://ec.europa.eu/research/openscience/pdf/information_note_platform_public.pdf#view=fit&pagemode=none.), the different laws of science, including the Spanish one (España, 2011España. Jefatura del Estado (2011). Ley 14/2011, de 1 de junio, de la Ciencia, la Tecnología y la Innovación. Boletín Oficial del Estado, nº 31. https://www.boe.es/boe/dias/2011/06/02/pdfs/BOE-A-2011-9617.pdf .), and the different calls for research projects’ proposals, among which there is the current European Research and Innovation Program, H2020[14], show that from a political perspective, open access to scientific knowledge is a firm and secure commitment. However, the adoption of open access entails many changes in the academic environment.

If authors are obliged to publish in open access, either by own conviction or because of the imperatives of project calls, they need to decide how to do it. They can opt for the green or the golden road. In that latter case –immediate publication of open access results-, the publication implies, usually, the payment of fees to the consolidated academic publishing houses. Without that payment, the content of the article would remain closed, only accessible to those with subscription to a specific journal, and only after the embargo period had expired would the author be able to deposit the article in a repository (green road).

Usually, those fees (APCs: Article Processing Charges) are paid by authors since their projects count with funds for open access publication. The public calls foresee specific budget items for open access publication. This implies an impulse to open access. Nevertheless, there are many doubts surrounding the sustainability of the system and, in fact, that is one of the main debate topics. On the one hand, it is not possible to fund all the publications produced. On the other, public funds devoted to research end up serving as payment to big publishing companies that not only design specific programs for open access publication in order to facilitate the accomplishment of the mandates on authors, but also are capable of attracting the reception of more original articles (and more income) because their journals are indexed in international databases, are part of the ‘system’ and are, consequently, widely recognized by evaluation agencies of many countries. The academic publishing industry has adopted solutions for making open access to research an opportunity rather than a threat. Studies by Björk and Solomon (Björk and Solomon, 2012Björk, B. C.; Solomon, D. (212). Open access versus subscription journals: a comparison of scientific impact. BMC medicine, 10 (1), 73. https://doi.org/10.1186/1741-7015-10-73.; Solomon and Björk, 2012aSolomon, D. J.; Björk, B. C. (2012a). A study of open access journals using article processing charges. Journal of the Association for Information Science and Technology, 63 (8), 1485-1495. https://doi.org/10.1002/asi.22673. and 2012bSolomon, D. J.; Björk, B. C. (2012b). Publication fees in open access publishing: Sources of funding and factors influencing choice of journal. Journal of the association for information science and technology, 63 (1), 98-107. https://doi.org/10.1002/asi.21660.) clearly show the remarkable increase in the number of scientific journals within large publishing houses that allow the publication in open access through the payment of the corresponding fees. Those same publishing houses have also created specific publication programs of monographs and edited volumes in open access. At the same time, small publishing houses and university and institutional presses publishing journals and scholarly books suffer the consequences of the commercial strategies of large publishing houses; when open access is supported, the funding model is different: the cost of that publication is assumed by the subsidy or funding of the university.

It is known that in Social Sciences and, particularly, in the case of the Humanities, books occupy a prominent place in the communication of research results. However, publishing them in open access is costly. The report by Jubb (2007Jubb, M. (2017). Academic Books and their Futures: A Report to the AHRC and the British Library. London.) places the figure for the fees in 6,500 pounds (BPCs: Book Processing Charges) for the publication with Cambridge University Press, 10,000 for the publication with Taylor & Francis and 11,000 in the case of Palgrave. Given that those costs, together with the lower availability of funds in the case of the Humanities and the Social Sciences, represent a serious sustainability problem, in recent years a series of initiatives are taking place in order to provide solutions to that challenge. For example, Knowledge Unlatched[15] proposes the costs associated to the publication of a monograph in open access, selected, edited and published by a consolidated publishing house to be assumed by university libraries wishing to participate in this co-funding system. More than 340 titles have been published using this model. Also, there are new publishing houses such as the Academic UCL Press that publish directly in open access and is funded by the University (University College London), in what constitutes a firm commitment, since scientific knowledge is therefore publicly available.

Regarding the evaluation of Humanities and Social Sciences, all the issues related to open access are of great interest. As already mentioned, evaluation processes take into account, among other things, publication channels. This implies that consolidated journals and publishing houses, which also allow the publication in open access once the fees have been paid, would be able to attract more and better original works. There remains, nevertheless, the economic sustainability problem, but also the problem of diversity. This can be observed, for example, in the case of university presses from different countries that publish directly in open access without the requirement of fees, or in the case of relevant publishers at the national level, both commercial and university presses, which have not adopted open access in the case of monographs because their business model consists, mainly, on selling books, because they do not participate in any publications co-funding system and the investment in publishing would be as sunk costs or, simply, because there are suspicions concerning intellectual property. Data help showing the dimensions of open access publication of books. According to DOAB (Directory of Open Access Books), an open source with worldwide information, 240 publishers are working with open access monographs. Although in that number are some of the most relevant publishers worldwide, with a large publishing output, these are a limited percentage with regards to existing scholarly publishers. Only in Spain, for example, there are more than 200 publishers with a purely scholarly profile (Giménez-Toledo, 2017Giménez-Toledo, E. (2017). Edición académica en español: la diversidad como desafío. Revista de Occidente, 438, 77-103. http://www.ortegaygasset.edu/publicaciones/revistadeoccidente/noviembre-2017.); in France, there are about 75 institutional publishers (Henny, 2015Henny, J. M. (ed.) (2015). L’edition scientifique institutionelle en France. Paris: AEDRES. Rapport.); in Colombia, more than 60 university presses[16]; and in Mexico, there are 42 publishers affiliated to Altexto[17] (network of university presses). As pointed out in the report A landscape study on open access and monographs (Ferwerda et al. 2017Ferwerda, E.; Pinter, F.; Stern, N. (2017). A landscape study on open access and monographs: Policies, funding and publishing in eight European countries. Knowledge Exchange. https://doi.org/10.5281/zenodo.815932.), international markets, the habits of readers or the funding possibilities in the different countries contribute shaping the publishing models and the predisposition to participate in open access monographs’ publishing projects.

The diversity in research and publishing within the Humanities and the Social Sciences is and will be a crucial factor in the evaluation of these fields. Since they generate a less monolithic knowledge than the positive sciences, the patterns and publication channels are more diverse, and this has implications in scientific evaluation: it requires more complete sources that provide information on the wide range of journals, publishers and other communication channels. These sources should combine the diversity and plurality in research topics, languages or methodologies with the quality and/or rigor in the selection of the texts that are published. Precisely because publication channels in these fields are far more numerous and because it is necessary to distinguish the more consistent research and the more selective channels, the development of objective indicators providing additional information is fundamental.

The multiplication of publication channels of scientific results together with the payment of fees for the publication in open access (or, simply, for publishing) is a critical issue in scientific communication. Since it has implications in scientific evaluation and, also, it is the cause of concern in the academic community, it is necessary to develop it. The payment of fees to publishers for open access publication is intended to partially cover the publishing costs and to compensate the publishers for the income they will not perceive for the sales of the contents when publishing in open access. It is assumed that such payment takes place once the article or the book have been approved by evaluators or reading committees for their publication. However, the debate taking place is if that payment is conditioning what is finally published. There are suspicions by part of the academic community, which sees in the payment for publishing an easy publication procedure, which can be omitting or altering the selection filters, which are fundamental guarantors of academic publishing and evaluation processes. Also, the existence of specific funds for publication has generated other types of situations.

One of them, very well documented, is the proliferation of predatory publishers, both in the case of books and journals. These are non professional structures that offer authors the publication of their articles or books in a short time in return for a publishing payment, without the delivery of publishing services. The bibliography on this topic is wide and controversial. Maybe there is a general vision in Anderson (2017Anderson, R. (2017). Cabell’s New Predatory Journal Blacklist: A Review. The Scholarly Kitchen. https://scholarlykitchen.sspnet.org/2017/07/25/cabells-new-predatory-journal-blacklist-review/.) and, of course, in Beall (2017Beall, J. (2017). What I learned from predatory publishers. Biochemia Medica, 27 (2), 273-278. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5493177/.), who coined the term predatory journal, built a worldwide list of these journals that had to be closed because of a problem in courts, also very controversial.

The other is the generation of constant opportunities for publishing that some consolidated publishing houses offer to authors. After the publication of an article in a journal or the presentation of a contribution to a conference, publishers contact the authors directly in order to offer them the publication of an extended version of their work or for their work to be part of a collective book, just as two examples. This produces certain confusion in the researcher, since it is not always clear up to which point these are interesting publication procedures and, at the same time, it implies planning a new publication that, possibly, was not initially scheduled. Also, the researcher can have doubts whether it can imply a certain cost, or on the consideration that the journal or publisher would have in a future evaluation exercise.

4.4. Alternative metrics and open citations

In parallel to the rise in the open access publication, there are developments in the framework of social networks –both, of general scope and academic- and technological innovations in the platforms offering scientific contents that allow the development of metrics different to the traditional ones: the so-called alternative metrics or Altmetrics. Scientific evaluation has been based, in recent years, in impact indicators from Journal Citation Reports and, more recently, in indicators based on Scopus data such as SNIP (Source Normalized Impact per Paper) and Scimago Journal Rank. Such indicators are derived from the scientific literature selected by the developers of the databases and are calculated from references contained in those databases. They are not accessible to those not counting with subscription to the databases. These are selective –non complete- and closed databases and they do not provide a precise idea of the citations received by each research. On the other hand, citations that can be openly obtained through Google Scholar, Microsoft Academic or PubMed do not represent the total number of the citations produced, since large commercial publishers do not offer, openly, the full corpus of references in the articles they publish. These tools present, for the moment, problems of transparency and data quality. All that underpins the claim of the bibliometric academic community regarding the request to publishers making accessible the lists of references associated to the articles that they publish: this is the so-called Initiative for Open Citations (ISSI, 2017ISSI (2017). Open citations: A letter from the scientometric community to scholarly publishers. http://issi-society.org/open-citations-letter/.).

In the field of Humanities, the value of citations has always been questioned because of the existence of schools of thought, the local / linguistic factor of research or the lack of immediacy with which citation takes place. However, there is little doubt regarding the fact that the existence of open sources allows the collection of more complete data on the academic impact. A different issue is how to treat, from the perspective of scientific research, those citation data that come from open sources and that, as pointed out in the Initiative for Open Citations, still present several problems related to the quality of the information. Possibly, the information on citations received by a publication nowadays is informally used in evaluation processes. Nevertheless, the formalization of that use seems still complicated, precisely because of the quality of data.

Something similar happens with other types of metrics: visualizations, downloads, recommendations or assessments by the users of the network. There is a growing number of tools that allow obtaining them (Zuccala et al., 2015Zuccala, A.; Verleysen, F.; Cornacchia, R.; Engels, T. (2015). Altmetrics for the humanities: Comparing Goodreads reader ratings with citations to history books. Aslib Journal of Information Management, 67 (3), 320-336. https://doi.org/10.1108/AJIM-11-2014-0152.) but their meaning, value and the possibility of sharing them among authors, fields, etc. are still uncertain, if not limited (Torres Salinas et al. 2017Torres-Salinas, D.; Robinson-Garcia, N.; Gorraiz, J. (2017). Filling the citation gap: measuring the multidimensional impact of the academic book at institutional level with PlumX. Scientometrics, 113 (3), 1371-1384. https://doi.org/10.1007/s11192-017-2539-z.). As pointed out in The Metric Tide, the options offered by systems such as Twitter, ResearchGate, Academia or Mendeley are overwhelming but ‘evidence on whether and how these may relate to research quality is very limited’. The availability of information and possible indicators is not synonym with their maturity in order to be used in evaluation. The research in the field of Scientometrics requires time and data cannot / should not be hurriedly used without previously having obtained solid conclusions on them (Wilsdon et al., 2015Wilsdon, J.; Allen, L.; Belfiore, E.; Campbell, P.; Curry, S.; Hill, S.; Jones, R.; Kain, R.; Kerridge, S.; Thelwall, M.; Tinkler, J.; Viney, I.; Hill, J.; Wouters, P.; Johnson, B. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. https://doi.org/10.13140/RG.2.1.4929.1363., p. 136). In this sense, it is relevant to mention the work by NISONISO Alternative Assessment Metrics (Altmetrics) Initiative (s.d.). http://www.niso.org/standards-committees/altmetrics. (National Information Standards Organisation) on the normalization of the alternative metrics as well as their application to academic results different from publications such as data sets, visualizations or infographics and software. Also important is the work by one of the centers of reference in the field of studies on science, the Center for Science and Technology Studies (CWTS), Leiden University. One of their lines of research is focused on the determination of the role that alternative metrics can play in Scientometrics. In the study made in 2015 (Costas et al, 2015Costas, R.; Zahedi, Z.; Wouters, P. (2015). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology, 66 (10), 2003-2019. https://doi.org/10.1002/asi.23309.) a still weak correlation between citations and altmetric indicators was observed, which could indicate that the two types of indicators reflect different things. Also in that study, more altmetric indicators were found for Humanities fields than for others. It has been observed how time can affect correlations between citations and indicators taken from the social Web (Thelwall et al., 2013Thelwall, M.; Haustein, S.; Larivière, V.; Sugimoto, C. R. (2013). Do altmetrics work? Twitter and ten other social web services. PloS one, 8 (5), e64841. https://doi.org/10.1371/journal.pone.0064841.).

Finally, it is worth mentioning that although alternative metrics are obtained both for the scientific literature under subscription access and the literature available in open access, the infrastructures created for the enhancement of open access are and will be a relevant source of indicators. Open access does not only imply the existence of collections of documents that can be accessed without barriers, but also that it is linked to the concept of open science and implies the existence of infrastructures and data interconnection, such as in the case of OpenAIRE[18] or as in the case of the developments that are taking place, especially, the Humanities and the Social Sciences such as the project OPERAS (Open access Publication in the European Research Area for Social Science and Humanities) or HIRMEOS, (High Integration of Research Monographs in the European Open Science infrastructure). Doubtlessly, those systems will represent an additional contribution to visibility, accessibility, metrics and the identification of the work dynamics in these fields.

4.5. Responsible metrics

Both traditional and new metrics can present valuable and strategic information for evaluation processes. New developments taking place in the evaluation of scientific publications, including books, book series, national journals, etc., and the alternative metrics are contributing to open up the possibilities of evaluation in the Humanities and the Social Sciences, until recently constrained to a set of strict limits established by international databases. But while the possibilities for evaluation are improving, it is also true that it is necessary to pay special attention to the indicators and sources used for such evaluation.

As pointed out in The Metric Tide report, responsible metrics imply working with robust and transparent data, which allow reflecting the diversity in research and in the trajectories of researchers. Also, it is necessary to reflect on the consequences of the application of the various metrics in evaluation processes and to recognize the limitations that each of them might present (Wilsdon et al., 2015Wilsdon, J.; Allen, L.; Belfiore, E.; Campbell, P.; Curry, S.; Hill, S.; Jones, R.; Kain, R.; Kerridge, S.; Thelwall, M.; Tinkler, J.; Viney, I.; Hill, J.; Wouters, P.; Johnson, B. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. https://doi.org/10.13140/RG.2.1.4929.1363., pp.133-134). This particularly affects alternative metrics, on which the scientometrics community itself warns about what precautions should be in place, in order to avoid making the same mistakes as in the case of traditional metrics (Thelwall, 2014Thelwall, M. (2014). Five recommendations for using alternative metrics in the future UK Research Excellence Framework. http://blogs.lse.ac.uk/impactofsocialsciences/2014/10/23/alternative-metrics-future-uk-research-excellence-framework-thelwall/.).

Another key question in the debate on how to use the metrics is the need to consider the objectives of evaluation, distinguishing the different evaluation processes and the nature of the research carried out. In short, what is being proposed is the use of the different indicators in a responsible way, adequate to the objective to be achieved and with a firm ethical commitment: to evaluate in the best possible way in favor of a better research.

4.6. Societal impact of Humanities and Social Sciences

The societal impact of research is one of the topics in which the academic community is working more intensely in order to gain recognition and value for this type of impact in the evaluation of scientific activity. The Research Excellence Framework 2014Research Excellence Framework (REF) (2014). Main panel D criteria. http://www.ref.ac.uk/2014/media/ref/content/pub/panelcriteriaandworkingmethods/01_12_2D.pdf. in the United Kingdom included the evaluation of such societal impacts, relating them with excellence in research. They have also been part of many evaluation procedures such the case of Humanities in Norway (2017) or in the individual research evaluation in Spain (CNEAI, Field 0. Transference of knowledge and innovation) (España, 2011España. Jefatura del Estado (2011). Ley 14/2011, de 1 de junio, de la Ciencia, la Tecnología y la Innovación. Boletín Oficial del Estado, nº 31. https://www.boe.es/boe/dias/2011/06/02/pdfs/BOE-A-2011-9617.pdf .).

The main idea behind societal impact is that the value of research lies not only on the recognition (citation) of a given scientific work by other researchers but also –or maybe, above all- in the changes and improvements that it can produce in society, culture and economy. Research is intended to serve that end: improve society. There is agreement on the fact that those contributions of science to society should happen and be valued. How to do it is more complicated, although there is clear advancement in that sense (Benneworth et al., 2017Benneworth, P.; Olmos-Peñuela, J.; Muhonen, R. (2017). Towards a common understanding on the societal impact of SSH research. ENRESSH2017. 2nd International Conference on Research Evaluation in the Social Sciences and Humanities, Antwerp, Belgium, pp. 32-35. https://www.uantwerpen.be/images/uantwerpen/container41447/files/A5-BOOK_RESSH2017_170717-interactive.pdf.). The countries more active in this area have created inventories of case studies, from the information that the researchers themselves provide to evaluation agencies. In this way, it is possible to reach certain taxonomy of the types of societal impacts that can be derived from research in Humanities and Social Sciences and to study the contexts in which they are produced (Sivertsen, 2017bSivertsen, G. (2017b). Frameworks for understanding the societal relevance of the Humanities. ENRESSH2017. 2nd International Conference on Research Evaluation in the Social Sciences and Humanities, Antwerp, Belgium, pp. 30-31. https://www.uantwerpen.be/images/uantwerpen/container41447/files/A5-BOOK_RESSH2017_170717-interactive.pdf.).

In the report Evaluation of the Humanities in Norway by the Research Council of Norway (2017Research Council of Norway (2017). Evaluation of the Humanities in Norway. Oslo: The Research Council of Norway. https://www.forskningsradet.no/en/Publications/1220788265688 .), 165 case studies were analyzed, showing a wide and diverse interaction between academia and society. The analysis also allowed observing that certain areas are more active in that type of transference and also that there is a certain confusion between what is public commitment of researchers with society, by disseminating the knowledge they generate, and what is impact, it is to say, the research result that precipitates social change. Another very interesting conclusion from the study is that societal impact generated by the Humanities is of national rather than international reach. This has implications on the relevance of humanistic and social research as well as regarding the relativisation that has to be made of the ‘internationalization’ among the evaluation criteria.

It its worth taking into account the type of interactions between academia and society without the existence of formalization –contract, agreement, etc.– informal interactions seem to be frequent in the Humanities and the Social Sciences (Olmos-Peñuela et al., 2014Olmos-Peñuela, J.; Molas-Gallart, J.; Castro-Martínez, E. (2014). Informal collaborations between social sciences and humanities researchers and non-academic partners. Science and Public Policy, 41 (4), 493-506. https://doi.org/10.1093/scipol/sct075.).

Terämä et al. (2016Terämä, E.; Smallman, M.; Lock, S. J.; Johnson, C.; Austwick, M. Z. (2016). Beyond academia–Interrogating research impact in the research excellence framework. PloS one, 11 (12), e0168533. https://doi.org/10.1371/journal.pone.0168533.) have been studying how researchers and institutions interpret what social impact of research is, from the contributions sent for evaluation in the framework of REF 2014Research Excellence Framework (REF) (2014). Main panel D criteria. http://www.ref.ac.uk/2014/media/ref/content/pub/panelcriteriaandworkingmethods/01_12_2D.pdf.. Zaltz (2017Zaltz Austwick, M. (2017). Does high-impact research come at the expense of quality? An automated analysis of the REF impact landscape. LSE Impact blog. http://blogs.lse.ac.uk/impactofsocialsciences/2017/03/02/does-high-impact-research-come-at-the-expense-of-quality-an-automated-analysis-of-the-ref-impact-landscape/.) groups those contributions in the following categories of ‘impact’: influence in education; public engagement; technologies and environmental solutions; impact in policy; clinical applications.

5. CONCLUSIONS Top

Diversification or attention to multiple dimensions of research is one of the key characteristics / claims of the new evaluation: research in Humanities and Social Sciences shows its results in different ways, not only through one type of publications. Considering the different communication channels as well as other results, different from publications, is one of the challenges to be addressed in evaluation processes. In that sense, transferred results with impact on society are particularly relevant. The academic community claims it, researchers specialized in scientific evaluation work on the methodologies allowing the evaluation of those other aspects and the agencies of evaluation and funding consider that variety of research returns. In this way, the frontiers of academic impact (measuring the influence of some researchers on others) are exceeded. The identification of results that represent true advancement in knowledge is also one of the great challenges for the new evaluation.

Another conclusion is that metrics are a support or an aid in evaluation processes and that they complement but do not replace expert judgment. One of the claims coming from the Humanities and the Social Sciences is that related to the need of evaluating the publications with complete data, that is, taking into account everything that is produced by a researcher and not only what is visible in international databases, traditionally used in evaluation processes. It is also confirmed that the countries that use publications’ indicators as a supporting source for evaluation tend to use different sources and do not rely exclusively on international and commercial databases. In fact, there are many countries that have developed, formally or informally, lists, categorizations, rankings and quality labels, both in the case of journals and publishers, book series or books, which are helpful in evaluation processes.

Indicator sources need to count with a series of characteristics providing them with robustness and acceptance by the scholarly community. They need to be developed by specialists, involve the academic and publishing community, be transparent in their methodologies and be publicly available. Many of these ideas are included in the report The Metric Tide (Wilsdon et al. 2015Wilsdon, J.; Allen, L.; Belfiore, E.; Campbell, P.; Curry, S.; Hill, S.; Jones, R.; Kain, R.; Kerridge, S.; Thelwall, M.; Tinkler, J.; Viney, I.; Hill, J.; Wouters, P.; Johnson, B. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. https://doi.org/10.13140/RG.2.1.4929.1363.) as part of what has been denominated responsible metrics. It is necessary that these ideas are correctly transferred and communicated to those with responsibilities in scientific evaluation.

The mechanistic and obsessive use of indicators can have undesirable consequences such as extreme competitiveness and malpractice (Wilsdon et al. 2015Wilsdon, J.; Allen, L.; Belfiore, E.; Campbell, P.; Curry, S.; Hill, S.; Jones, R.; Kain, R.; Kerridge, S.; Thelwall, M.; Tinkler, J.; Viney, I.; Hill, J.; Wouters, P.; Johnson, B. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. https://doi.org/10.13140/RG.2.1.4929.1363.). Nevertheless, the wrong use of indicators should not be mistaken with their lack of value. The indicators that are well constructed can and should be considered a help to evaluators and –this is not always evident- a support to researchers that, being coherent with their research topic and research career, contribute to its diversification publishing in channels that are plural in terms of contents, languages or research approaches.

It is crucial that in the design of evaluation procedures, the objectives of the evaluation are taken carefully into consideration and, of course, the context (field, research system in each country, etc.). Methodologies should always be well adjusted to the objectives. An individual evaluation process should be different to that applied to a research team involved in a risky research project and those should be different from the evaluation process having as objective the assignation of resources to university departments. Also, differences between research practices among the different disciplines and the type of research (interdisciplinary, theoretical, applied, etc.) need to be considered.

Taking into account those variables, present in many of the reports or manifestos defending a new form of scientific evaluation, would generate trust and transparency, key elements for the acceptance of evaluation systems by the academic community.

6. NOTES Top

7. REFERENCESTop