Recent research shows the existing diversity of evaluation models present in Europe at the moment (Ochsner et al. 2017b). Such diversity is defined by various variables such as centralization or decentralization, the availability of information about scientific research at the institutional, regional or national level, the greater or more limited consideration of research practices in different fields or the inclusion of one or more dimensions of research in those evaluation processes. Certainly, many of them have scholarly publications as main object of evaluation. Not only are they the most tangible and measurable output of research activity, but they can also be considered the tip of the iceberg: where there are publications, there are research, dissemination, innovation or collaboration networks.

Maybe a first distinction in the models can be made between those countries with national information systems (Current Research Information Systems) integrating the whole of the scientific output and those which do not count with them. Countries such as Norway, Finland, Denmark and the Flanders region (Belgium) have built CRIS which allow them to count with the full output of each country/region. Also, the Lattes6 platform in Brazil can be considered an information system at the national level. Counting with full publication data allows making evaluation in context. The starting point of evaluation is the whole scientific output of a country; from that it is possible to identify publication patterns in the different fields and apply assessments and / or weighting procedures to publication channels, mainly journals and scholarly book publishers (Giménez-Toledo et al., 2016). Categorizations or levels assigned to scientific output are developed by expert panels from information generated by research groups and publishers’ associations.

It is relevant to underline, as an starting point, in these systems all the scientific output is taken into account, not considering only a small fraction which is visible in international databases such as Web of Science (Clarivate Analytics) or Scopus (Elsevier). This is a fundamental difference. In other countries, the model is (or has been) the opposite: there are no national databases and researchers send evaluation agencies their curricula for these to be assessed, and then, two situations can occur.

The first of them, quite common until recently, gives recognition –primarily- to scientific output included in the commercial databases Web of Science and Scopus. That scientific output is limited to the selection of sources made by the producers and it is limited, essentially, to scientific journals and books from a very limited set of scholarly publishers. It is relevant to remind here the limitations produced by evaluation relying on those databases. There are innumerable studies showing the biases of those databases and the consequent difficulties for the evaluation of output in Humanities and Social Sciences. One of the most recent ones (Gingras and Khelfaoui, 2017) shows how citations play in favor of countries with a closer relationship to the United States. The geographical distribution of the journals and publishers selected shows how incomplete these sources are for the evaluation of the scientific output of a country. Just as an example: a recent study on the Dominican Republic’ scientific output in international databases reveals that they index only 49% of the output. It is to say, 51% remains invisible if other sources are not taken into account (Riggio, 2017). Consequently, when the decision is made to evaluate using only those information sources, not only is a large part of the research activity being undervalued, but also relevance is being downplayed on the type of research that can be more useful for that region or country, which is usually the research that remains out of the international databases.

The second refers to evaluation processes which are not backed by a CRIS but using a richer approach for the evaluation of research outputs. In fact, in recent years it is possible to note that some national evaluation exercises do assess research output counting with diverse sources of indicators. Apart from WoS or Scopus, other databases such as ERIH Plus (European Reference Index for the Humanities in its most recent version) or other international categorizations of journals; also, databases or national indexes providing information on the scientific journals’ characteristics are taken into account. Such is the case of the lists of publication channels in the BFI / BRI Bibliometric Research Indicator in Denmark7, the Finnish system, categorized by the Julkaisuforumi8, the Norwegian system, developed by the Norwegian Register for Scientific Journals, Series and Publishers9, or the evaluation system of the Flanders Region (Belgium), through the Flemish Academic Bibliography for the Social Sciences and Humanities VABB-SHW10. Even in some cases, such as in Spain, information sources on scholarly books such Scholarly Publishers Indicators (SPI)11 or quality labels for book series (CEA / APQ)12 are mentioned as information sources.

Rankings, categorizations or quality labels for books and scientific journals are tools being used in almost any evaluation process. Somehow, the use of those sources confirms the need for supporting decisions made on scientific output on objective indicators. Probably, the use of the tools is the most controversial part. Indicators on publications should be used always in combination with the criterion of experts. This has always been the golden standard in bibliometrics. Nevertheless, in some occasions evaluation processes are excessively simplified and automatized, making automatic translations of the value of a scientific contribution with regards to the position of a journal or a publisher in a ranking. Indicators of publishers should provide solid and useful information to the expert, but they are not entitled to determine the final result of the evaluation. “Metrics should support, not supplant, expert judgement” is the conclusion of The Metric Tide (Wilsdon et al., 2015), an independent report on metrics in scientific research.

The lack of CRIS prevents from counting with a detailed knowledge on the levels of output by scientific fields, co-authorship indexes o international collaboration, the relevance of the different communication channels or languages for the communication of scientific knowledge. In all, it inhibits counting with context data at the country level on the communication pattern in the different disciplines. However, in some occasions, there are no national information systems but institutional ones. Those systems, repositories or databases specifically designed for tracing of an institution’s scientific output allow counting with precise information for evaluation in a specific university or research institution (De Filippo et al., 2011; Reale et al, 2011). In example, the Spanish National Research Council (CSIC) counts with the information system ConCiencia, while a large proportion of Spanish universities work with the software Universitas XXI for the monitoring of their research activity, among other objectives.

In the evaluation processes at the national level of countries without CRIS, the different committees or expert panels have that context defined, more or less, but without the data susceptible of confirming the publication patterns and their evaluation through time. Besides, through giving priority to scientific output included in international databases, a secondary relevance is given to research closer to the territory, which can have a societal impact –and not only a scholarly one- on the closest community.

In that sense, international publication is favored (understanding ‘international’ in a restrictive sense, since it is what is selected by the database producers) and, therefore, it takes into account topics and methodologies which are interesting in an international and globalized context. Nevertheless, it is necessary to consider too the publications which are outside that framework, not only because these disseminate scientific knowledge in other topics which are relevant for the close communities but also because, in this way, the thematic, linguistic, methodological and ideological diversity present in research is being preserved (Oschner et al., 2017b; Giménez-Toledo, 2016). For example, research questions and their corresponding answers in indexed international journals of almost any discipline are not, in most cases, the primary concern of Latin America and Spain. Vessuri et al. (2013) allude to the internationalization of research in the following terms: ‘If the science of Latin America wants to become international on a solid base, it should not consider erroneously that Northern Atlantic’s science is the totality of the world’s science’ (p.12). The various arguments in that sense point out that if non-indexed publications are not taken into account, a large part of the essential research for national or regional contexts will be lost.

Recognizing the singularities of the scientific production in Humanities and Social Sciences and, at the same time, the interest in the observation of that production in an international framework -not only in the national ones- and from a compared perspective, the creation of a database covering the publications produced in those fields in Europe has been proposed long ago (Martin et al, 2010). Although the proposal was not carried out in practice, the idea is still valid. In fact, it is part of the objectives of the COST action European Network for Research Evaluation in the Social Sciences and the Humanities (ENRESSH, 2015). This network is collaborating with the VIRTA (Puuska et al, 2017) project in a technological infrastructure which would allow the aggregation of scientific output data from different countries in a normalized and interoperable way. The feeding system is decentralized, from each of the countries or institutions. As the project advances, and more countries participate, it will be possible to create a large database of scientific output that would allow not only the development of studies on publication dynamics, collaboration, etc. in different fields but also to count with information for the evaluation of the Humanities and the Social Sciences.

Content evaluation from published research has been and is still a common claim of humanists and social scientists. In some occasions they criticize the evaluation made on the publication channels, mainly journals and publishers, dissociating the content from the continent. In the light of the studies made on this issue (McCulloch, 2017; Giménez-Toledo, 2016), the scholarly community feels the pressure exerted on them by evaluation, the demands for publishing more and through some specific channels and, in relation to this, the deviation from their research topics in order to publish in the journals or publishers better valued in evaluation. As Oschner et al. (2017a) point out, researchers in these areas have strong reservations with regards to quantification.

There are several factors to be considered with regards to such positioning (Giménez-Toledo, 2016, p. 23 and next pages). First, there are few qualitative scientific research evaluation schemes. The last Research Excellence Framework (REF 2014) (Panel D) in the UK was one of them, to the point of explicitly noting in its call that the publication channel or the journals or publishers’ classifications were not going to be considered by Humanities and Social Sciences’ panels (Research Excellence Framework, 2014). Expert panels would be the main procedure. The methodology was applied thanks to a vast investment in the evaluation process, since it is costly in money and time. At the same time, since it is a voluntary evaluation exercise which is performed on the researchers’ output for a six-year period, it is feasible, by contrast with the periodical evaluation processes, usually with an annual periodicity, which are done at the institutional level. The REF 2014 has been, nevertheless a controversial and criticized evaluation exercise.

Qualitative evaluation based only on experts’ criteria is not free from problems, since there are not only schools of thought, ideologies or methodological appreciations that imply positioning oneself against or in favor of a given line of research. Also, as Oschner et al. (2017a) point out: “the way SSH scholars appreciate research output of colleagues is quite different from how STEM researchers do. SSH scholars are much more critical. They criticize even work they value as excellent”. The fact that evaluations were based only on the specialists’ judgment would not guarantee a better acceptance of evaluation processes. That reason, together with costs and evaluation time spans involved in evaluation processes, contributes to the fact that many evaluation models are based on a combination of experts’ criteria (the qualitative side) and indicators (the quantitative side).

Concerning the lack of acceptance of quantitative indicators by humanists and social scientists, it is relevant to establish differences between indicators and, of course, to claim the value and usefulness of some of them. In the light of some of the reluctances that they cause, it might seem as if the indicators of a publication –its prestige, internationality, rigor in the selection of manuscripts- were values completely foreign to the idea, judgment or perception that the academia has on that publication. It might seem that the judgment of experts on a given publication channel (journal, publisher, etc.) cannot be translated, by no means, into a numerical value or a category. Bibliometric indicators are not always mere measures of the output quantity or citations received. Maybe because in some occasions they have been excessively or incorrectly used, or perhaps because numeric measurements are usually foreign to Humanities, there is a reluctance concerning their use. Nevertheless, a numerical indicator or a category might represent different qualities of the research or of the publications. When an expert panel meets in order to determine the quality level of a set of journals and they resume their judgments in a set of categories (see the Finnish, Danish or Norwegian system in Giménez-Toledo et al, 2016) a translation is taking place in practice, which implies facilitating an evaluation process taking into account the specialists’ knowledge. The same happens in a consultation to the scholarly community on the prestige of publishers, as the one involved in the creation of the indicators’ system Scholarly Publishers Indicators. Or when, for the creation of that same system, thousands of bibliographic registers are analyzed in order to determine the specialization profile of a publisher (Giménez-Toledo and Mañana-Rodríguez, 2016). The construction of an indicator implies the formalization of knowledge that, until that moment, has remained ‘informal’ or tacit, thus achieving its usefulness for the evaluation processes. An enlightening article by Ferrara and Bonaccorsi (2016) shows how the theoretical distance between the opinions of experts on journals and the indicators is not such, or is not so much. They compared two evaluation processes of scientific journals, one based on qualitative judgments and the other on categorizations and show, through regression analysis, that the results are in fact quite similar. This implies that it is feasible to make that translation of qualitative criteria into quantitative indicators and that the tools that include them allow to efficiently face the overloaded evaluation processes of scientific activity.

It is also necessary to distinguish between the evaluation systems and their sources with regards to the objectives of each of them. Often, evaluation is mentioned in general terms, but it is not the same carrying out an institutional evaluation process in order to improve transparency and to develop better guidelines in science policy, than to evaluate for the assignment of resources of research, or to evaluate an individual researcher for promotion. Procedures and tools would vary depending on the objectives. Sivertsen (2017a) establishes a clear classification of evaluation systems in countries counting with funding systems based on research results. In that classification it can be observed how the different objectives of the evaluation systems have propitiated, in some cases, the use of expert panels in evaluation and in others the use of indicators, with differences also between the types of indicators being used. Going into such diversity of objectives and procedures should make clear that generalizations, in research evaluation too, are neither positive nor constructive.

One of the key ideas concerning evaluation processes that are claimed nowadays is that the scholarly community has to participate in the design of the process or the indicators. The so-called bottom-up approach has been thought of as a good solution for the improvement of the evaluation processes (Oschner et al, 2017a). The reason is that these procedures take into account the scientific research and communication practices of humanists and social scientists, which implies not only an evaluation more adjusted to reality but also, a priori, a better acceptance of the results by the academic community. It is necessary to underline ‘a priori’ because even the procedures or sources of information that are specifically created thinking in these disciplines and which involve their scholars are often controversial or opposed by researchers. The first stage of ERIH, European Reference Index for the Humanities, is an example of this, maybe particularly notorious because of the European scope of the project. This project became ERIH Plus (2017), a bibliographic information source of Humanities and Social Sciences’ scholarly journals which informs on the accomplishment of certain quality criteria that are demanded to that type of publications: external peer review, existence of an editorial committee, institutional affiliation of authors, etc. Paradoxically, the intervention of specialists in the fields – a common claim- in order to categorize the journals in ERIH in its first stage was so controversial (Journals under threat, 2009) that nowadays it is not taken into account. ERIH Plus is a source of bibliographic information more than a source of information potentially useful for research evaluation.

Not only ERIH has been discussed by the academic community. Journals or publishers’ categorizations, weightings of indicators or the different facets of research activity that are taken into account (in example, the value of scientific dissemination or the management of scientific journals) are a permanent object of discussion in the national evaluation processes. It is a common feature to all countries, independently from the evaluation model in each of them. Sometimes the reaction against the evaluation systems or its resolutions is so overwhelming that it is taken to court. This is a surprising fact since nowadays evaluation is carried out with much more information, better indicators and, often, with the participation of the scientific community. Expert panels are usually part of most evaluation processes. Also, scientific associations communicate evaluation agencies their opinions and position concerning evaluation criteria. In general, the conditions for the evaluation have improved. However, maybe because of reductionist views on science –only what is published in international, indexed journals is valuable- or because of the systematic opposition to evaluation procedures or to quantitative indicators, there is still a lot to be done in order to reach evaluation models with wide acceptance.

In most cases, when proposing the integration of bottom up approaches to evaluation processes, it implies the combined use of indicators and peer review with the participation of the corresponding academic communities. Some examples of those approaches are the ones applied in the countries using the so-called Norwegian model or the one used for the creation of the prestige rankings of publishers Scholarly Publishers Indicators (SPI)13. In the first case, as mentioned before, the starting point is the set of the full output of a country. It is evaluated in context, since it is possible to identify patterns of scholarly work and publication in the different fields and, also, the judgment of specialists is considered. The Norwegian model counts with expert panels that validate the publication channels and establish quality levels for them.

In the case of SPI, the information on the prestige of publishers is obtained from readers and authors of the books published by those publishers (Giménez-Toledo, 2016). It is based on a consultation to the academic community that, in its condition of specialist, is the collective in the best position to evaluate the quality of the publishers’ title list. The consultation allowed the publication of prestige rankings of the publishers that was understood as an orientation in the evaluation processes. That approach is not the only one, since the SPI system also provides information on the specialization of the publishers’ title list, the manuscript selection process, the use of metadata, the grouping of their title list into collections or their presence in other information systems. Both, prestige indicators and the manuscript selection processes are elements explicitly mentioned by Spanish evaluation agencies, Agencia Nacional de Evaluación de la Calidad y Acreditación (ANECA, 2017) and Comisión Nacional Evaluadora de la Actividad Investigadora (Ministerio de Educación, Cultura y Deporte, 2017) in fields of Humanities and Social Sciences.

A key transformation in the way of making science and communicating research results is taking place because of the principles of open science. From the postulation stating that society as a whole should benefit from and have access to the results of scientific research developed with public funding, new academic practices emerge:

All these are topics to which a great deal of attention is being given in research and that have entered the scientific policy agendas, particularly with regards to open publication of results and data. The Recommendation of the European Commission concerning open access and the preservation of scientific information (European Commission, 2012), the future creation of a platform for the publication of open results, also proposed by the European Commission (European Commission, 2017), the different laws of science, including the Spanish one (España, 2011), and the different calls for research projects’ proposals, among which there is the current European Research and Innovation Program, H202014, show that from a political perspective, open access to scientific knowledge is a firm and secure commitment. However, the adoption of open access entails many changes in the academic environment.

If authors are obliged to publish in open access, either by own conviction or because of the imperatives of project calls, they need to decide how to do it. They can opt for the green or the golden road. In that latter case –immediate publication of open access results-, the publication implies, usually, the payment of fees to the consolidated academic publishing houses. Without that payment, the content of the article would remain closed, only accessible to those with subscription to a specific journal, and only after the embargo period had expired would the author be able to deposit the article in a repository (green road).

Usually, those fees (APCs: Article Processing Charges) are paid by authors since their projects count with funds for open access publication. The public calls foresee specific budget items for open access publication. This implies an impulse to open access. Nevertheless, there are many doubts surrounding the sustainability of the system and, in fact, that is one of the main debate topics. On the one hand, it is not possible to fund all the publications produced. On the other, public funds devoted to research end up serving as payment to big publishing companies that not only design specific programs for open access publication in order to facilitate the accomplishment of the mandates on authors, but also are capable of attracting the reception of more original articles (and more income) because their journals are indexed in international databases, are part of the ‘system’ and are, consequently, widely recognized by evaluation agencies of many countries. The academic publishing industry has adopted solutions for making open access to research an opportunity rather than a threat. Studies by Björk and Solomon (Björk and Solomon, 2012; Solomon and Björk, 2012a and 2012b) clearly show the remarkable increase in the number of scientific journals within large publishing houses that allow the publication in open access through the payment of the corresponding fees. Those same publishing houses have also created specific publication programs of monographs and edited volumes in open access. At the same time, small publishing houses and university and institutional presses publishing journals and scholarly books suffer the consequences of the commercial strategies of large publishing houses; when open access is supported, the funding model is different: the cost of that publication is assumed by the subsidy or funding of the university.

It is known that in Social Sciences and, particularly, in the case of the Humanities, books occupy a prominent place in the communication of research results. However, publishing them in open access is costly. The report by Jubb (2007) places the figure for the fees in 6,500 pounds (BPCs: Book Processing Charges) for the publication with Cambridge University Press, 10,000 for the publication with Taylor & Francis and 11,000 in the case of Palgrave. Given that those costs, together with the lower availability of funds in the case of the Humanities and the Social Sciences, represent a serious sustainability problem, in recent years a series of initiatives are taking place in order to provide solutions to that challenge. For example, Knowledge Unlatched15 proposes the costs associated to the publication of a monograph in open access, selected, edited and published by a consolidated publishing house to be assumed by university libraries wishing to participate in this co-funding system. More than 340 titles have been published using this model. Also, there are new publishing houses such as the Academic UCL Press that publish directly in open access and is funded by the University (University College London), in what constitutes a firm commitment, since scientific knowledge is therefore publicly available.

Regarding the evaluation of Humanities and Social Sciences, all the issues related to open access are of great interest. As already mentioned, evaluation processes take into account, among other things, publication channels. This implies that consolidated journals and publishing houses, which also allow the publication in open access once the fees have been paid, would be able to attract more and better original works. There remains, nevertheless, the economic sustainability problem, but also the problem of diversity. This can be observed, for example, in the case of university presses from different countries that publish directly in open access without the requirement of fees, or in the case of relevant publishers at the national level, both commercial and university presses, which have not adopted open access in the case of monographs because their business model consists, mainly, on selling books, because they do not participate in any publications co-funding system and the investment in publishing would be as sunk costs or, simply, because there are suspicions concerning intellectual property. Data help showing the dimensions of open access publication of books. According to DOAB (Directory of Open Access Books), an open source with worldwide information, 240 publishers are working with open access monographs. Although in that number are some of the most relevant publishers worldwide, with a large publishing output, these are a limited percentage with regards to existing scholarly publishers. Only in Spain, for example, there are more than 200 publishers with a purely scholarly profile (Giménez-Toledo, 2017); in France, there are about 75 institutional publishers (Henny, 2015); in Colombia, more than 60 university presses16; and in Mexico, there are 42 publishers affiliated to Altexto17 (network of university presses). As pointed out in the report A landscape study on open access and monographs (Ferwerda et al. 2017), international markets, the habits of readers or the funding possibilities in the different countries contribute shaping the publishing models and the predisposition to participate in open access monographs’ publishing projects.

The diversity in research and publishing within the Humanities and the Social Sciences is and will be a crucial factor in the evaluation of these fields. Since they generate a less monolithic knowledge than the positive sciences, the patterns and publication channels are more diverse, and this has implications in scientific evaluation: it requires more complete sources that provide information on the wide range of journals, publishers and other communication channels. These sources should combine the diversity and plurality in research topics, languages or methodologies with the quality and/or rigor in the selection of the texts that are published. Precisely because publication channels in these fields are far more numerous and because it is necessary to distinguish the more consistent research and the more selective channels, the development of objective indicators providing additional information is fundamental.

The multiplication of publication channels of scientific results together with the payment of fees for the publication in open access (or, simply, for publishing) is a critical issue in scientific communication. Since it has implications in scientific evaluation and, also, it is the cause of concern in the academic community, it is necessary to develop it. The payment of fees to publishers for open access publication is intended to partially cover the publishing costs and to compensate the publishers for the income they will not perceive for the sales of the contents when publishing in open access. It is assumed that such payment takes place once the article or the book have been approved by evaluators or reading committees for their publication. However, the debate taking place is if that payment is conditioning what is finally published. There are suspicions by part of the academic community, which sees in the payment for publishing an easy publication procedure, which can be omitting or altering the selection filters, which are fundamental guarantors of academic publishing and evaluation processes. Also, the existence of specific funds for publication has generated other types of situations.

One of them, very well documented, is the proliferation of predatory publishers, both in the case of books and journals. These are non professional structures that offer authors the publication of their articles or books in a short time in return for a publishing payment, without the delivery of publishing services. The bibliography on this topic is wide and controversial. Maybe there is a general vision in Anderson (2017) and, of course, in Beall (2017), who coined the term predatory journal, built a worldwide list of these journals that had to be closed because of a problem in courts, also very controversial.

The other is the generation of constant opportunities for publishing that some consolidated publishing houses offer to authors. After the publication of an article in a journal or the presentation of a contribution to a conference, publishers contact the authors directly in order to offer them the publication of an extended version of their work or for their work to be part of a collective book, just as two examples. This produces certain confusion in the researcher, since it is not always clear up to which point these are interesting publication procedures and, at the same time, it implies planning a new publication that, possibly, was not initially scheduled. Also, the researcher can have doubts whether it can imply a certain cost, or on the consideration that the journal or publisher would have in a future evaluation exercise.

In parallel to the rise in the open access publication, there are developments in the framework of social networks –both, of general scope and academic- and technological innovations in the platforms offering scientific contents that allow the development of metrics different to the traditional ones: the so-called alternative metrics or Altmetrics. Scientific evaluation has been based, in recent years, in impact indicators from Journal Citation Reports and, more recently, in indicators based on Scopus data such as SNIP (Source Normalized Impact per Paper) and Scimago Journal Rank. Such indicators are derived from the scientific literature selected by the developers of the databases and are calculated from references contained in those databases. They are not accessible to those not counting with subscription to the databases. These are selective –non complete- and closed databases and they do not provide a precise idea of the citations received by each research. On the other hand, citations that can be openly obtained through Google Scholar, Microsoft Academic or PubMed do not represent the total number of the citations produced, since large commercial publishers do not offer, openly, the full corpus of references in the articles they publish. These tools present, for the moment, problems of transparency and data quality. All that underpins the claim of the bibliometric academic community regarding the request to publishers making accessible the lists of references associated to the articles that they publish: this is the so-called Initiative for Open Citations (ISSI, 2017).

In the field of Humanities, the value of citations has always been questioned because of the existence of schools of thought, the local / linguistic factor of research or the lack of immediacy with which citation takes place. However, there is little doubt regarding the fact that the existence of open sources allows the collection of more complete data on the academic impact. A different issue is how to treat, from the perspective of scientific research, those citation data that come from open sources and that, as pointed out in the Initiative for Open Citations, still present several problems related to the quality of the information. Possibly, the information on citations received by a publication nowadays is informally used in evaluation processes. Nevertheless, the formalization of that use seems still complicated, precisely because of the quality of data.

Something similar happens with other types of metrics: visualizations, downloads, recommendations or assessments by the users of the network. There is a growing number of tools that allow obtaining them (Zuccala et al., 2015) but their meaning, value and the possibility of sharing them among authors, fields, etc. are still uncertain, if not limited (Torres Salinas et al. 2017). As pointed out in The Metric Tide, the options offered by systems such as Twitter, ResearchGate, Academia or Mendeley are overwhelming but ‘evidence on whether and how these may relate to research quality is very limited’. The availability of information and possible indicators is not synonym with their maturity in order to be used in evaluation. The research in the field of Scientometrics requires time and data cannot / should not be hurriedly used without previously having obtained solid conclusions on them (Wilsdon et al., 2015, p. 136). In this sense, it is relevant to mention the work by NISO (National Information Standards Organisation) on the normalization of the alternative metrics as well as their application to academic results different from publications such as data sets, visualizations or infographics and software. Also important is the work by one of the centers of reference in the field of studies on science, the Center for Science and Technology Studies (CWTS), Leiden University. One of their lines of research is focused on the determination of the role that alternative metrics can play in Scientometrics. In the study made in 2015 (Costas et al, 2015) a still weak correlation between citations and altmetric indicators was observed, which could indicate that the two types of indicators reflect different things. Also in that study, more altmetric indicators were found for Humanities fields than for others. It has been observed how time can affect correlations between citations and indicators taken from the social Web (Thelwall et al., 2013).

Finally, it is worth mentioning that although alternative metrics are obtained both for the scientific literature under subscription access and the literature available in open access, the infrastructures created for the enhancement of open access are and will be a relevant source of indicators. Open access does not only imply the existence of collections of documents that can be accessed without barriers, but also that it is linked to the concept of open science and implies the existence of infrastructures and data interconnection, such as in the case of OpenAIRE18 or as in the case of the developments that are taking place, especially, the Humanities and the Social Sciences such as the project OPERAS (Open access Publication in the European Research Area for Social Science and Humanities) or HIRMEOS, (High Integration of Research Monographs in the European Open Science infrastructure). Doubtlessly, those systems will represent an additional contribution to visibility, accessibility, metrics and the identification of the work dynamics in these fields.

Both traditional and new metrics can present valuable and strategic information for evaluation processes. New developments taking place in the evaluation of scientific publications, including books, book series, national journals, etc., and the alternative metrics are contributing to open up the possibilities of evaluation in the Humanities and the Social Sciences, until recently constrained to a set of strict limits established by international databases. But while the possibilities for evaluation are improving, it is also true that it is necessary to pay special attention to the indicators and sources used for such evaluation.

As pointed out in The Metric Tide report, responsible metrics imply working with robust and transparent data, which allow reflecting the diversity in research and in the trajectories of researchers. Also, it is necessary to reflect on the consequences of the application of the various metrics in evaluation processes and to recognize the limitations that each of them might present (Wildson et al., 2015, pp.133-134). This particularly affects alternative metrics, on which the scientometrics community itself warns about what precautions should be in place, in order to avoid making the same mistakes as in the case of traditional metrics (Thelwall, 2014).

Another key question in the debate on how to use the metrics is the need to consider the objectives of evaluation, distinguishing the different evaluation processes and the nature of the research carried out. In short, what is being proposed is the use of the different indicators in a responsible way, adequate to the objective to be achieved and with a firm ethical commitment: to evaluate in the best possible way in favor of a better research.

The societal impact of research is one of the topics in which the academic community is working more intensely in order to gain recognition and value for this type of impact in the evaluation of scientific activity. The Research Excellence Framework 2014 in the United Kingdom included the evaluation of such societal impacts, relating them with excellence in research. They have also been part of many evaluation procedures such the case of Humanities in Norway (2017) or in the individual research evaluation in Spain (CNEAI, Field 0. Transference of knowledge and innovation) (España, 2011).

The main idea behind societal impact is that the value of research lies not only on the recognition (citation) of a given scientific work by other researchers but also –or maybe, above all- in the changes and improvements that it can produce in society, culture and economy. Research is intended to serve that end: improve society. There is agreement on the fact that those contributions of science to society should happen and be valued. How to do it is more complicated, although there is clear advancement in that sense (Benneworth et al., 2017). The countries more active in this area have created inventories of case studies, from the information that the researchers themselves provide to evaluation agencies. In this way, it is possible to reach certain taxonomy of the types of societal impacts that can be derived from research in Humanities and Social Sciences and to study the contexts in which they are produced (Sivertsen, 2017b).

In the report Evaluation of the Humanities in Norway by the Research Council of Norway (2017), 165 case studies were analyzed, showing a wide and diverse interaction between academia and society. The analysis also allowed observing that certain areas are more active in that type of transference and also that there is a certain confusion between what is public commitment of researchers with society, by disseminating the knowledge they generate, and what is impact, it is to say, the research result that precipitates social change. Another very interesting conclusion from the study is that societal impact generated by the Humanities is of national rather than international reach. This has implications on the relevance of humanistic and social research as well as regarding the relativisation that has to be made of the ‘internationalization’ among the evaluation criteria.

It its worth taking into account the type of interactions between academia and society without the existence of formalization –contract, agreement, etc.– informal interactions seem to be frequent in the Humanities and the Social Sciences (Olmos-Peñuela et al., 2014).

Terämä et al. (2016) have been studying how researchers and institutions interpret what social impact of research is, from the contributions sent for evaluation in the framework of REF 2014. Zaltz (2017) groups those contributions in the following categories of ‘impact’: influence in education; public engagement; technologies and environmental solutions; impact in policy; clinical applications.