Google Scholar (GS) is an academic search engine created by Google Inc. (now Alphabet) on 18 November 2004, and its main purpose is to provide “a simple way to broadly search for scholarly literature” and to help users to “find relevant work across the world of scholarly research”.1

The way it functions is similar to the general Google search engine in that it is a system based on providing the best possible results to user queries entered into a stripped-down search box (Ortega, 2014). In the case of GS, it returns results for millions of academic documents (abstracts, articles, theses, books, book chapters, conference papers, technical reports or their drafts, pre-prints, post-prints, patents and court opinions) that the Google Scholar crawlers automatically locate in the academic web space: from academic publishers, universities, scientific and professional societies, to any website containing academic material (Orduna-Malea et al., 2016).

As with Google, the results retrieved for a particular query are ranked by an algorithm that takes into account a large number of variables (where it was published, who it was written by, how often and how recently it has been cited in other scholarly literature, etc.), although the exact components of this algorithm and the weight of each variable is unknown, for industrial property reasons. However, several empirical studies have demonstrated that the number of citations received by a document is one of the key ranking factors (Beel and Gipp, 2009; Martín-Martín et al., 2017). Another essential feature of Google Scholar is that the entire process is automated, without any human intervention, from the location of documents (crawling) to the bibliographic description (metadata parsing) and the extraction of the bibliographic references (reference parsing) that are used to compute the number of citations received by each retrieved document from all other documents.

Google Scholar was not the first tool of this type; other pioneering systems had already appeared on the scene (Citeseer, the first version of which dates from 1997, is considered the first academic search engine). However, the fact that it was developed under the umbrella of a company like Google, and used part of its technology, led to immediate acceptance by a significant proportion of the academic publishing world and by some professionals and researchers, a fact that was widely criticised by Jacsó (2006a), who openly mocked this new state of affairs (“As Google wandered into the territory by launching Google Scholar (GS) at the end of 2004, the topic is expected to appear in the ultra-light morning television chat shows run by ultra-light TV personalities who are meant to light up our mornings”).

Given the characteristics of Google Scholar, it can and should be studied from two complementary but different angles (not only its characteristics but also its effects and consequences). Firstly, GS may be evaluated as a discovery tool (Breeding, 2015), that is, a search engine the purpose of which is to provide the best results to each query and a pleasant user experience based on usability, ease of use and, above all, speed (Bosman et al., 2006). Secondly, Google Scholar may be analysed as a tool that can be used to evaluate scholarly activity. This use, which came about due to it providing citation figures for each document indexed by the system, has led to the increasing use of Google Scholar by users (teachers, researchers, students) and professionals (companies, assessment bodies) as a bibliometric tool for various evaluation processes (authors, journals, universities), although it was not designed with this purpose in mind and lacked the required basic functions (Torres-Salinas et al., 2009). It is precisely this aspect (Google Scholar as a valid tool for carrying out bibliometric studies) that the objectives of this bibliographic review will be based on.

The debate about the advantages and disadvantages of using Google Scholar began immediately after it first appeared (November 2004), giving rise to good and bad criticism in equal measure, as Giles (2005) pointed out in his column in Nature. The first analyses of Google Scholar came from technology blogs and websites, such as Sullivan’s (2004) more neutral and informative piece for Search Engine Watch (https://searchenginewatch.com), or Kennedy and Price’s (2004) more sensationalist piece for the now-defunct Resource Shelf, affirming that “as you’ve read here many times, Google is brilliant (that is, ingenious at marketing and trying new things), and this is yet another example of their savvy”. These messages propagated fast on the internet.

In spite of the general enthusiasm, critical voices soon made themselves heard, one of whom was Péter Jacsó (2004), who tested the search engine between 18 and 27 November 2004, publishing his findings informally on a blog.2 In his study, Professor Jacsó, a specialist on database evaluation with extensive experience, conducted an analysis of the general coverage of various publishers on Google Scholar using the “site” command, and identified a number of important limitations, leading him to conclude that “Google Scholar needs much refinement in collecting, filtering, processing and presenting this valuable data” (Jacsó, 2004). The issues identified by Jacsó included unfriendly search syntax, little or no information about the features of the search engine, and inconsistent results. He found specific errors, such as the way in which it displayed results in which there were changes in the word order of the title, or generated completely erroneous bibliographic descriptions (the book, Computers and Intractability, by Garey and Johnson, detected errors and inconsistencies in the title, subtitle, author names, publisher’s names, locations and years). He also noted a wide range of additional errors like inflated hit counts, inflated citedness, full-text links pointing to erroneous documents and unmerged document versions.

At that precise moment, and in the wake of Jacso’s criticism, a wave of criticism was directed against the general drawbacks of Google Scholar (Price, 2004; Goodman, 2004; Abram, 2005; Gardner and Eng, 2005; Notess, 2005; Ojala, 2005; Vine, 2005; Wleklinski, 2005; Adlington and Benda, 2006; White, 2006), alongside more neutral articles, such as the study published by Noruzi (2005), that, while acknowledging its many drawbacks, also pointed to its potential benefits and possible improvements. At the same time, other articles adopted a markedly neutral attitude towards GS. These included the columns by Butler (2004) in Nature and Leslie (2004) in Science, brief news features that did not discuss or even mention critical aspects, perhaps due in part to the fact that both the Nature Publishing Group and the American Association for the Advancement of Science (AAAS), the publishers of Nature and Science, respectively, had reached agreements to provide access to the full text of their publications to Google Scholar crawlers.

On the other hand, the paper published by Belew (2005) was a significant departure in the debate about the value of Google Scholar. This author analysed a corpus of 203 publications concluding that, surprisingly, there was a high correlation between the citations received by these documents according to Google Scholar and to ISI (the author did not indicate what exact database he used or the discipline to which the documents belonged, simply that six authors from the same interdisciplinary department had been chosen at random; in any case, the use of WoS may be surmised in the area of computer science). Similarly, Pauly and Stergiou (2005) conducted a citation analysis on a corpus of 114 articles from a wide range of disciplines (mathematics, chemistry, physics, computing sciences, molecular biology, ecology, fisheries, oceanography, geosciences, economics, and psychology), and also observed a high correlation (R² = 0.994 for articles published from 2000 to 2004), which led them to affirm that “GS can substitute for ISI”, and that “GS may gradually outperform ISI given its potentially broader base of citing articles”. Finally, that same year, the seminal article by Bauer and Bakkalbasi (2005) appeared, an analysis published in D-Lib in which they compared “the citation counts provided by WoS, Scopus, and Google Scholar for articles from the Journal of the American Society for Information Science and Technology (JASIST) published in 1985 and in 2000”. This study concluded that for articles published in 2000, Google Scholar provided statistically significant higher citation counts than either Web of Science or Scopus, and was significant because the authors brought to light the importance that citation analysis had acquired, not only for crawling academic publications or measuring their impact, but also for justifying tenure and funding decisions, underlining the future role that GS could play in this complex matter. Indeed, in the light of Bauer and Bakkalbasi’s article, The Scientist devoted an article to the future of citation analysis and the role that the web in general and GS in particular could play in bibliometric analysis (Perkel, 2005).

Jacsó’s response to these articles was not long in coming; he lambasted them in a column published in Online Information Review (Jacsó, 2006a). First, he declared his utter disagreement with Butler, claiming that he did not seem to have understood his illustrative examples of Google Scholar’s errors, “even if my examples were as much tailor-made for Nature as bespoke suits by Savile Row tailors for the ultra rich”. Second, he warned readers not to limit their reading to Belew’s work. Third, with respect to Pauly and Stergiou, he openly criticised their claim that GS can replace ISI, particularly since their claim was arrived at by “handpicking” only a few articles, without filtering or even cleaning them up, and they contained numerous errors in the form of inflated and phantom citation counts. Two years later, Harzing and Van der Wal (2008) criticised Jacsó for his criticisms of Pauly and Stergiou in a seminal article published in the same journal in which these authors published their earlier article (Ethics in Science and Environmental Politics). They accused Jacsó of also “handpicking” examples of errors, with few and unrepresentative samples, and while they did acknowledge the errors pointed out by Jacsó, these errors were basically inconsistencies in the results for specific queries.

Lastly, Jacsó (2006a) acknowledged the validity of Bauer and Bakkalbasi’s findings, although he recommended that readers take a critical look at the volume of citations not only in the 2000 sample (where GS was superior to WoS), but also in the 1985 sample (where WoS outperformed GS), data that seemed to have been overlooked by the academic community, which was more interested in highlighting only the positive aspects of GS and hiding or minimising its limitations, according to Jacsó.

From that moment, and in the same column in Online Information Review (called Savvy Searching), Péter Jacsó published a series of articles from 2006 to 2012 aimed at identifying, describing, categorising and denouncing the many errors and limitations of Google Scholar (listed in Appendix I, along with various data related to the errors identified and samples used in each study). Much of his research was also published on his personal website (www.jacso.info), as a way of archiving the evidence.

In spite of the strong and harsh criticism that he then fired off from his platform on Online Information Review (some of his most vehement remarks are listed in Table I), and which will be described in detail in the following sections, Jacsó was always rigorous, admitting that Google Scholar is an excellent tool for locating documents that might not be accessible through traditional databases, as well as for accessing full texts (i.e. as a discovery tool). However, “using it for bibliometric or scientometric purposes, such as for determining the h-index of a person or a journal, is another question” (Jacsó, 2008c). This fact led him to criticise colleagues that used Google Scholar for said purposes even if they did admit the limitations of the database. For example, Bar-Ilan (2008), in her study of highly-cited Israeli authors, admitted that “the sources and the validity of the citations in GS were not examined in this study”. In the light of this observation, Jacsó (2008b) raised his dissenting voice, although he did qualify his position with an understanding that it is sometimes not only tedious but impossible to verify the origin and validity of the citations due to the system’s significant limitations, laconically concluding, “I cannot blame her and others who accept the citation counts as reported by GS”.

The debate came to a head in 2012 when a controversial article published by Jerome K. Vanclay (2012) in the journal Scientometrics strongly criticised the Impact Factor and advocated the use of alternative sources for the evaluation of journals, including Google Scholar. The controversy was heightened all the more by the tone employed by Vanclay. Therefore, Tibor Braun (the founder and editor-in-chief of Scientometrics at the time) invited Jacsó to reply (Jacsó, 2012b). Jacso’s criticisms were extremely strong (“utterly demagogue rhetoric, featuring false accusations, misleading statements, claims and comparisons, delusional ideas, arrogance and ignorance in the Vanclay-set”), so much so that he even questioned the review and publication process (“part of a mock-up scenario to test how poorly researched, prejudiced, biased, duplicate papers using ‘flawed methodology’, ignorant arguments, erroneous calculations, loaded rhetoric, and misleading examples can get through the current quality filters of editorial preview and peer reviews”).

The ideas of Vanclay (2012) were equally criticised by Butler (2011) and by Bensman (2012), who highlighted Vanclay’s lack of knowledge about the workings and purposes of the WoS and JCR databases, and his excessive idealism in the “promise of a far better assessment of research/publication performance through the h-index based on GS”.

Jacsó (2012b) once again reiterated that his criticism of Google Scholar was not directed towards its undoubted advantages for thematic searches, but towards its serious limitations, which make it inappropriate for bibliometric analysis (“extremely lenient citation matching algorithm”), an aspect with which Aguillo (2012) also concurred. In particular, Jacsó argued that the adulation shown by the bibliometric community towards Google Scholar is due in part to the fact that it retrieves a greater number of publications and citations and, consequently, a higher h-index than many researchers would deserve. This may have a perverse effect on the evaluation of the quantity and quality of publications in “decisions related to tenure, promotion and grant applications of individual researchers and research groups, as well as in journal subscriptions and cancellations” (Jacsó, 2012c).

Between 2004 and 2008, criticism of Google Scholar was largely sustained by Jacsó’s articles. However, other authors also expressed their reservations about this search engine, particularly because of the lack of improvements and updates, as Gregg Notess in 2008 noted in the forum Search Engine Showdown.3 At around the same time, a report by the consulting firm comScore, published by the prestigious technology blog Techcrunch,4 reported a fall in the number of unique visitors to Google Scholar during the November 2006 to November 2007 period. This news was picked up on by Jacsó (2008b), although there was no mention of the fact that the Google Scholar team seemed to have declared unofficially that these numbers were not correct (a fact that was mentioned by Notess, 2008). In any case, it seems that there was some decline in the initial euphoria amongst various experts about the potential of Google Scholar. A notable example was Dean Giustini, who had started a blog dedicated to Google Scholar5, and who admitted that “Scholar is not as useful as promised” (cited by Jacsó, 2008b), in reference to the inability of Google Scholar to resolve the limitations that had existed since its launch in 2004. Giustini went on to state that “unless it changes its course, GS will go the way of the dodo bird eventually”. Google Scholar did indeed change.

The evolution of Google Scholar was slow, especially during its first years of existence. This may have been due to the fact that the team at the beginning was made up of only two people (Orduna-Malea et al., 2016). In fact, some of the limitations or criticisms that it received in its early days, such as coverage and speed of indexation (Jacsó, 2005a), were later transformed into strengths (Moed et al., 2016; Thelwall and Kousha, 2017, in press).

In 2008, some of the Google Scholar errors that had led to erroneous results and citations, or large-scale duplication thereof, were corrected, “which is the appropriate reaction to the criticism” (Jacsó, 2008c). A number of other errors could also no longer be reproduced, although many others of a similar magnitude still remained after an apparent cleaning-up of the data. This seemed to indicate that when bad practice was exposed in the press, Google Scholar fixed it so that users could no longer find the exact examples that were reported; they therefore tended to think that the issues had been resolved, although this was not entirely true (Jacsó, 2010). Not only did they persist, but all Google Scholar-based evaluations that had been conducted previously would have irreparably harmed both the individuals and journals that were evaluated.

Jacsó (2010) also complained of the lack of gratitude shown by the Google Scholar team for his and other authors’ contributions to correcting the errors, something that had in fact occurred in the case of the Google Books team, which publicly thanked Nunberg (2009) for contributing to the improvement of that tool with his criticism. Another significant complaint was related to Google Scholar’s tendency to blame its errors on publisher metadata rather than its own parser, similar to the Google Books team’s excuses for errors as reported by Nunberg, as described by Oder (2009), who reproduced the letter from Google in response to the query about the errors detected: “Without good metadata, effective search is impossible”.

However, Google Scholar continued to evolve and to grow until, on its fifth anniversary (2009) it eliminated the “beta” tag that it had retained since its launch (Jacsó, 2010), and many of its systematic errors were fixed (corrected or deleted). Subsequently, Jacsó (2011) reported that the Google Scholar parser had improved, such that tests carried out in mid-November 2010 did not detect some of the previous errors, and many others were reduced significantly, although he did continue to warn that it was not yet reliable enough to be used to calculate bibliometric indicators in the evaluation of research activity. Finally, Jacsó (2012a) recognised that the volume of errors was insignificant when compared to the errors identified at the beginning, although the affected authors would not be of the same opinion. A few years later, in Jacsó’s prologue to La revolución Google Scholar: la caja de Pandora académica (Orduna-Malea et al., 2016), he contended that the reduction in the number of errors, even when positive, was manifestly insufficient since errors persist due to functional issues with the system that have not been resolved.

Indeed, despite the fact that 2011/2012 was a milestone in the history of Google Scholar with the emergence of the related services Google Scholar Citations (aimed at authors) and Google Scholar Metrics (aimed at journals), and definitive growth in its coverage and speed of indexation, many of the errors reported during the 2004 to 2012 period still persist today.

Given the growing use of Google Scholar not only as a gateway to searching for academic literature, but as a bibliometric tool, the identification, classification and quantification of its errors and limitations when calculating bibliometric indicators is of paramount importance.

However, scholarly literature dedicated to this matter has not been systematic. With the exception of Jacsó, few authors have directly sought to detect, describe or gauge the influence of errors in Google Scholar. Occasionally, these limitations have been given passing mention in certain publications, but they have received scant attention in the way of description or explanation or have been quite simply overlooked.

Moreover, in many cases these limitations have been mistaken for errors, when they are in fact related but different aspects. The limitations of Google Scholar are related to certain services or features that prevent it being used as a bibliometric analysis tool. These limitations include not being able to sort the results by the number of citations or the year of publication, the absence of an API (Application Programming Interface), the maximum of 1,000 results per search or limited capabilities for exporting search results, to give only a few illustrative examples. The objective and purpose of Google Scholar is not bibliometric analysis but searching for scholarly literature. Therefore, if such analysis is tedious, we should mark it down as a mere limitation but not as an error.

Conversely, an error arises in relation to features that Google Scholar should provide or execute correctly if is to fulfil the goals and tasks that it officially declares itself to offer. For example, the system claims to report the number of citations received by a publication from the other publications indexed on Google Scholar. Therefore, if this number is incorrect, we have located an error. Since these functional errors directly affect the calculation of bibliometric indicators, knowing what types of errors exist, and to what extent, are important challenges in present-day bibliometrics.

This study is therefore a first step along this line of research. Its main objective is to carry out an exhaustive bibliographic review of what has been said and done about errors in Google Scholar, to then categorise the findings of the studies that we have included in our review.

As mentioned previously, the literature that has dealt with errors in Google Scholar was divided into two bibliographic corpuses. The first (corpus A) comprising the work of Jacsó (16 publications, Appendix I), and the second (corpus B) comprising other publications that have addressed, directly or indirectly, the issue of errors in this database (33 publications, Appendix II), forming in total a corpus of 49 publications.

Of the total number of publications, 40% (20) are concentrated in the period 2005 to 2008, corresponding to the launching of the search engine and the bulk of the articles published by Jacsó, who after then authored an annual review for his column in Online Information Review (2009-2011). 2012 is an exception (four works by Jacsó), coinciding both with the update of the search engine and the birth of Google Scholar Citations and Google Scholar Metrics. From then on, Péter Jacsó ceased his fertile output dedicated to Google Scholar. Corpus B, for its part, developed strongly during the first years, although no remarkable pattern is observed. One possible reason for this is that a significant proportion of these publications did not focus on the errors of Google Scholar, which nevertheless appeared during their development; the errors were then only reviewed in passing (in varying levels of detail). In any case, the number of publications grew in 2016 (five in total).

With regard to thematic coverage, 53% of all the publications (corpus A and B) focus on specific disciplines while the remaining 47% are multidisciplinary studies. These data are influenced by Jacsó’s work, as 12 of his 16 studies (75%) cannot be ascribed to any disciplinary area, since they are based on the testing of different search options through general queries. As far as geographic coverage is concerned, 76% (37) of the publications are international in scope, while only 24% (12) focus on specific countries. Again, Jacsó’s work influences this distribution since all his articles have an international approach (or rather, they have no geographical restrictions). Finally, most of the publications have analysed authors (41% of the total), followed by journals (25%) and documents (17%). Figure 1 gives a summary of the descriptive data of the analysed bibliographic corpus.

Following the scheme proposed in Table II, contributions were classified into those that identify errors related to coverage, parsing, matching and searching.

All of the studies reviewed above operate in the Google Scholar environment. However, there are platforms, both external and linked to this service, designed for working with more filtered and structured data, which may in some cases help to fix some of the errors seen in the previous sections, although they may similarly introduce new errors.

The last body of publications on Google Scholar errors has tried to categorise and classify existing errors. However, it should be pointed out that these classifications are not only incomplete (not reflecting all types of errors), but were carried out in a way that complemented or supplemented the original work, the main objectives of which were not to create or construct a taxonomy of errors. For example, the most detailed classification (although it mainly focuses on parsing aspects) is found in the work of Adriaanse and Rensleigh (2013), whose analysis was based on a sample of only 14 South African environment journals.

In any case, and given their interest, Table VI is a compilation of the main types of errors published to date, the article in which they appeared, and their main items.