SISA, KEA and MAUI are three automatic information processing tools specifically for automatic document indexing. The reasons that led to KEA and MAUI being selected include the following: all three have their origins in academic settings; they are free, open-source software tools; they all have in common the option of performing automatic indexing by assignment, which means terms being assigned from a given controlled vocabulary; and, finally, KEA and MAUI use an automatic indexing model based on a machine learning algorithm and statistical features of terms.

As has been pointed out above, the evaluation-comparison of these three indexing tools will make it possible to examine in depth two different approaches to the automation of indexing, and therefore to find out which of them is closer to human indexing. One approach is based on the position that terms occupy in the different parts of documents (keeping in mind here that the ISO standard for document indexing itself indicates to which parts of documents indexers should pay attention when extracting and assigning terms) while the other is a machine learning approach (based on a model which has been trained through a small collection of documents and the golden indexing of those documents to predict the indexing terms for a new set of documents). The three automatic indexing systems are presented below.

A document collection was created, made up of 230 scientific articles published in the Revista Española de DocumentaciónCientífica of the Consejo Superior de Investigaciones Científicas (CSIC- the Spanish National Research Council). This long-running journal began to publish its articles in PDF, HTML and XML formats in 2013, using an XML format with DTD NLM evolved in the current JATS standard. The criteria for selecting the 230 documents that made up the document collection were simple: articles published in the “Studies” section from 2013 to 2020 that were written in Spanish. To complete the collection, the “Note and experiences” section, which publishes scientific articles of similar relevance and formal structure, was also included.

The training set for KEA and MAUI software comprised 30 randomly selected articles, since SISA does not require an initial machine learning phase for the system. The remaining 200 articles constituted the test set that was applied to evaluate the automatic indexing performed by SISA, KEA and MAUI and later, to carry out the comparative studies. SISA used the 200 articles in XML format while KEA and MAUI worked with the extracted text in TXT format.

A controlled vocabulary composed of 10,981 terms was used, of which 8,214 were preferred terms and 2,767 non-preferred terms. The controlled vocabulary was provided by those responsible for the InDICEs database of the Consejo Superior de Investigaciones Científicas. In this database, the controlled vocabulary is utilized for human indexing of scientific journals published in Spain, as well as conference proceedings on Library and Information Science. Therefore, the 230 articles that make up the test collection were indexed using this controlled vocabulary from 2013 to 2020.

As golden indexing, we used the indexing assigned by the CSIC’s indexers to each of the 200 documents that made up the test set. For this purpose, a query was made in the InDICEs database to retrieve all the documents published in the Revista Española de Documentación Científica from 2013 to 2020. Subsequently, the indexing assigned to each of the 200 documents was compiled and was used for the evaluation processes. It was also ascertained whether all the indexing terms assigned to the 200 documents in the test set were present in the controlled vocabulary. Various terms were identified which had been used as indexing terms but did not appear in the controlled vocabulary, and we proceeded to include them.

The metrics used to compare indexing were of two types: on the one hand, precision, recall and F-measure, and on the other hand, the indexing consistency measure.

Precision, recall and F-measure are used extensively in the field of information retrieval (Gupta et al.,2015; Gil-Leiva, 2017a; Al-Zoghby, 2018; Seiler, Hübner & Paech, 2019; and Lin et al., 2020, to cite some recent examples). These metrics have been adapted for other purposes such as evaluating information extraction, summarizing and comparing automatic indexing with golden indexing.

In order to evaluate the systems’ automatic indexing output, the output keywords (the generated keywords) were compared with the human-assigned keywords for each document. We considered an output keyword to be “relevant” if it was an exact match with a human-assigned keyword. Precision, recall and F1 scores were calculated at document level, and then aggregated over the document collection.

Examples of use of these metrics to compare indexing pairs include their use by Krapivin et al. (2008), Shams and Mercer (2012b) and El-Haj et al. (2013) to compare KEA results; by Bandim and Correa (2019) and Silva, Correa and Gil-Leiva (2020) to compare SISA; by Névéol et al. (2005) and Chebil et al. (2012) to compare CISMEF automatic indexing; and by Rae et al. (2021) to evaluate the Medical Text Indexer (MTI) system.

The indexing consistency measures applied are those proposed by Hooper (1965) and Rolling (1981), specifically designed to compare indexing. Experiments using Hooper’s measure include those by Mork, et al. (2017) in their evaluation of Medical Text Indexer system and by Silva, et al. (2020) to evaluate SISA; however, Sinkkilä et al. (2011), for example, used Rolling’s measure to evaluate MAUI.

In our experiment, we applied Hooper’s measures to compare the indexing performed automatically by SISA, KEA and MAUI with human indexing (golden indexing).

To apply KEA and MAUI, it was first necessary to convert the XML and PDF documents into text files (.txt files), as well as extract the descriptors of the human indexing and fill the key files (.key files).

Having compiled the golden indexing for the test set and following the training phase required by KEA and MAUI using 30 documents, the automatic indexing of the test set of 200 articles in XML and TXT format by the three tools was begun. The use of the same test set and controlled vocabulary for the automatic indexing of articles allows comparisons to be made between the automatic indexing of the three software programs.

We set a threshold of 10 as the maximum number of keywords that KEA and MAUI could generate. The threshold value was chosen to reflect the number of keywords that we estimated it would be reasonable for KEA and MAUI to extract from the documents in the test set, given the number of keywords assigned by the golden indexing.

Garcia Gutierrez (1984: 115) cited an experiment carried out in the early 1970s to learn about the reality of indexing in Great Britain, in which about twelve minutes were spent to obtain eleven to twenty keywords; regarding the findings of Garcia Gutierrez, the present study considers that the time taken to convert keywords in natural language into descriptors of a thesaurus or controlled vocabulary should be added. However, Farrow (1994: 158), citing Cleverdon (1962), indicated that the optimum time for indexing technical reports could be four minutes plus 60%, depending on the working conditions. On the other hand, Amat (1989: 176) stated that an average time of twenty minutes would be required to obtain about ten terms. And finally, in the AusLit repository they recommend using the following benchmarks: novel/drama, 30 minutes; biography/autobiography, 30 minutes; short story, 10-20 minutes; verse, 5 minutes; and critical article, 20-30 minutes. We considered that the indexing of a scientific article could take between ten and fifteen minutes, to try to establish some correlation between human and automatic indexing (Table I).

Among the automatic indexing systems, SISA takes by far the longest time to process documents. However, for all the systems, processing times are much shorter than the processing time required for human indexing.

Graph 1 shows the mean number of indexing terms assigned in human indexing (golden indexing). In addition, for each automatic indexing system, it shows the average number of indexing terms assigned as well as the average number of terms in common with the terms assigned during golden indexing. Here we can see that KEA and MAUI have similar average values. SISA has an average number of assigned terms per document that is closer to the average number of golden indexing terms and an average number of terms shared with golden indexing per document almost one unit higher than KEA and MAUI.

The classic formulas used to calculate the measurements in the experiment are shown in Table II, as well as the results obtained for article number 4 by the three automatic indexing systems.

In Table II, the D column value is the identifier of a document in the test set; the C column value is the number of correct terms assigned by each system in comparison to golden indexing for a document; the A column value is the total number of terms assigned by each system for a document; the GI column value is the number of golden indexing terms for a given document; the P column value is the precision; the R column value is the recall; and the F1 column value is the F-measure obtained by each system for a document.

Table II shows that, overall, the values of the metrics obtained by each system for article number 4 are different, with the exception of the similarity observed in the results obtained by SISA and MAUI.

Graph 2 shows the values of the metrics for the three documents where each automatic indexing system has performed best.

It can be observed that the best performance for each system is based on different documents. In addition, SISA has obtained higher values for precision, recall and F-measure. The values obtained by KEA and MAUI for F-measure are similar to each other, but KEA has better values for precision while MAUI has better values for recall.

On the other hand, the three worst non-zero results for the F-measure range between 5.9% and 11.8% for KEA, 7.4% and 10.5% for SISA, and 8.3% and 9.5% for MAUI. Thus, SISA, KEA and MAUI have similar values for precision, recall and F-measure in the worst cases.

Table III shows the statistics for the metric values achieved by the automatic indexing systems based on the test set.

According to unpaired t test results with conventional criteria, the difference between the means of the values of the metrics for SISA and those of each of the other systems is considered extremely statistically significant. However, according to unpaired t test results with conventional criteria, the difference between the means of the values of the metrics for KEA and MAUI is not statistically significant.

Graph 3 illustrates the similar mean performance of KEA and MAUI, and the superior mean performance of SISA.

Table IV shows the number of documents for which the performance of the automatic indexing systems falls in the intervals of F-measure values.

We can see in Table IV that the systems have the majority of the F-measure values falling in the interval of 16 to 49% for F-measure. SISA has roughly three times the number of Fmeasure values equal to or higher than 50% compared to KEA and MAUI. KEA and MAUI have similar numbers of documents in each interval of F-measure values. In addition, the number of documents with an F-measure of zero is as follows: SISA produced six, KEA eight, and MAUI ten.

According to Medelyan (2009), the similarity between the values achieved by KEA and the data for MAUI is explained by the fact that MAUI was developed from the foundations and structure of KEA. However, we can observe that besides their similar performance, there is a performance variation for each document in these systems, so that they capture both common and different aspects of the statistical nature of indexing terms.

SISA was conceived with the aim of mimicking a human indexer by directing the focus towards places in the text where it can identify significant terms. ISO standard 5963-1985 on human indexing of documents states that “important parts of the text need to be considered carefully, and particular attention should be paid to the following: a) the title, b) the abstracts, if provided; c) the list of contents; d) the introduction, the opening phrases of chapters and paragraphs, and the conclusion; e) illustration, diagrams, tables and their captions”. This recommendation by the standard derives from the experience of human indexers, hence SISA’s imitation of the behaviour of human indexers is perhaps the reason why SISA has achieved better results than KEA and MAUI, which do not base their processing on the structural positions of terms in documents.

Indexing consistency seeks to determine the similarity between sets of indexing terms from analysis of the same document. Therefore, there are a number of possible combinations for comparison: between human indexing, between automatic indexing, and between human indexing and automatic indexing.

Hooper’s measure has been extensively used to calculate indexing consistency. In the formula, C is the number of terms assigned by SISA, KEA and MAUI that match those assigned by golden indexing, A is the number of terms assigned by SISA, KEA or MAUI, and GI is the number of golden indexing terms for a given document. Table V summarizes the data achieved in our experiment.

Graph 4 shows the three documents with the highest consistency. SISA’s levels of consistency are higher than those of KEA and MAUI. KEA and MAUI have similar values of consistency for the best cases, but for different documents.

The three worst results achieved a range from 3.8% to 5.8% for SISA, 3% to 6.3% for KEA and 4.3% to 5% for MAUI. Again, the systems generate very similar values for the three worst cases.

In this section we analyze the three best performance results and the three worst non-zero results achieved by the three systems.

In Table VI we show the indexing that achieved the highest F-measure.

In Table VI, we can see in bold type the terms that golden indexing and the automatic indexing had in common with each other for the best cases. The number of terms they had in common is five or six terms. The terms proposed by automatic indexing which were not common to both the automatic and manual systems are however linked in some way to the golden indexing terms for each document. In addition, the number of terms in golden indexing and the number proposed by the systems is similar. In Annex, Table A shows the three best results with the same patterns as those indicated above.

Table B in Annex shows the worst non-zero results for SISA, KEA and MAUI. The number of terms they had in common is one or two terms. The terms proposed by automatic indexing which were not common to both the automatic and manual systems are linked in some way to the golden indexing terms for each document. In addition, a greater difference between the number of terms in golden indexing and the number of those proposed by the systems can be observed for some documents. In Annex, Table A shows the three worst non-zero results with the same patterns as those indicated here.

In Annex 1, Table A shows the three best and three worst non-zero results for SISA, KEA and MAUI. In the case of SISA, there is a considerable disparity between the number of terms assigned by SISA and the number of terms assigned by the database’s human indexers. In the case of Article 91, the human indexers assigned almost three times as many terms as SISA while for Article 94 they assigned twice as many. For Article 87, SISA assigned almost twice as many terms as those assigned by human indexing. This large disparity between SISA and human indexing also occurred for other articles that achieved a low F-measure of between 10-16% (Articles 1, 97, and 168, for example), although in other articles with a very similar number of assigned terms (Articles 10, 36 and 138) the F-measure was equally low.

If the indexing by a given tool gives an F-measure that is similar to the examples shown in Table VI, we consider that use of some automatic indexing software or semi-automatic could be of great help in order to increase document processing capacity in real working environments, or even by enabling the automatic indexing of documents that are not manually indexed.