In this contribution, the effects of enhancing direct citations, with respect to publication-publication relatedness measurement, by indirect citation relations (bibliographic coupling, co-citation, and extended direct citations) and text relations on clustering solution accuracy are analyzed. We include in the study, for comparison reasons, each approach that is involved in the enhancement of direct citations. In total, we investigate the relative performance of seven approaches. For the evaluation of the approaches, we use a methodology proposed by earlier research. However, the used evaluation criterion is based on MeSH, one of the most sophisticated publication-level classification schemes available. We also introduce an approach, based on interpolated accuracy values, by which overall relative clustering solution accuracy can be studied. The results show that the co-citation approach has the worst performance, and that the direct citations approach is outperformed by the other five investigated approaches. The extended direct citations approach has the best performance, followed by an approach in which direct citations are enhanced by the BM25 textual relatedness measure. An approach that combines direct citations with bibliographic coupling and co-citation performs slightly better than the bibliographic coupling approach, which in turn has a better performance than the BM25 approach.
This paper treats document-document similarity approaches in the context of science mapping. Five approaches, involving nine methods, are compared experimentally. We compare text-based approaches, the citation-based bibliographic coupling approach, and approaches that combine text-based approaches and bibliographic coupling. Forty-three articles, published in the journal Information Retrieval, are used as test documents. We investigate how well the approaches agree with a ground truth subject classification of the test documents, when the complete linkage method is used, and under two types of similarities, first-order and second-order. The results show that it is possible to achieve a very good approximation of the classification by means of automatic grouping of articles. One text-only method and one combination method, under second-order similarities in both cases, give rise to cluster solutions that to a large extent agree with the classification.
We compared three different bibliometric evaluation approaches: two citationbased approaches and one based on manual classification of publishing channels into quality levels. Publication data for two universities was used, and we worked with two levels of analysis: article and department. For the article level, we investigated the predictive power of field normalized citation rates and field normalized journal impact with respect to journal level. The results for the article level show that evaluation of journals based on citation impact correlate rather well with manual classification of journals into quality levels. However, the prediction from field normalized citation rates to journal level was only marginally better than random guessing. At the department level, we studied three different indicators in the context of research fund allocation within universities and the extent to which the three indicators produce different distributions of research funds. It turned out that the three distributions of relative indicator values were very similar, which in turn yields that the corresponding distributions of hypothetical research funds would be very similar.
In this large-scale contribution, we deal with the relationship between properties of cited references of Web of Science articles and the field normalized citation rate of these articles. Using nearly 1 million articles, and three classification systems with different levels of granularity, we study the effects of number of cited references, share of references covered by Web of Science, mean age of references and mean citation rate of references on field normalized citation rate. To expose the relationship between the predictor variables and the response variable, we use quantile regression. We found that a higher number of references, a higher share of references to publications within Web of Science and references to more recent publications correlate with citation impact. A correlation was observed even when normalization was done with a finely grained classification system. The predictor variables affected citation impact to a larger extent at higher quantile levels. Regarding the relative importance of the predictor variables, citation impact of the cited references was in general the least important variable. Number of cited references carried most of the importance for both low and medium quantile levels, but this importance was lessened at the highest considered level.
This work was conducted by the UArctic Thematic Network on Research Analytics and Bibliometrics. It was supported by Global Affairs Canada through the Global Arctic Leadership Initiative.
A similarity-oriented approach for deriving reference values used in citation normalization is explored and contrasted with the dominant approach of utilizing database-defined journal sets as a basis for deriving such values. In the similarity-oriented approach, an assessed article's raw citation count is compared with a reference value that is derived from a reference set, which is constructed in such a way that articles in this set are estimated to address a subject matter similar to that of the assessed article. This estimation is based on second-order similarity and utilizes a combination of 2 feature sets: bibliographic references and technical terminology. The contribution of an article in a given reference set to the reference value is dependent on its degree of similarity to the assessed article. It is shown that reference values calculated by the similarity-oriented approach are considerably better at predicting the assessed articles' citation count compared to the reference values given by the journal-set approach, thus significantly reducing the variability in the observed citation distribution that stems from the variability in the articles' addressed subject matter.
The purpose of this thesis is to contribute to the methodology at the intersection of relational and evaluative bibliometrics. Experimental investigations are presented that address the question of how we can most successfully produce estimates of the subject similarity between documents. The results from these investigations are then explored in the context of citation-based research evaluations in an effort to enhance existing citation normalization methods that are used to enable comparisons of subject-disparate documents with respect to their relative impact or perceived utility. This thesis also suggests and explores an approach for revealing the uncertainty and stability (or lack thereof) coupled with different kinds of citation indicators.This suggestion is motivated by the specific nature of the bibliographic data and the data collection process utilized in citation-based evaluation studies.
The results of these investigations suggest that similarity-detection methods that take a global view of the problem of identifying similar documents are more successful in solving the problem than conventional methods that are more local in scope. These results are important for all applications that require subject similarity estimates between documents. Here these insights are specifically adopted in an effort to create a novel citation normalization approach that – compared to current best practice – is more in tune with the idea of controlling for subject matter when thematically different documents are assessed with respect to impact or perceived utility. The normalization approach is flexible with respect to the size of the normalization baseline and enables a fuzzy partition of the scientific literature. It is shown that this approach is more successful than currently applied normalization approaches in reducing the variability in the observed citation distribution that stems from the variability in the articles’ addressed subject matter. In addition, the suggested approach can enhance the interpretability of normalized citation counts. Finally, the proposed method for assessing the stability of citation indicators stresses that small alterations that could be artifacts from the data collection and preparation steps can have a significant influence on the picture that is painted by the citationindicator. Therefore, providing stability intervals around derived indicators prevents unfounded conclusions that otherwise could have unwanted policy implications.
Together, the new normalization approach and the method for assessing the stability of citation indicators have the potential to enable fairer bibliometric evaluative exercises and more cautious interpretations of citation indicators.
In this paper, we compare two sophisticated publication-level approaches to ex-post citation normalization: an item-oriented approach and an approach falling under the general algorithmically constructed classification system approach. Using articles published in core journals in Web of Science (SCIE, SSCI & A&HCI) during 2009 (n=955,639), we first examine, using the measure Proportion explained variation (PEV), to what extent the publication-level approaches can explain and correct for variation in the citation distribution that stems from subject matter heterogeneity. We then, for the subset of articles from life science and biomedicine (n=456,045), gauge the fairness of the normalization approaches with respect to their ability to identify highly cited articles when subject area is factored out. This is done by utilizing information from publication-level MeSH classifications to create high quality subject matter baselines and by using the measure Deviations from expectations (DE). The results show that the item-oriented approach had the best performance regarding PEV. For DE, only the most fine-grained clustering solution could compete with the item-oriented approach. However, the item-oriented approach performed better when cited references were heavily weighted in the similarity calculations.
The measurement of similarity between objects plays a role in several scientific areas. In this article, we deal with document–document similarity in a scientometric context. We compare experimentally, using a large dataset, first-order with second-order similarities with respect to the overall quality of partitions of the dataset, where the partitions are obtained on the basis of optimizing weighted modularity. The quality of a partition is defined in terms of textual coherence. The results show that the second-order approach consistently outperforms the first-order approach. Each difference between the two approaches in overall partition quality values is significant at the 0.01 level.
This paper deals with document-document similarity approaches, the issue of similarity order, and clustering methods, in the context of science mapping. Using two data sets of bibliographic records, associated with the fields of information retrieval and scientometrics, we investigate how well two document-document similarity approaches, a text-based approach and bibliographic coupling, agree with ground truth classifications (obtained by subject experts), under first-order and second-order similarities, and under four different clustering methods. The clustering methods are average linkage, complete linkage, Ward’s method and consensus clustering. The performance of first-order and second-order similarities is compared within the two document-document similarity approaches, and under each clustering method. We also compare the performance of the clustering methods. The results show that the text-based approach consistently outperformed bibliographic coupling with regard to the information retrieval data set, but performed consistently worse than the latter approach regarding the scientometrics data set. For the similarity order issue, second-order similarities performed better than first-order in 12 out of 16 cases. Average linkage had the best overall performance among the clustering methods, followed by consensus clustering. The main conclusion of the study is that second-order similarities seem to be a better choice than first-order in the science mapping context.
In this paper we study the effects of field normalization baseline on relative performance of 20 natural science departments in terms of citation impact. Impact is studied under three baselines: journal, ISI/Thomson Reuters subject category, and Essential Science Indicators field. For the measurement of citation impact, the indicators item-oriented mean normalized citation rate and Top-5% are employed. The results, which we analyze with respect to stability, show that the choice of normalization baseline matters. We observe that normalization against publishing journal is particular. The rankings of the departments obtained when journal is used as baseline, irrespective of indicator, differ considerably from the rankings obtained when ISI/Thomson Reuters subject category or Essential Science Indicators field is used. Since no substantial differences are observed when the baselines Essential Science Indicators field and ISI/Thomson Reuters subject category are contrasted, one might suggest that people without access to subject category data can perform reasonable normalized citation impact studies by combining normalization against journal with normalization against Essential Science Indicators field.
This article highlights the media historical possibilities to analyse linguistic patterns in massive amounts of texts using digital methods. Our starting point is the fact that The National Library of Sweden has made over 12 million newspaper pages available in digital format. An important question is how to research them. The article presents a media history of the Swedish newspaper digitisation, as well as new ways of conducting historical newspaper research using digital methods. A case study is presented where the conceptualisation of a new media technology (the internet) in newspapers from the 1990s is tracked with a digital tool searching for word co-occurrences. The possibilities of digital methods are often incredible, but we should not underestimate the problematic aspects of using digital tools to explore digitised newspapers. The poor quality of the OCR (Optical Character Recognition) is described as one of the major challenges facing historical newspaper research in a digital environment
Publishing in peer-reviewed journals as a part of the doctoral education is common practice in many countries. The publication output of doctoral students is increasingly used in selection processes for funding and employment in their early careers. Against the backdrop of this development, the aim of this study is to examine (1) how performance during the doctoral education affect the probability of attaining research excellence in the early career; and (2) if there is performance differences between males and females in the early career and to which degree these gender differences can be explained by performance differences during the doctoral education. The data consist of Swedish doctoral students employed at the faculty of science and technology and the faculty of medicine at a Swedish university. Our main conclusions are that (1) research performance during the doctoral education has a positive effect on attaining excellence in the early career; (2) there is an interaction between publication volume and excellence during doctoral education suggesting that a combination of quantity and quality in doctoral students’ performance is indicative of future excellence; (3) there are performance differences in the early career indicating that males have a higher probability of attaining excellence than females, and; (4) this difference is partly explained by performance differences during the doctoral education.
The purpose of this study was to investigate the predictive value of using bibliometric indicators of scientific performance during doctoral studies to predict who will attain future excellence in a local organizational context. The data consisted of 479 Swedish doctoral students employed at a single Swedish university that completed their studies between 2003 and 2009. We used a probit regression model to estimate the probability for future excellence, operationalized with a citation based indicator. The model included five predictors: publication volume, excellence during doctoral studies, collaboration, age at thesis completion, and gender. Our main results were: (1) an interaction between publication volume and attaining excellence during doctoral studies, indicating that the effect of publication volume on the probability of attaining future excellence is much stronger for the group of excellent doctoral students than for the group of non-excellent students; (2) collaboration and age are significant predictors of future excellence; (3) examining potential gender bias the results were somewhat inconclusive. Male doctoral students had a higher probability of attaining future excellence. However, the effect was not significant (p>0.05). Our main conclusion is that bibliometric indicators has some predictive validity for post-doctoral performance in a local organizational context and that a combination of quantity and quality in doctoral students’ performance generated the highest probabilities of future excellence.
This article provides an explanation for previously observed gender differences in scientific performance during doctoral studies and the early career. Data is based on doctoral students in science, technology, and medicine at a Swedish university. We collected information on each doctoral student’s publication and employment history. We also created publication histories for the doctoral candidates main supervisors. The data was supplemented with information on gender, age, and research area. Informed by theories on academic socialization, our research questions focus on how gender differences in productivity during doctoral studies and the early career relate to research collaboration and behaviour/characteristics of the main supervisor. Results show that the gender gap in productivity during doctoral studies, and the early career, can be explained by the degree to which the doctoral students co-author publications with their main supervisors and the size of their collaborative networks.
Bibliometric methods were used to examine: (1) research themes in sport and exercise psychology articles published between 2008 and 2011; and (2) the intellectual base of the field of sport and exercise psychology, defined as influential literature being cited in these articles. The dataset consisted of 795 articles from five sport and exercise psychology journals and 345 articles obtained through citation-based extension (n = 1140 articles). A cluster analysis yielded 73 clusters showing themes in sport and exercise psychology research. Principal component analysis was used to identify and analyze relationships between 14 highly cited research areas constituting the intellectual base of sport and exercise psychology. Some main findings were: (1) the identification of many re-emerging themes, (2) research related to motivation seems to be extensive, (3) sport psychology and exercise psychology research share theoretical frameworks to some extent, however (4) differences compared to previous reviews indicate that sport psychology and exercise psychology may be regarded as two distinct research fields, rather than one united field, and (5) isolated research areas were identified indicating potential for research integration. Suggestions for future research are provided. The bibliometric approach presented a broad overview of trends and knowledge base in sport and exercise psychology research.
In this contribution, the effects of enhancing direct citations, with respect to publication-publication relatedness measurement, by indirect citation relations (bibliographic coupling and co-citation) and text relations on clustering accuracy are analyzed. In total, we investigate six approaches. In one of these, direct citations are enhanced by both bibliographic coupling and co-citation, whereas text relations are used to enhance direct citations in another approach. In addition to an approach based on direct citations only, we include in the study, for comparison reasons, each approach that is involved in the enhancement of direct citations. For the evaluation of the approaches, we use a methodology proposed by earlier research. However, the used evaluation criterion is based on MeSH, arguable the most sophisticated item-level classification scheme available. The results show that the co-citation approach has the worst performance, and that the direct citations approach is outperformed by the other four investigated approaches. An approach in which direct citations are enhanced by the BM25 textual relatedness measure has the best performance, followed by the approach that combines direct citations with bibliographic coupling and co-citation. The latter performs slightly better than the bibliographic coupling approach, which in turn has a better performance than the BM25 approach.