umu.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Science mapping and research evaluation: a novel methodology for creating normalized citation indicators and estimating their stability
Umeå University, Faculty of Social Sciences, Department of Sociology. (Inforsk)ORCID iD: 0000-0002-7653-4004
2014 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

The purpose of this thesis is to contribute to the methodology at the intersection of relational and evaluative bibliometrics. Experimental investigations are presented that address the question of how we can most successfully produce estimates of the subject similarity between documents. The results from these investigations are then explored in the context of citation-based research evaluations in an effort to enhance existing citation normalization methods that are used to enable comparisons of subject-disparate documents with respect to their relative impact or perceived utility. This thesis also suggests and explores an approach for revealing the uncertainty and stability (or lack thereof) coupled with different kinds of citation indicators.This suggestion is motivated by the specific nature of the bibliographic data and the data collection process utilized in citation-based evaluation studies.

The results of these investigations suggest that similarity-detection methods that take a global view of the problem of identifying similar documents are more successful in solving the problem than conventional methods that are more local in scope. These results are important for all applications that require subject similarity estimates between documents. Here these insights are specifically adopted in an effort to create a novel citation normalization approach that – compared to current best practice – is more in tune with the idea of controlling for subject matter when thematically different documents are assessed with respect to impact or perceived utility. The normalization approach is flexible with respect to the size of the normalization baseline and enables a fuzzy partition of the scientific literature. It is shown that this approach is more successful than currently applied normalization approaches in reducing the variability in the observed citation distribution that stems from the variability in the articles’ addressed subject matter. In addition, the suggested approach can enhance the interpretability of normalized citation counts. Finally, the proposed method for assessing the stability of citation indicators stresses that small alterations that could be artifacts from the data collection and preparation steps can have a significant influence on the picture that is painted by the citationindicator. Therefore, providing stability intervals around derived indicators prevents unfounded conclusions that otherwise could have unwanted policy implications.

Together, the new normalization approach and the method for assessing the stability of citation indicators have the potential to enable fairer bibliometric evaluative exercises and more cautious interpretations of citation indicators.

Place, publisher, year, edition, pages
Umeå: Umeå universitet , 2014. , 37 p.
Series
Akademiska avhandlingar vid Sociologiska institutionen, Umeå universitet, ISSN 1104-2508 ; 76
Keyword [en]
document-document similarity, science mapping, citation analysis, citation normalization, stability analysis, citation impact, research evaluation
National Category
Social Sciences Interdisciplinary Information Studies
Research subject
biblioteks- och informationsvetenskap
Identifiers
URN: urn:nbn:se:umu:diva-94189ISBN: 978-91-7601-134-8 (print)OAI: oai:DiVA.org:umu-94189DiVA: diva2:752675
Public defence
2014-10-31, Hörsal 1031, Norra Beteendevetarhuset, Umeå universitet, Umeå, 13:15 (English)
Opponent
Supervisors
Available from: 2014-10-10 Created: 2014-10-06 Last updated: 2015-04-01Bibliographically approved
List of papers
1. Document-document similarity approaches and science mapping: experimental comparison of five approaches
Open this publication in new window or tab >>Document-document similarity approaches and science mapping: experimental comparison of five approaches
2009 (English)In: Journal of Informetrics, ISSN 1751-1577, E-ISSN 1875-5879, Vol. 3, no 1, 49-63 p.Article in journal (Refereed) Published
Abstract [en]

This paper treats document-document similarity approaches in the context of science mapping. Five approaches, involving nine methods, are compared experimentally. We compare text-based approaches, the citation-based bibliographic coupling approach, and approaches that combine text-based approaches and bibliographic coupling. Forty-three articles, published in the journal Information Retrieval, are used as test documents. We investigate how well the approaches agree with a ground truth subject classification of the test documents, when the complete linkage method is used, and under two types of similarities, first-order and second-order. The results show that it is possible to achieve a very good approximation of the classification by means of automatic grouping of articles. One text-only method and one combination method, under second-order similarities in both cases, give rise to cluster solutions that to a large extent agree with the classification.

Place, publisher, year, edition, pages
Elsevier BV, 2009
Keyword
Bibliometrics, Citation data, Text mining, Cluster analysis, Data source combination, Science mapping
National Category
Computer and Information Science
Identifiers
urn:nbn:se:umu:diva-37580 (URN)10.1016/j.joi.2008.11.003 (DOI)000262496700005 ()
Available from: 2010-11-09 Created: 2010-11-09 Last updated: 2017-12-12Bibliographically approved
2. Experimental comparison of first and second-order similarities in a scientometric context
Open this publication in new window or tab >>Experimental comparison of first and second-order similarities in a scientometric context
2012 (English)In: Scientometrics, ISSN 0138-9130, E-ISSN 1588-2861, Vol. 90, no 2, 675-685 p.Article in journal (Refereed) Published
Abstract [en]

The measurement of similarity between objects plays a role in several scientific areas. In this article, we deal with document–document similarity in a scientometric context. We compare experimentally, using a large dataset, first-order with second-order similarities with respect to the overall quality of partitions of the dataset, where the partitions are obtained on the basis of optimizing weighted modularity. The quality of a partition is defined in terms of textual coherence. The results show that the second-order approach consistently outperforms the first-order approach. Each difference between the two approaches in overall partition quality values is significant at the 0.01 level.

Place, publisher, year, edition, pages
Budapest, Hungary: Akademiai Kiado, 2012
Keyword
Bibliographic coupling, Cluster analysis, Document–document similarity, Science mapping, Similarity order, Textual coherence
National Category
Other Computer and Information Science
Research subject
biblioteks- och informationsvetenskap
Identifiers
urn:nbn:se:umu:diva-46805 (URN)10.1007/s11192-011-0491-x (DOI)
Available from: 2011-09-15 Created: 2011-09-15 Last updated: 2017-12-08Bibliographically approved
3. A novel approach to citation normalization: a similarity-based method for creating reference sets
Open this publication in new window or tab >>A novel approach to citation normalization: a similarity-based method for creating reference sets
2015 (English)In: Journal of the Association for Information Science and Technology, ISSN 2330-1635, E-ISSN 2330-1643, Vol. 66, no 3, 489-500 p.Article in journal (Refereed) Published
Abstract [en]

A similarity-oriented approach for deriving reference values used in citation normalization is explored and contrasted with the dominant approach of utilizing database-defined journal sets as a basis for deriving such values. In the similarity-oriented approach, an assessed article's raw citation count is compared with a reference value that is derived from a reference set, which is constructed in such a way that articles in this set are estimated to address a subject matter similar to that of the assessed article. This estimation is based on second-order similarity and utilizes a combination of 2 feature sets: bibliographic references and technical terminology. The contribution of an article in a given reference set to the reference value is dependent on its degree of similarity to the assessed article. It is shown that reference values calculated by the similarity-oriented approach are considerably better at predicting the assessed articles' citation count compared to the reference values given by the journal-set approach, thus significantly reducing the variability in the observed citation distribution that stems from the variability in the articles' addressed subject matter.

Place, publisher, year, edition, pages
Wiley-Blackwell, 2015
Keyword
bibliometrics, scientometrics, citation analysis, normalized citation impact, similarity measures
National Category
Information Studies
Research subject
biblioteks- och informationsvetenskap
Identifiers
urn:nbn:se:umu:diva-88877 (URN)10.1002/asi.23193 (DOI)000350100500005 ()
Available from: 2014-05-16 Created: 2014-05-16 Last updated: 2017-12-05Bibliographically approved
4. The effects and their stability of field normalization baseline on relative performance with respect to citation impact: a case study of 20 natural science departments
Open this publication in new window or tab >>The effects and their stability of field normalization baseline on relative performance with respect to citation impact: a case study of 20 natural science departments
2011 (English)In: Journal of Informetrics, ISSN 1751-1577, E-ISSN 1875-5879, Vol. 5, no 1, 101-113 p.Article in journal (Refereed) Published
Abstract [en]

In this paper we study the effects of field normalization baseline on relative performance of 20 natural science departments in terms of citation impact. Impact is studied under three baselines: journal, ISI/Thomson Reuters subject category, and Essential Science Indicators field. For the measurement of citation impact, the indicators item-oriented mean normalized citation rate and Top-5% are employed. The results, which we analyze with respect to stability, show that the choice of normalization baseline matters. We observe that normalization against publishing journal is particular. The rankings of the departments obtained when journal is used as baseline, irrespective of indicator, differ considerably from the rankings obtained when ISI/Thomson Reuters subject category or Essential Science Indicators field is used. Since no substantial differences are observed when the baselines Essential Science Indicators field and ISI/Thomson Reuters subject category are contrasted, one might suggest that people without access to subject category data can perform reasonable normalized citation impact studies by combining normalization against journal with normalization against Essential Science Indicators field.

Keyword
bibliometrics, stability analysis, field normalization baseline, journal, ISI/Thomson Reuters subject category, Essential Science Indicators field, citation impact, scientometrics
National Category
Information Studies
Research subject
biblioteks- och informationsvetenskap
Identifiers
urn:nbn:se:umu:diva-37582 (URN)10.1016/j.joi.2010.09.003 (DOI)000285626000009 ()
Available from: 2010-11-09 Created: 2010-11-09 Last updated: 2017-12-12Bibliographically approved

Open Access in DiVA

spikblad(67 kB)23 downloads
File information
File name SPIKBLAD01.pdfFile size 67 kBChecksum SHA-512
647ac77a50460c6991cfaf7a9bb8eb9cf3cd03cc2851e315b08498f7b3af72460dc9d0c67ed1619da1344624e90211526fe491c13331c96070e72d5a6f5bc5f7
Type spikbladMimetype application/pdf
kappa(424 kB)532 downloads
File information
File name INSIDE01.pdfFile size 424 kBChecksum SHA-512
435c22261410a2e9f265079676a58fe138ebea2dc3c390d32d1a62d8830422d4b2c26cbbbd71f5a2f8adcbfd5f1c5ed694efa659080f7bbf3bf3d80f09b956a3
Type fulltextMimetype application/pdf
cover(6112 kB)31 downloads
File information
File name COVER01.pdfFile size 6112 kBChecksum SHA-512
e95ef8d3487909afdf7dfea62be0b71f4a7e1d5a5bce17577dd3c1c937f646b19a6d4931fc6807a9129e9c84c9303deba7cd02005906762dbbf43337c0d45bff
Type coverMimetype application/pdf

Search in DiVA

By author/editor
Colliander, Cristian
By organisation
Department of Sociology
Social Sciences InterdisciplinaryInformation Studies

Search outside of DiVA

GoogleGoogle Scholar
Total: 0 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 820 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf