Umeå University's logo

Spoken language is central to many human interactions and provides the medium through which activities and events across many humanities and social sciences disciplines are studied. It is also the object of active study in itself. As central to humanity as spoken language is, regulations aimed at mitigating privacy concerns also affect the affordance for collaborations on a national or larger scale based on spoken materials.  

The Visible Speech (VISP) platform is a web-based research infrastructure at Humlab, Umeå University, designed to handle audio recordings of speech in compliance with the national implementation of GDPR and security requirements. VISP provides a centralized environment for research of all disciplines in which recordings of spoken language constitute the primary material, meeting both researchers’ needs for efficient workflows and legislators’ demands for secure data management.

One of VISP's primary advantages is its ability to facilitate research on audio recordings that now constitute personally identifiable information (PII) under the application of the GDPR in Sweden. These recordings may contain sensitive content or have been made in sensitive contexts, classifying them as sensitive PII under national legislation. Sensitive contents may occur in relation to, for instance, the ethnicity and religious beliefs of the speaker, and sensitive contexts may occur when the recording is made in a healthcare context or in a context where a person’s membership with a union organization is divulged. While the challenges in conducting larger research efforts on the types of materials are currently aggravated by the implementation of the GDPR locally in Sweden, it is currently not clear to what extent upcoming AI regulation will, in effect, migrate identical or similar constraints to research in other countries in the EU as well.

The VISP platform offers a unified environment for storage, controlled access, direct work, and reproducible speech signal processing. VISP is built on the foundation of earlier research efforts1,2 and includes a comprehensive set of speech and voice analysis procedures within one framework. Thus, national research groups can collectively store interviews or other spoken language recordings, have automatic transcriptions or other speech processing performed, and access the results for complementary manual annotation or analysis simultaneously and securely. Additionally, VISP facilitates the digital archiving of projects through a uniform, documented, and transparent directory structure, reducing barriers to making data available following the FAIR principles. Research projects dealing with sensitive personal data in audio recording form require review by the Ethical Approval Authority and may subsequently take advantage of the VISP facilities.

A significant feature of VISP is its integration with the Swedish Academic Identity Federation (SWAMID, connected to eduGAIN), which enables researchers across Sweden to have secure, federated logins. This national federated login system allows researchers to access project data and collaborate on material processing in ways that were previously not possible. Moreover, VISP supports projects by lowering the step in to digital signal processing and audio analysis of the collected audio signals. This capability allows researchers to perform hands-on processing and analysis without the risk of disseminating sensitive audio recordings. By leveraging SWAMID, VISP ensures that researchers can work seamlessly and securely on collected materials, enhancing collaborative efforts and data handling efficiency. By providing tools for direct manipulation and examination of audio data, VISP ensures that all stages of data handling, from collection to analysis, are conducted within a secure environment, thereby maintaining the integrity and confidentiality of sensitive information.

The work conducted within VISP is part of SweCLARIN, the Swedish node of the European Research Infrastructure Consortium (ERIC) CLARIN. SweCLARIN aims to develop and provide national and European infrastructure for speech and text-based e-science, offering extensive digitized materials and advanced language technology tools. By combining the advanced technologies developed by CLARIN ERIC partners1 with stringent security protocols and leveraging federated login systems, VISP enables efficient and secure research on audio recordings of speech. The VISP components are available for download and setup of local instances and for modification, and the framework, therefore, promises to provide an invaluable tool for researchers, facilitating unprecedented collaboration and data processing within the digital humanities on both a national and larger scale.

References:

[1] R. Winkelmann, J. Harrington, K. Jänsch, EMU-SDMS: Advanced speech database management and analysis in R, Comput. Speech Lang. 45 (2017) 392–410.

[2] R. Winkelmann, J. Harrington, EMU-SDMS: R Centric Semi-automatic Speech Database Processing and Analysis. In Sasha Calhoun, Paola Escudero, Marija Tabain & Paul Warren (eds.) Proceedings of the 19th International Congress of Phonetic Sciences, Melbourne, Australia 2019 (pp. 1317--1321).  Canberra, Australia: Australasian Speech Science and Technology Association Inc

In this article, we re-trace the history of the Swedish SF archive and reflect on how this collection of historic newsreels has been reappropriated and remixed through-out more recent media history. In particular, we focus on the work of director and film historian Gardar Sahlberg, who made extensive use of the SF archive, first in a series of documentary films, then in a number of historical TV programmes. We are interested in how historic film footage travels and circulates through time, but foremost we explore how algorithms can help identify instances of audio-visual reuse in large datasets. Hence the article discusses algorithmic ways of examining archival film reuse, introducing a method for mapping video reuse with the help of artificial intelligence or more precisely machine learning that uses so-called convo-lutional neural nets. The article presents the Video Reuse Detector (VRD), a tool that uses machine learning to identify visual similarities within a given audiovisual database such as the SF archive.

Background: Bioconversion of wood into bioproducts and biofuels is hindered by the recalcitrance of woody raw material to bioprocesses such as enzymatic saccharification. Targeted modification of the chemical composition of the feedstock can improve saccharification but this gain is often abrogated by concomitant reduction in tree growth.

Results: In this study, we report on transgenic hybrid aspen (Populus tremula × tremuloides) lines that showed potential to increase biomass production both in the greenhouse and after 5 years of growth in the field. The transgenic lines carried an overexpression construct for Populus tremula × tremuloides vesicle-associated membrane protein (VAMP)-associated protein PttVAP27-17 that was selected from a gene-mining program for novel regulators of wood formation. Analytical-scale enzymatic saccharification without any pretreatment revealed for all greenhouse-grown transgenic lines, compared to the wild type, a 20–44% increase in the glucose yield per dry weight after enzymatic saccharification, even though it was statistically significant only for one line. The glucose yield after enzymatic saccharification with a prior hydrothermal pretreatment step with sulfuric acid was not increased in the greenhouse-grown transgenic trees on a dry-weight basis, but increased by 26–50% when calculated on a whole biomass basis in comparison to the wild-type control. Tendencies to increased glucose yields by up to 24% were present on a whole tree biomass basis after acidic pretreatment and enzymatic saccharification also in the transgenic trees grown for 5 years on the field when compared to the wild-type control.

Conclusions: The results demonstrate the usefulness of gene-mining programs to identify novel genes with the potential to improve biofuel production in tree biotechnology programs. Furthermore, multi-omic analyses, including transcriptomic, proteomic and metabolomic analyses, performed here provide a toolbox for future studies on the function of VAP27 proteins in plants.

Data integration has been proven to provide valuable information. The information extracted using data integration in the form of multiblock analysis can pinpoint both common and unique trends in the different blocks. When working with small multiblock datasets the number of possible integration methods is drastically reduced. To investigate the application of multiblock analysis in cases where one has a few number of samples and a lack of statistical power, we studied a small metabolomic multiblock dataset containing six blocks (i.e., tissue types), only including common metabolites. We used a single model multiblock analysis method called the joint and unique multiblock analysis (JUMBA) and compared it to a commonly used method, concatenated principal component analysis (PCA). These methods were used to detect trends in the dataset and identify underlying factors responsible for metabolic variations. Using JUMBA, we were able to interpret the extracted components and link them to relevant biological properties. JUMBA shows how the observations are related to one another, the stability of these relationships, and to what extent each of the blocks contribute to the components. These results indicate that multiblock methods can be useful even with a small number of samples

Patents are an important source of technological knowledge, but the amount of existing patents is vast and quickly growing. This makes development of tools and methodologies for quickly revealing patterns in patent collections important. In this paper, we describe how structured chemometric principles of multivariate data analysis can be applied in the context of text analysis in a novel combination with common machine learning preprocessing methodologies. We demonstrate our methodology in 2 case studies. Using principal component analysis (PCA) on a collection of 12338 patent abstracts from 25 companies in big pharma revealed sub-fields which the companies are active in. Using PCA on a smaller collection of patents retrieved by searching for a specific term proved useful to quickly understand how patent classifications relate to the search term. By using orthogonal projections to latent structures (O-PLS) on patent classification schemes, we were able to separate patents on a more detailed level than using PCA. Lastly, we performed multi-block modeling using OnPLS on bag-of-words representations of abstracts, claims, and detailed descriptions, respectively, showing that semantic variation relating to patent classification is consistent across multiple text blocks, represented as globally joint variation. We conclude that using machine learning to transform unstructured data into structured data provide a good preprocessing tool for subsequent chemometric multivariate data analysis and provides an easily interpretable and novel workflow to understand large collections of patents. We demonstrate this on collections of chemical and pharmaceutical patents.

Understanding and making the most of complex data collected from multiple sources is a challenging task. Data integration is the procedure of describing the main features in multiple data blocks, and several methods for multiblock analysis have been previously developed, including OnPLS and JIVE. One of the main challenges is how to visualize and interpret the results of multiblock analyses because of the increased model complexity and sheer size of data. In this paper, we present novel visualization tools that simplify interpretation and overview of multiblock analysis. We introduce a correlation matrix plot that provides an overview of the relationships between blocks found by multiblock models. We also present a multiblock scatter plot, a metadata correlation plot, and a variation distribution plot, that simplify the interpretation of multiblock models. We demonstrate our visualizations on an industrial case study in vibration spectroscopy (NIR, UV, and Raman datasets) as well as a multiomics integration study (transcript, metabolite, and protein datasets). We conclude that our visualizations provide useful tools to harness the complexity of multiblock analysis and enable better understanding of the investigated system.

In the present paper, we introduce an end-to-end workflow called joint and unique multiblock analysis (JUMBA), which allows multiple sources of data to be analyzed simultaneously to better understand how they complement each other. In near-infrared (NIR) spectroscopy, calibration models between NIR spectra and responses are used to replace wet-chemistry methods, and the models tend to be instrument-specific. Calibration-transfer techniques are used for standardization of NIR-instrumentation, enabling the use of one model on several instruments. The current paper investigates both the similarities and differences among a variety of NIR instruments using JUMBA. We demonstrate JUMBA on both a previously unpublished data set in which five NIR instruments measured mushroom substrate and a publicly available data set measured on corn samples. We found that NIR spectra from different instrumentation largely shared the same underlying structures, an insight we took advantage of to perform calibration transfer. The proposed JUMBA transfer displayed excellent calibration-transfer performance across the two analyzed data sets and outperformed existing methods in terms of both prediction accuracy and stability. When applied to a multi-instrument environment, JUMBA transfer can integrate all instruments in the same model and will ensure higher consistency among them compared with existing calibration-transfer methods.

Modern profiling technologies enable obtaining large amounts of data which can be later used for comprehensive understanding of the studied system. Proper evaluation of such data is challenging, and cannot be faced by bare analysis of separate datasets. Integrated approaches are necessary, because only data integration allows finding correlation trends common for all studied data sets and revealing hidden structures not known a priori. This improves understanding and interpretation of the complex systems. Joint and Unique MultiBlock Analysis (JUMBA) is an analysis method based on the OnPLS-algorithm that decomposes a set of matrices into joint parts containing variation shared with other connected matrices and variation that is unique for each single matrix. Mapping unique variation is important from a data integration perspective, since it certainly cannot be expected that all variation co-varies. In this work we used JUMBA for integrated analysis of lipidomic, metabolomic and oxylipin datasets obtained from profiling of plasma samples from children infected with P. falciparum malaria. P. falciparum is one of the primary contributors to childhood mortality and obstetric complications in the developing world, what makes development of the new diagnostic and prognostic tools, as well as better understanding of the disease, of utmost importance. In presented work JUMBA made it possible to detect already known trends related to disease progression, but also to discover new structures in the data connected to food intake and personal differences in metabolism. By separating the variation in each data set into joint and unique, JUMBA reduced complexity of the analysis, facilitated detection of samples and variables corresponding to specific structures across multiple datasets and by doing this enabled fast interpretation of the studied system. All this makes JUMBA a perfect choice for multiblock analysis of systems biology data.

Thanks to improvements in technology more data than ever before is generated in almost all fields of science and industry.

The data is analyzed to hopefully provide valuable information and knowledge about a product or process, such as how to improve the quality of a manufactured product.

Analysis of collected data is often performed on a single dataset or data source at a time. In this thesis, I have focused on multiblock analysis, a concept that includes multiple sources or data blocks.  Analogous to how the human senses combine to let us experience the world around us, multiblock analysis integrates multiple data sources, providing a fuller examination of the product or process under study.

My thesis introduces Joint and Unique Multiblock Analysis, JUMBA, a complete analysis workflow for data integration. I describe each step of JUMBA, including data pre-treatment, model building and validation as well as model interpretation. Special focus is put on several newly developed visualizations for model validation and interpretation to make it as easy as possible to draw conclusions from the analysis.

By reading my thesis, the reader will gain a working understanding of the process of performing multiblock analysis, including solutions to common problems that are often encountered.

Tack vare tekniska framsprång genereras det idag stora mängder data inom forskning och industri. Genom att analysera sådan data kan det i slutändan leda till att värdefull kunskap om en produkt eller process erhålls och kvaliteten på de studerade produkterna därmed kan ökas.

Analysen av data sker ofta på en enda datakälla, som då representeras av en matris, även kallat ett datablock. I denna avhandling har jag istället fokuserat på koncept som kan analysera flera datakällor samtidigt och integrera dessa. I likhet med hur människans sinnen låter oss uppleva världen runt omkring medför integrerandet av flera datakällor att undersökningen av en produkt eller process blir mer omfattande.

I min avhandling introduceras arbetsflödet JUMBA (Joint and Unique Multiblock Analysis, eng), som är ämnat för att utföra en fullständig integration av data. Jag beskriver varje enskilt steg av JUMBA, allt från förbehandling av data till byggande och validering av modeller samt deras tolkning. Jag har lagt särskild vikt vid att beskriva flera nyskapade typer av visualiseringar som underlättar att korrekta slutsatser kan dras från analysen.

Jag hoppas att läsaren av min avhandling kommer få förståelse för hur man utför analys av flera datablock och denne hittar även lösningar på problem man normalt sett kan ställas inför vid genomförandet.

Background: Secretory Carrier-Associated Membrane Proteins (SCAMPs) are highly conserved 32–38 kDa proteins that are involved in membrane trafficking. A systems approach was taken to elucidate function of SCAMPs in wood formation of Populus trees. Phenotypic and multi-omics analyses were performed in woody tissues of transgenic Populus trees carrying an RNAi construct for Populus tremula x tremuloides SCAMP3 (PttSCAMP3;Potri.019G104000).

Results: The woody tissues of the transgenic trees displayed increased amounts of both polysaccharides and lignin oligomers, indicating increased deposition of both the carbohydrate and lignin components of the secondary cell walls. This coincided with a tendency towards increased wood density as well as significantly increased thickness of the suberized cork in the transgenic lines. Multivariate OnPLS (orthogonal projections to latent structures) modeling of five different omics datasets (the transcriptome, proteome, GC-MS metabolome, LC-MS metabolome and pyrolysis-GC/MS metabolome) collected from the secondary xylem tissues of the stem revealed systemic variation in the different variables in the transgenic lines, including changes that correlated with the changes in the secondary cell wall composition. The OnPLS model also identified a rather large number of proteins that were more abundant in the transgenic lines than in the wild type. Several of these were related to secretion and/or endocytosis as well as both primary and secondary cell wall biosynthesis.

Conclusions: Populus SCAMP proteins were shown to influence accumulation of secondary cell wall components, including polysaccharides and phenolic compounds, in the woody tissues of Populus tree stems. Our multi-omics analyses combined with the OnPLS modelling suggest that this function is mediated by changes in membrane trafficking to fine-tune the abundance of cell wall precursors and/or proteins involved in cell wall biosynthesis and transport. The data provides a multi-level source of information for future studies on the function of the SCAMP proteins in plant stem tissues.