Umeå universitets logga

umu.sePublikationer
Ändra sökning
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
Timing performance benchmarking of out-of-distribution detection algorithms
Umeå universitet, Teknisk-naturvetenskapliga fakulteten, Institutionen för tillämpad fysik och elektronik.
Umeå universitet, Teknisk-naturvetenskapliga fakulteten, Institutionen för tillämpad fysik och elektronik.ORCID-id: 0000-0003-4228-2774
Umeå universitet, Teknisk-naturvetenskapliga fakulteten, Institutionen för tillämpad fysik och elektronik.
Umeå universitet, Teknisk-naturvetenskapliga fakulteten, Institutionen för tillämpad fysik och elektronik.ORCID-id: 0000-0003-0730-9441
Visa övriga samt affilieringar
2023 (Engelska)Ingår i: Journal of Signal Processing Systems, ISSN 1939-8018, E-ISSN 1939-8115, Vol. 95, nr 12, s. 1355-1370Artikel i tidskrift (Refereegranskat) Published
Abstract [en]

In an open world with a long-tail distribution of input samples, Deep Neural Networks (DNNs) may make unpredictable mistakes for Out-of-Distribution (OOD) inputs at test time, despite high levels of accuracy obtained during model training. OOD detection can be an effective runtime assurance mechanism for safe deployment of machine learning algorithms in safety–critical applications such as medical imaging and autonomous driving. A large number of OOD detection algorithms have been proposed in recent years, with a wide range of performance metrics in terms of accuracy and execution time. For real-time safety–critical applications, e.g., autonomous driving, timing performance is of great importance in addition to accuracy. We perform a comprehensive and systematic benchmark study of multiple OOD detection algorithms in terms of both accuracy and execution time on different hardware platforms, including a powerful workstation and a resource-constrained embedded device, equipped with both CPU and GPU. We also profile and analyze the internal details of each algorithm to identify the performance bottlenecks and potential for GPU acceleration. This paper aims to provide a useful reference for the practical deployment of OOD detection algorithms for real-time safety–critical applications.

Ort, förlag, år, upplaga, sidor
Springer-Verlag New York, 2023. Vol. 95, nr 12, s. 1355-1370
Nyckelord [en]
Deep Learning, Embedded systems, Machine Learning, Out-of-Distribution detection, Real-time systems
Nationell ämneskategori
Datavetenskap (datalogi)
Identifikatorer
URN: urn:nbn:se:umu:diva-206357DOI: 10.1007/s11265-023-01852-0ISI: 000955519800001Scopus ID: 2-s2.0-85150652364OAI: oai:DiVA.org:umu-206357DiVA, id: diva2:1753257
Tillgänglig från: 2023-04-26 Skapad: 2023-04-26 Senast uppdaterad: 2024-05-10Bibliografiskt granskad
Ingår i avhandling
1. Towards safe and efficient application of deep neural networks in resource-constrained real-time embedded systems
Öppna denna publikation i ny flik eller fönster >>Towards safe and efficient application of deep neural networks in resource-constrained real-time embedded systems
2023 (Engelska)Doktorsavhandling, sammanläggning (Övrigt vetenskapligt)
Abstract [en]

We consider real-time safety-critical systems that feature closed-loop interactions between the embedded computing system and the physical environment with a sense-compute-actuate feedback loop. Deep Learning (DL) with Deep Neural Networks (DNNs) has achieved success in many application domains, but there are still significant challenges in its application in real-time safety-critical systems that require high levels of safety certification under significant hardware resource constraints. This thesis considers the following overarching research goal: How to achieve safe and efficient application of DNNs in resource-constrained Real-Time Embedded (RTE) systems in the context of safety-critical application domains such as Autonomous Driving? Towards reaching that goal, this thesis presents a set of algorithms and techniques that aim to address three Research Questions (RQs): RQ1: How to achieve accurate and efficient Out-of-Distribution (OOD) detection for DNNs in RTE systems? RQ2: How to predict the performance of DNNs under continuous distribution shifts? RQ3: How to achieve efficient inference of Deep Reinforcement Learning (DRL) agents in RTE systems?

For RQ1, we present a framework for OOD detection based on outlier detection in one or more hidden layers of a DNN with either Isolation Forest (IF) or Local Outlier Factor (LOF). We also perform a comprehensive and systematic benchmark study of multiple well-known OOD detection algorithms in terms of both accuracy and execution time on different hardware platforms, in order to provide a useful reference for the practical deployment of OOD detection algorithms in RTE systems. For RQ2, we present a framework for predicting the performance of DNNs for end-to-end Autonomous Driving under continuous distribution shifts with two approaches: using an Autoencoder that attempts to reconstruct the input image; and applying Anomaly Detection algorithms to the hidden layer(s) of the DNN. For RQ3, we present a framework for model compression of the policy network of a DRL agent for deployment in RTE systems by leveraging the relevance scores computed by Layer-wise Relevance Propagation (LRP) to rank and prune the convolutional filters, combined with fine-tuning using policy distillation.

The algorithms and techniques developed in this thesis have been evaluated on standard datasets and benchmarks. To summarize our findings, we have developed novel OOD detection algorithms with high accuracy and efficiency; identified OOD detection algorithms with relatively high accuracy and low execution times through benchmarking; developed a framework for DNN performance prediction under continuous distribution shifts, and identified most effective Anomaly Detection algorithms for use in the framework; developed a framework for model compression of DRL agents that is effective in reducing model size and inference time for deployment in RTE systems. The research results are expected to assist system designers in the task of safe and efficient application of DNNs in resource-constrained RTE systems.

Ort, förlag, år, upplaga, sidor
Umeå: Umeå University, 2023. s. 49
Nyckelord
Machine Learning/Deep Learning, Real-Time Embedded systems, Out-of-Distribution Detection, Distribution Shifts, Deep Reinforcement Learning, Model Compression, Policy Distillation.
Nationell ämneskategori
Datavetenskap (datalogi)
Identifikatorer
urn:nbn:se:umu:diva-214365 (URN)978-91-8070-161-7 (ISBN)978-91-8070-160-0 (ISBN)
Disputation
2023-10-09, Triple Helix, Samverkanshuset, Umeå, 13:00 (Engelska)
Opponent
Handledare
Tillgänglig från: 2023-09-18 Skapad: 2023-09-12 Senast uppdaterad: 2023-09-13Bibliografiskt granskad

Open Access i DiVA

fulltext(3113 kB)37 nedladdningar
Filinformation
Filnamn FULLTEXT02.pdfFilstorlek 3113 kBChecksumma SHA-512
95e024c0faeaf01495c017b8b6ca3ee3d2062a3b5cdc1b9b21ecd86f42814989e99539f34257a344afde69e7c0363dbe989799b08d20aba79c0f88e49c9e2297
Typ fulltextMimetyp application/pdf

Övriga länkar

Förlagets fulltextScopus

Person

Luan, SiyuGu, ZonghuaSaremi, AminFreidovich, Leonid B.Jiang, Lili

Sök vidare i DiVA

Av författaren/redaktören
Luan, SiyuGu, ZonghuaSaremi, AminFreidovich, Leonid B.Jiang, Lili
Av organisationen
Institutionen för tillämpad fysik och elektronikInstitutionen för datavetenskap
I samma tidskrift
Journal of Signal Processing Systems
Datavetenskap (datalogi)

Sök vidare utanför DiVA

GoogleGoogle Scholar
Totalt: 176 nedladdningar
Antalet nedladdningar är summan av nedladdningar för alla fulltexter. Det kan inkludera t.ex tidigare versioner som nu inte längre är tillgängliga.

doi
urn-nbn

Altmetricpoäng

doi
urn-nbn
Totalt: 493 träffar
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf