Accelerating non-AI/ML applications using AI/ML accelerators

The explosive demand on AI/ML workloads drive the emergence of AI/ML accelerators, including commercialized NVIDIA Tensor Cores and Google TPUs. These AI/ML accelerators are essentially matrix processors and are theoretically helpful to any application with matrix operations. This project bridges the missing system/architecture/programming language support in democratizing AI/ML accelerators. As matrix operations are conventionally inefficient, this project also revises the core algorithm in compute kernels to better utilize operators of AI/ML accelerators.

• Kuan-Chieh Hsu and Hung-Wei Tseng. Simultaneous and Heterogeneous Multithreading. In the 56th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2023. [arXiv] [GitHub]
• Dongho Ha, Woo Won Ro, Hung-Wei Tseng. TensorCV: Accelerating Non-AI/ML Stages in Computing Vision Pipelines using Tensor Processors. In 2023 ACM/IEEE International Symposium on Low Power Electronics and Design (ISLPED 2023)
• Yunan Zhang, Po-An Tsai and Hung-Wei Tseng. SIMD2: A Generalized Matrix Instruction Set for Accelerating Tensor Computation beyond GEMM. In 49th International Symposium on Computer Archi- tecture, ISCA 2022, 2022. [arXiv] [GitHub]
• Yu-Ching Hu, Yuliang Li and Hung-Wei Tseng. TCUDB: Accelerating Database with Tensor Processors. In the 2022 ACM SIGMOD/PODS International Conference on Management of Data, SIGMOD 2022, 2022. [arXiv] [GitHub]
• Kuan-Chieh Hsu and Hung-Wei Tseng. Accelerating Applications using Edge Tensor Processing Units. In The International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2021. [arXiv] [GitHub]
• Yu-Chia Liu and Hung-Wei Tseng. NDS: N-Dimensional Storage. In the 54th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2021. (Best Paper Nominee) [GitHub]

Intelligent Data Storage

As parallel computer architectures significantly shrinking the execution time in compute kernels, the performance bottlenecks of applications shift to the rest of part of execution, including data movement, object deserialization/serialization as well as other software overheads in managing data storage. To address this new bottleneck, the best approach is to not move data and endow storage devices with new roles.

Morpheus is one of the very first research project that implements this concept in real systems. We utilize existing, commercially available hardware components to build the Morpheus-SSD. The Morpheus model not only speeds up a set of heterogeneous computing applications by 1.32x, but also allows these applications to better utilize emerging data transfer methods that can send data directly to the GPU via peer-to-peer to further achieve 1.39x speedup. Summarizer further provides mechanisms to dynamically adjust the workload between the host and intelligent SSDs, making more efficient use of all computing units in a system and boost the performance of big data analytics. This line of research also helps Hung-Wei's team receive Facebook research award, 2018.

• Yu-Chia Liu, Kuan-Chieh Hsu and Hung-Wei Tseng. Autonomous In-Storage Processing. In 60th Design Automation Conference (DAC 2023)
• Yu-Chia Liu and Hung-Wei Tseng. NDS: N-Dimensional Storage. In the 54th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2021. (Best Paper Nominee)
• Yu-Ching Hu, Murtuza Lokhandwala, Te I and Hung-Wei Tseng. Varifocal Storage: Dynamic Multi-Resolution Data Storage. In IEEE Micro (Micro Toppicks from Computer Architecture Conferences), 2020.
• Yu-Ching Hu, Murtuza Lokhandwala, Te I and Hung-Wei Tseng. Dynamic Multi-Resolution Data Storage. In in the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2019 (Best Paper Honorable Mention)
• Kiran Kumar Matam, Gunjae Koo, Haipeng Zha, Hung-Wei Tseng and Murali Anavarum. GraphSSD: Graph Semantics Aware SSD. In the 46th International Symposium on Computer Architecture, ISCA 2019.
• Facebook research award, 2018
• Hung-Wei Tseng, Qianchen Zhao, Yuxiao Zhou, Mark Gahagan and Steven Swanson. Morpheus: Exploring the Potential of Near-Data Processing for Creating Application Objects in Heterogeneous Computing. SIGOPS Operating Systems Review, volume 51(2):71 -- 83, August 2018.
• Gunjae Koo, Kiran Kumar Matam, Te I, Hema Venkata Krishna Giri Narra, Jing Li, Steven Swanson, Murali Annavaram, and Hung-Wei Tseng. Summarizer: Trading Bandwidth with Computing Near Storage. In 50th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2017
• Yanqin Jin, Hung-Wei Tseng, Steven Swanson and Yannis Papakonstantinou. KAML: A Flexible, High-Performance Key-Value SSD. In 23rd International Symposium on High Performance Computer Architecture (HPCA 2017). February 2017.
• Hung-Wei Tseng, Qianchen Zhao, Yuxiao Zhao, Mark Gahagan and Steven Swanson. Morpheus: Creating Application Objects Efficiently for Heterogeneous Computing. In 43rd International Symposium on Computer Architecture (ISCA 2016). June 2016.

Building Efficient Heterogeneous Computers

As the discontinuation of Dannard scaling and Moore's Law, computers become heterogeneous. However, moving data among heterogeneous computing units and storage devices becomes an emerging bottleneck in these systems.

My research proposes the "Hippogriff" system that revisits the programming model of moving data in heterogeneous computer systems. Instead of using the conventional CPU-centric, programmer-specified methods, the Hippogriff system simplifies the application interface and provide a middle layer to efficiently handle the data movement. We also implemented peer-to-peer data transfer between the GPU and the SSD in the Hippogriff system.

The preliminary result demonstrates 46% performance gain by applying Hippogriff to a set of rodinia GPU applications. For highly optimized GPU MapReduce framework, Hippogriff still demonstrates up to 27% performance gain.

• Yu-Chia Liu and Hung-Wei Tseng. NDS: N-Dimensional Storage. In the 54th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2021. [GitHub]
• Abenezer Wudenhe and Hung-Wei Tseng. TPUPoint: Automatically Characterizing Hardware Accelerated Data Center Machine Learning Program Behavior. In the 2021 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS 2021). [Github]
• Jinyoung Choi, Sergey Blagodurov and Hung-Wei Tseng. Dancing in the Dark: Profiling in the Age of Tiered Memory. In 35th IEEE International Parallel & Distributed Processing Symposium (IPDPS), 2021. [Github]
• Yu-Ching Hu, Murtuza Lokhandwala, Te I and Hung-Wei Tseng. Varifocal Storage: Dynamic Multi-Resolution Data Storage. In IEEE Micro (Micro Toppicks from Computer Architecture Conferences), 2020.
• Yu-Ching Hu, Murtuza Lokhandwala, Te I and Hung-Wei Tseng. Dynamic Multi-Resolution Data Storage. In the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2019
• Jing Li, Hung-Wei Tseng, Chunbin Lin, Steven Swanson, and Yannis Papakonstantinou. HippogriffDB: Balancing I/O and GPU Bandwidth in Big Data Analytics. Proceedings of VLDB Endowment, Volume 9(14), 2016.
• Yang Liu, Hung-Wei Tseng, Mark Gahagan, Jing Li, Yanqin Jin and Steven Swanson. Hippogriff: Efficiently Moving Data in Heterogeneous Computing Systems. In 34th IEEE International Conference on Computer Design (ICCD 2016). Oct. 2016.
• Yang Liu, Hung-Wei Tseng and Steven Swanson. SPMario: Scale Up MapReduce with I/O-Oriented Scheduling for the GPU. In 34th IEEE International Conference on Computer Design (ICCD 2016). Oct. 2016.
• Hung-Wei Tseng, Yang Liu, Mark Gahagan, Jing Li, Yanqin Jin, and Steven Swanson. Gullfoss: Accelerating and Simplifying Data Movement among Heterogeneous Computing and Storage Resources . Department of Computer Science and Engineering, University of California, San Diego technical report technical report CS2015-1015, 2015.

High-Quality, Low-Latency Realtime Systems

With hardware accelerators improving the latency in computation, the system software stack that were traditionally underrated in designing applications becomes more critical. In ESCAL, we focus on those underrated bottlenecks to achieve significant performance improvement without using new hardware. The most recent example is the OpenUVR system, where we eliminate unnecessary memory copies and allow the VR system loop to complete within 14 ms latency with just modern desktop PC, existing WiFi network links, raspberry Pi 4b+ and an HDMI compatible head mount display.

• Alec Rohloff, Zackary Allen, Kung-Min Lin, Joshua Okrend, Chengyi Nie, Yu-Chia Liu, and Hung-Wei Tseng. OpenUVR: an Open-Source System Framework for Untethered Virtual Reality Applications. In 27th IEEE Real-Time and Embedded Technology and Applications Symposium, RTAS 2021, 2021. [arXiv] [Github]