Menu
Home
Research

Research

My research interest lies in Analog In-memory Computing, optimization and distributed or decentralized machine learning.

Training on Analog In-memory Computing Hardware

Modern deep model training often requires processing vast amounts of weights and data, which necessitates frequent data transfer between memory and processor, leading to the "von Neumann bottleneck" that can significantly hinder computation speed and efficiency. In this context, Analog in-memory computing (AIMC) is an innovative computing paradigm that utilizes the physical properties of emerging Resistive Processing Unit (RPU) devices to perform computations directly within the memory array. Its core principle is to harness the analog storage and processing capabilities of these devices—leveraging the physical laws of Ohm and Kirchhoff—to execute Matrix-Vector Multiplication (MVM) operations in a highly parallel and energy-efficient manner.

To fully realize the massive parallelism and energy efficiency benefits of AIMC, it is essential to perform the fully on-chip training process of Deep Neural Networks (DNNs) directly on the AIMC hardware. However, this ambitious goal faces significant challenges stemming from the inherent non-idealities of analog hardware. Key difficulties include the non-linear response of RPU devices, the pervasive presence of analog noise during computation, and the limited precision inherent in analog operations. Achieving the high accuracy required for training deep models, especially given these imperfections, remains a formidable obstacle.

Crucially, since these hardware imperfections, such as device non-linearities and analog noise, are fundamentally physical and cannot be entirely eliminated in the foreseeable future, algorithmic solutions become essential. To address this reality, my research focuses on developing novel algorithms and techniques that enable the effective training of DNNs on AIMC hardware. Our core objective is to establish a robust training paradigm that enables deep learning models to seamlessly "coexist" with the intrinsic imperfections of analog accelerators. Specifically, we aim to develop algorithms that are inherently robust against hardware imperfections, ultimately bridging the gap between the computational demands of DNNs and the realities of non-ideal analog in-memory accelerators.

alt text

Distributed Optimization

Distributed or decentralized learning, which involves a series of single devices (workers) collaborating to train a machine learning model, usually serves as a promising solution in the following scenarios:
  • Accelerate large-scale machine learning through parallel computation in data centers.
  • Exploit the potential value of large-volume, heterogeneous, and privacy-sensitive data located at geographically distributed devices in settings like federated learning (FL) or multi-agent reinforcement learning (MARL).
As one of the central topics in distributed networks, my recent work focus on robust distributed optimization algorithms. Despite the well-known advantages, the distributed nature of networks makes them vulnerable to workers’ misbehaviors, especially in FL scenarios. This misbehavior, including malicious misleading, data poisoning, backdoor injection, and so on, can be abstracted into a so-called Byzantine attack model, where some workers (attackers) send arbitrary malicious messages to others. Hoping to fulfill the robustness requirement, I dedicate myself to designing and analyzing Byzantine-resilient optimization algorithms.


alt text

Journals

Conference