Baidu.Inc, Beijing, China

       Research Intern,   Summer 2023

       Mentor: Yiding Liu, Dawei Yin

The University of British Columbia, Vancouver, Canada

       Research Intern(MITACS program),   Summer 2022

       Mentor: Xiaoxiao Li

The Pennsylvania State University, Pennsylvania, US

       Research Intern,   Summer 2021

       Mentor: Fenglong Ma

The University of Chicago, Chicago, US

       Online Intern,   Mar 2020-Mar 2021

       Mentor: Shinan Liu

University of Electronic Science and Technology of China, Chengdu, China

       Research Assisant,   Oct 2020-Present

       Mentor: Hongfang Yu

The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)
MSU & Baidu, advised by Professor Jiliang Tang, Dawei Yin, and Yiding Liu

Retrieval-augmented generation (RAG) is a powerful technique to facilitate language model with proprietary and private data, where data privacy is a pivotal concern. We’ve uncovered two pivotal aspects: privacy challenges within RAG’s own data and RAG’s potential to safeguard training data. (a). Data Leak Quantified: RAG systems can leak private retrieval data, with our study showing about 50% of sensitive retrieval data being output. (b). Mitigation Efforts: We have explored naive defenses such as summarization and retrieval thresholds. These methods help mitigate risks but do not completely resolve the issue, indicating the gravity of privacy risks in RAG. (c).Training Data Safeguard: RAG shows promise in protecting training data, offering a strategy to bolster privacy in AI systems.

Exploring Memorization in Fine-tuned Language Models
MSU & Baidu, advised by Professor Jiliang Tang, Dawei Yin, and Yiding Liu

In this work, we conduct the first comprehensive analysis to explore language models' (LMs) memorization during fine-tuning across tasks. (a). Fine-tuning Risks: Utilizing copyrighted or private data in fine-tuning poses privacy/IP risks. (b). Task Disparity: Summarization & Dialogue show high memorization, while QA and Classification are lower. (c). Task-specific Scaling: For high memorization tasks, memorization increases with larger models. Conversely, for low memorization tasks, increasing model size has little impact on memorization. (d). Attentions Role: High memorization tasks have uniform, sparse attention patterns. We unravel the nuances between attention & memorization.

Heterogeneous Federated Learning with Memorable Data Semantics in Industrial Metaverse
UESTC, Advised by professor Hongfang Yu

We present a high-performance and efficient system named HFEDMS for incorporating practical FL into Industrial Metaverse. HFEDMS reduces data heterogeneity through dynamic grouping and training mode conversion (Dynamic Sequential-to-Parallel Training, STP). Then, it compensates for the forgotten knowledge by fusing compressed historical data semantics and calibrates classifier parameters (Semantic Compression and Compensation, SCC). Finally, the network parameters of feature extractor and classifier are synchronized in different frequencies (Layer-wise Alternative Synchronization Protocol, LASP) to reduce communication costs. HFEDMS improves the classification accuracy by at least 6.4% and saves both the overall runtime and transfer bytes by up to 98%.

Knowledge-Enhanced Semi-Supervised Federated Learning for Aggregating Heterogeneous Lightweight Clients in IoT
PSU, Advised by Professor Fenglong Ma

We propose a novel SemiFL framework named pFedKnow. pFedKnow generates lightweight personalized client models via neural network pruning techniques to reduce communication cost. Moreover, it incorporates pretrained large models as prior knowledge to guide the aggregation of personalized client models and further enhance the framework performance. Experiment results on both image and text datasets show that the proposed pFedKnow outperforms state-of-the-art baselines as well as reducing considerable communication cost.