Yue Jiang
Email: yue.jiang [at] aalto.fi

CV (Updated in Oct. 2024) | Linkedin | Google Scholar
WeChat Blog: yuejiang_phd

I am a Ph.D. student in the Computational Behavior Lab directed by Prof. Antti Oulasvirta, working on Computational User Interface Understanding and Eye Tracking on User Interfaces (Secondary advisor: Prof. Vikas Garg) at Aalto University and the Finnish Center for Artificial Intelligence (FCAI), Finland ๐Ÿ‡ซ๐Ÿ‡ฎ. I served as an Accessibility Chair for CHI2023 and CHI2024. I received the Meta PhD Fellowship, Google Europe Scholarship, and Nokia Scholarship. I was selected as a Heidelberg Laureate Forum Young Researcher and UCSD Data Science Rising Star.

I was a visiting Ph.D. student in Prof. Jeffrey Bigham's BIG Lab at Carnegie Mellon University in 2024. I also collaborate closely with Prof. Wolfgang Stuerzlinger (Simon Fraser University, Canada ๐Ÿ‡จ๐Ÿ‡ฆ) and Prof. Christof Lutteroth (University of Bath, UK ๐Ÿ‡ฌ๐Ÿ‡ง) on our series of OR-Constraint (ORC) Adaptive GUI layout projects.

Previously, I worked on Human Motion Capture and the Reconstruction of Non-Rigid Deformations at the Max Planck Institute for Informatics, Germany ๐Ÿ‡ฉ๐Ÿ‡ช. I got my Master's degree in Computer Graphics under Prof. Matthias Zwicker from the University of Maryland, College Park, USA ๐Ÿ‡บ๐Ÿ‡ธ, and my Bachelor's degree in Computer Science Specialist and Mathematics Major under Prof. Gerald Penn from the University of Toronto, Canada ๐Ÿ‡จ๐Ÿ‡ฆ.
Photo taken in Pittsburgh in 2024.

My Research

I aim to develop human-centered technologies that enhance human capabilities by adapting to users and their contexts.

Computational Representation for Graphical User Interfaces

     Explore how to best represent graphical UIs for computation.

      - Capture the key properties and characteristics of UI designs so that we do not need to encode them manually.
      - Effectively carry out the operations we need for downstream applications, such as how to learn, infer, optimize, and adapt.

Show/Hide Papers

Understanding Human Behaviors and Interactions

     - Capture human behaviors and interactions including eye tracking, human poses, and hand gestures.

     - Create models to predict and simulate human behaviors.

Show/Hide Papers

Highlights

  • I am currently serving as an Associate Chair (AC) for Full Papers at CHI2025, a Program Committee Member for full paper at IUI2025, and a Program Committee Member for the ASSETS2024 Experience Reports track.
  • I have been selected as a Heidelberg Laureate Forum Young Researcher.
  • I served as the Accessibility Co-Chair, Associate Chair (AC) for Late Breaking Works, and Session Chair for two sessions at CHI2024.
  • I am visiting Prof. Jeffrey P. Bigham's BIG Lab at Carnegie Mellon University in 2024.
  • I am excited to receive the 2023 Meta Research PhD Fellowship! Yay! :-)
  • We are organizing our third Computational User Interface workshop, 'Computational Methodologies for Understanding, Automating, and Evaluating User Interfaces' at CHI2024. Welcome to participate!
  • I served as the Accessibility Co-Chair, Workshop Juror, Associate Chair (AC) for Late Breaking Works, and Session Chair for two sessions at CHI2023.
  • I enjoyed my internship with the UI Understanding team under Jeff Nichols at Apple AIML so much! I wish it could be longer!
  • We organized our second Computational User Interface workshop, 'The Future of Computational Approaches for Understanding and Adapting User Interfaces' at CHI2023.
  • I served as a PC member at IUI2023.
  • I received the 2022 Google Europe Students with Disabilities Scholarship.
  • I received Special Recognition for Outstanding Reviews for CHI2024, UIST2022 and CHI2021.

Full Paper Publications

EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement Learning
Yue Jiang*, Zixin Guo*, Hamed R. Tavakoli, Luis A. Leiva, Antti Oulasvirta (*Equal Contribution)
ACM Symposium on User Interface Software and Technology (UIST2024)
DreamStruct: Understanding Slides and UIs via Synthetic Data Generation
Yi-Hao Peng, Faria Huq, Yue Jiang, Jason Wu, Xin Yue Li, Jeffrey Bigham, Amy Pavel
European Conference on Computer Vision (ECCV2024)
Paper | Supp
Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces
Yue Jiang, Changkong Zhou, Vikas Garg*, Antti Oulasvirta*
ACM CHI Conference on Human Factors in Computing Systems (CHI2024)
AXNav: Replaying Accessibility Tests from Natural Language
Maryam Taeb, Amanda Swearngin, Eldon Schoop, Ruijia Cheng, Yue Jiang, Jeffrey Nichols
ACM CHI Conference on Human Factors in Computing Systems (CHI2024)
Paper | Video | Video Preview
FlexDoc: Flexible Document Adaptation through Optimizing both Content and Layout
Yue Jiang, Christof Lutteroth, Rajiv Jain, Christopher Tensmeyer, Varun Manjunatha, Wolfgang Stuerzlinger, Vlad Morariu
IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC2024)
Paper | Video
VisRecall++: Analysing and Predicting Visualisation Recallability from Gaze Bahaviour
Yao Wang, Yue Jiang, Zhiming Hu, Constantin Ruhdorfer, Mihai Bรขce, Andreas Bulling
ACM Symposium on Eye Tracking Research and Applications (ETRA2024)
Impact of Design Decisions in Scanpath Modeling
Parvin Emami, Yue Jiang, Zixin Guo, Luis A. Leiva
ACM Symposium on Eye Tracking Research and Applications (ETRA2024)
UEyes: Understanding Visual Saliency across User Interface Types
Yue Jiang, Luis A. Leiva, Hamed R. Tavakoli, Paul R. B. Houssel, Julia Kylmala, Antti Oulasvirta
ACM CHI Conference on Human Factors in Computing Systems (CHI2023)
HiFECap: Monocular High-Fidelity and Expressive Capture of Human Performances
Yue Jiang, Marc Habermann, Vladislav Golyanik, Christian Theobalt
British Machine Vision Conference (BMVC2022)
Pretty Princess vs. Successful Leader: Gender Roles in Greeting Card Messages
Jiao Sun, Tongshuang Wu, Yue Jiang, Ronil Awalegaonkar, Xi Victoria Lin, Diyi Yang
ACM CHI Conference on Human Factors in Computing Systems (CHI2022) (Best Paper Honorable Mention)
ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints
Yue Jiang, Wolfgang Stuerzlinger, Christof Lutteroth
ACM CHI Conference on Human Factors in Computing Systems (CHI2021)
Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition
Karan Ahuja, Yue Jiang, Mayank Goel, Chris Harrison
ACM CHI Conference on Human Factors in Computing Systems (CHI2021)
โ€œPositive Energyโ€: Perceptions and Attitudes Towards COVID-19 Information on Social Media in China
Zhicong Lu, Yue Jiang, Chenxinran Shen, Margaret C Jack, Daniel Wigdor, Mor Naaman
ACM CHI Computer-Supported Cooperative Work and Social Computing (CSCW2021)
SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization
Yue Jiang, Dantong Ji, Zhizhong Han, Matthias Zwicker
IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR2020) (Oral Top 5%)
ORCSolver: An Efficient Solver for Adaptive GUI Layout with OR-Constraints
Yue Jiang, Wolfgang Stuerzlinger, Matthias Zwicker, Christof Lutteroth
ACM CHI Conference on Human Factors in Computing Systems (CHI2020)
The Government's Dividend: Complex Perceptions of Social Media Misinformation in China
Zhicong Lu, Yue Jiang, Cheng Lu, Mor Naaman, Daniel Wigdor
ACM CHI Conference on Human Factors in Computing Systems (CHI2020)
Paper
ORC Layout: Adaptive GUI Layout with OR-Constraints
Yue Jiang, Ruofei Du, Christof Lutteroth, Wolfgang Stuerzlinger
ACM CHI Conference on Human Factors in Computing Systems (CHI2019)

Workshop Papers, Extended Abstracts, and Doctoral Consortium

Workshop Organization

Computational Methodologies for Understanding, Automating, and Evaluating User Interfaces
Yue Jiang*, Yuwen Lu* (Equal Contribution), Tiffany Knearem, Clara Kliman-Silver, Christof Lutteroth, Toby Jia-Jun Li, Jeffrey Nichols, Wolfgang Stuerzlinger
ACM CHI Conference on Human Factors in Computing Systems (CHI2024)
The Future of Computational Approaches for Understanding and Adapting User Interfaces
Yue Jiang, Yuwen Lu, Christof Lutteroth, Toby Jia-Jun Li, Jeffrey Nichols, Wolfgang Stuerzlinger
ACM CHI Conference on Human Factors in Computing Systems (CHI2023)
Pervasive Eye Tracking and Mobile Eye-based Interaction (PETMEI)
Yao Wang, Ludwig Sidenmark, Teresa Hirzle, Yue Jiang, Andreas Bulling
ACM Symposium on Eye Tracking Research and Applications (ETRA2023)
Computational Approaches for Understanding, Generating, and Adapting User Interfaces
Yue Jiang*, Yuwen Lu*, Jeffrey Nichols, Wolfgang Stuerzlinger, Chun Yu, Christof Lutteroth, Yang Li, Ranjitha Kumar, Toby Jia-Jun Li
ACM CHI Conference on Human Factors in Computing Systems (CHI2022)

Education

  • Honors Bachelor of Science in Computer Science Specialist & Mathematics Major (High Distinction) @ University of Toronto ๐Ÿ‡จ๐Ÿ‡ฆ
    Sept. 2014 - Apr. 2017 (granted in 2018)
    Supervisor: Prof. Gerald Penn

Selected Professional Experience and Internship


Academic Service

  • Associate Chair (AC), CHI2025 Full Papers 2025
  • Accessibility Co-Chair x 2, CHI2023, CHI2024 FAQ 2023, 2024
  • Program Committee Member, ASSETS2024 Experience Reports 2024
  • Associate Chair (AC) x 5, CHI Late Breaking Works 2020, 2021, 2022, 2023, 2024
  • Session Chair, Gaze Interaction in Immersive Environments, CHI2024 2024
  • Session Chair, AI and UI Design, CHI2024 2024
  • HCI Master Program Admission Committee Member, Aalto University, Finland 2024
  • Ph.D. Admission Committee Member, Finnish AI Center, Finland 2024
  • Program Committee Member, AI and HCI Workshop, ICML2023 2023
  • Workshop Juror, CHI2023 2023
  • Program Committee Member, IUI2023 2023
  • Session Chair, Human AI Collaboration, CHI2023 2023
  • Session Chair, Text Input and Textual Communication, CHI2023 2023
  • Co-Organizer & Poster Chair, HelsinCHI 2023 HelsinCHI 2023 Webpage 2023
  • Organizing Committee Member, European Lab for Learning & Intelligent Systems (ELLIS) Doctoral Symposium 2023 2023
  • Co-Organizer, Seminar of HCI for Chinese HCI Researchers around the World 2021 - Now
  • Committee Member, ACM SIGGRAPH Research Development Committee --- DEI and Accessibility 2021 - 2023
  • Ph.D. Admission Committee Member, Finnish AI Center, Finland 2023
  • Co-Host, DIS2022 Ask-Me-Anything Session 2022
  • Program Committee Member, IUI2021 Demos and Posters 2021
  • Session Chair, MobileHCI 2021
  • Ph.D. & Master Admission Committee Member x 3, University of Maryland, College Park, USA 2019, 2020, 2021
  • Ph.D Student Panel Leader for Prospective Students, University of Maryland, College Park, USA 2019
  • Graduate Mentor for Technica Research Bootcamp x 2, USA 2018, 2019
  • Department of Mathematics Ambassador, University of Toronto, Canada 2016


Selected Honors and Grants

  • Nokia Scholarship 2024
  • Heidelberg Laureate Forum Young Researcher 2024
  • Meta Research PhD Fellowship 2023
  • Google Europe Students with Disabilities Scholarship 2022
  • Special Recognitions for Outstanding Reviews x 3 (CHI2021, UIST2022, CHI2024) 2021, 2022, 2024
  • Best Paper Honorable Mention, CHI2022 2022
  • Finnish Center for Artificial Intelligence (FCAI) Research Visit Funding (6 Months) 2023 - 2024
  • Finnish Center for Artificial Intelligence (FCAI) PhD Grant 2022 - 2025
  • ACM SIGCHI Gary Marsden Travel Award 2022
  • ACM SIGCHI Gary Marsden Travel Award (Virtual) 2021
  • Graduate Dean's Scholarship , University of Maryland, College Park 2018 - 2019, 2019 - 2020
  • CVPR2020 Oral Paper Top 5% 2020
  • Adobe Research Gift Grant for Document Intelligent Research Apr. 2020
  • Adobe Research Gift Grant for Document Intelligent Research Sept. 2020
  • Jacob K. Goldhaber Travel Grant Award 2019
  • Department of Computer Science Travel Grant Award , University of Maryland 2019
  • Grace Hopper Scholarship 2018
  • Dean's List Scholar x 4 , University of Toronto 2015, 2016, 2017, 2018
  • New College Council In-Course Scholarship x 2 , University of Toronto 2014 - 2015, 2015 - 2016
  • Award for Outstanding Achievement in Mathematical Expression and Reasoning for Computer Science , University of Toronto 2015
  • National Olympiad in Biology in Provinces, First Prize 2012
  • China Adolescents Science and Technology Innovation Contest, First Prize 2011


Selected Teaching Experience

  • Guest Lecturer on the topic of Visual Rerception Human Factors Engineering (Aalto University) ๐Ÿ‡ซ๐Ÿ‡ฎ May 2023
  • Research Project Mentor, Research Project in Human-Computer Interaction (Aalto University) ๐Ÿ‡ซ๐Ÿ‡ฎ Jan. 2023 - May 2023
  • Guest Lecturer on the topic of Visual Saliency Computational Design and Interaction (Aalto University) ๐Ÿ‡ซ๐Ÿ‡ฎ Nov. 2022
  • Co-Instructor, Seminar: Computing the User and their Interface (Saarland University) ๐Ÿ‡ฉ๐Ÿ‡ช Oct. 2021 - Feb. 2022
  • Course Co-Supervisor, Seminar: Computer Vision and Machine Learning for Computer Graphics (MPI) ๐Ÿ‡ฉ๐Ÿ‡ช Apr. 2021 - Aug. 2021
  • Teaching Assistant, CMSC740 Advanced Computer Graphics (UMD) ๐Ÿ‡บ๐Ÿ‡ธ Jan. 2020 - May. 2020
  • Teaching Assistant, CMSC427 Computer Graphics (UMD) ๐Ÿ‡บ๐Ÿ‡ธ Aug. 2019 - Dec. 2019
  • Teaching Assistant, CMSC427 Computer Graphics (UMD) ๐Ÿ‡บ๐Ÿ‡ธ Aug. 2018 - Dec. 2018
  • Teaching Assistant, CSC263 Data Structures and Analysis (UofT) ๐Ÿ‡จ๐Ÿ‡ฆ Jan. 2017 - Apr. 2017
  • Teaching Assistant, CSC263 Data Structures and Analysis (UofT) ๐Ÿ‡จ๐Ÿ‡ฆ Sept. 2016 - Dec. 2016
  • Teaching Assistant, CSC165: Mathematical Expression and Reasoning for Computer Science (UofT) ๐Ÿ‡จ๐Ÿ‡ฆ Jan. 2016 - Apr. 2016
  • Peer Tutor, Calculus, Linear Algebra I, Linear Algebra II (UofT) ๐Ÿ‡จ๐Ÿ‡ฆ Sept. 2015 - Apr. 2016
  • Teaching Volunteer, Elementary School and Junior High School Mathematics and Physics, Qinyuan, Shanxi, China ๐Ÿ‡จ๐Ÿ‡ณ 2014


My Name

    My first name is a combination of my mother's last name and "the Moon", which has the same pronunciation as "the Moon" in Chinese. My last name is "river". So my full name is "the Moon over the river".
    In China, "the Moon over the river" is widely used in poems to express the emotion that although we cannot live together with our family, we can still share the same Moon at night, and the Moon over the river can always send the best wishes to the ones we love!
    I hope all the families can have more chances to reunite. If it cannot be realized, I hope the Moon over the river can help us send the best wishes.


Acknowledgment

    I appreciate all the people who have helped or encouraged me throughout my life! I am an unlucky but also super lucky person. I may have gone through a little bit more ups and downs in my life than most people, but I always received countless support from so many people, no matter what situation I was in. My goal is to be a good mentor in the future since some unforgettable professors and mentors always supported me and served as examples for me.
    Prof. Wolfgang Stuerzlinger is the most important person in my academic journey. I am grateful for the opportunity he gave me to work on exciting HCI projects, even though we had never met in person until two years later. He was the person who transformed me from a novice student to a researcher. He continuously taught me the skills to conduct high-quality research and has been an unwavering source of encouragement and support both academically and emotionally throughout these years. It was also an unforgettable moment when Wolfgang squatted down to my level the first time we met, as he is tall. Together with Prof. Christof Lutteroth, they made a significant impact on my academic journey. The second project I worked on with them remains my favorite project to date. I thoroughly enjoyed the topic and the experience of working with both of them. Their support and encouragement meant a great deal to me and made me feel happy every day. I was inspired by their hard work and dedication, especially considering I was just a remote student.
    Prof. Faith Ellen was my idol during my undergrad years. She encouraged me to speak English, as I almost could not speak English when I went to Canada, and let me fall in love with data structure and computer theory. She taught me that persistence is much more important than intelligence.
    Prof. Gerald Penn was my undergraduate advisor in the Mathematical Linguistics group. He has some illusion about me that he often told other people that he would rank me as the best undergraduate student he has ever worked with. I really thank him for overpraising me.
    I was a data structure TA under Prof. Sam Toueg and Prof. Francois Pitt. I appreciate them giving me the chance to teach students data structure proofs for two hours every week. I broke all the chalks in the classroom during my first time teaching since I was too nervous. I still remember Sam and I worked past midnight to do the grading.
    My graduate advisors, Prof. Antti Oulasvirta and Prof. Matthias Zwicker have been instrumental in advancing my research and teaching abilities. Through their guidance and support, they have helped me to make meaningful progress in these areas.

Template: this.