Yue Jiang
Email: yue.jiang [at] aalto.fi
CV (Updated in Oct. 2024) |
Linkedin |
Google Scholar
WeChat Blog: yuejiang_phd
I am a Ph.D. student in the Computational Behavior Lab directed by Prof. Antti Oulasvirta, working on Computational User Interface Understanding and Eye Tracking on User Interfaces (Secondary advisor: Prof. Vikas Garg) at
Aalto University and the Finnish Center for Artificial Intelligence (FCAI), Finland ๐ซ๐ฎ. I served as an Accessibility Chair for CHI2023 and CHI2024. I received the Meta PhD Fellowship, Google Europe Scholarship, and Nokia Scholarship. I was selected as a Heidelberg Laureate Forum Young Researcher and UCSD Data Science Rising Star.
I was a visiting Ph.D. student in Prof. Jeffrey Bigham's BIG Lab at Carnegie Mellon University in 2024.
I also collaborate closely with Prof. Wolfgang Stuerzlinger (Simon Fraser University, Canada ๐จ๐ฆ) and Prof. Christof Lutteroth (University of Bath, UK ๐ฌ๐ง) on our series of OR-Constraint (ORC) Adaptive GUI layout projects.
Previously, I worked on Human Motion Capture and the Reconstruction of Non-Rigid Deformations at the Max Planck Institute for Informatics, Germany ๐ฉ๐ช. I got my Master's degree in Computer Graphics under Prof. Matthias Zwicker from the
University of Maryland, College Park, USA ๐บ๐ธ, and my Bachelor's degree in
Computer Science
Specialist and
Mathematics
Major under Prof. Gerald Penn from the
University of Toronto, Canada ๐จ๐ฆ.
|
Photo taken in Pittsburgh in 2024.
|
I am on the job market for tenure-track faculty positions or relevant industry research positions (2024-2025).
Feel free to reach out if you are aware of any suitable openings!
My Research
Drawing on my research insights in human-computer interaction, computer vision, and deep learning, my goal is to develop human-centered technologies that adapt across various devices, taking user needs and context into account, to enhance human capabilities.
Computational Representation for Graphical User Interfaces
Explore how to best represent graphical UIs for computation.
- Capture the key properties and characteristics of UI designs so that we do not need to encode them manually.
- Effectively carry out the operations we need for downstream applications, such as how to learn, infer, optimize, and adapt.
Show/Hide Papers
Understanding Human Behaviors and Interactions
- Capture human behaviors and interactions including eye tracking, human poses, and hand gestures.
- Create models to predict and simulate human behaviors.
Show/Hide Papers
Highlights
-
I am currently serving as an Associate Chair (AC) for Full Papers at CHI2025,
a Program Committee Member for full paper at IUI2025,
and a Program Committee Member for the ASSETS2024 Experience Reports track.
-
I have been selected as a Heidelberg Laureate Forum Young Researcher.
-
I served as the Accessibility Co-Chair, Associate Chair (AC) for Late Breaking Works, and Session Chair for two sessions at CHI2024.
-
I am visiting Prof. Jeffrey P. Bigham's BIG Lab at Carnegie Mellon University in 2024.
-
I am excited to receive the 2023 Meta Research PhD Fellowship! Yay! :-)
-
We are organizing our third Computational User Interface workshop, 'Computational Methodologies for Understanding, Automating, and Evaluating User Interfaces' at CHI2024. Welcome to participate!
-
I served as the Accessibility Co-Chair, Workshop Juror, Associate Chair (AC) for Late Breaking Works, and Session Chair for two sessions at CHI2023.
-
I enjoyed my internship with the UI Understanding team under Jeff Nichols at Apple AIML so much! I wish it could be longer!
-
We organized our second Computational User Interface workshop, 'The Future of Computational Approaches for Understanding and Adapting User Interfaces' at CHI2023.
-
I served as a PC member at IUI2023.
-
I received the 2022 Google Europe Students with Disabilities Scholarship.
-
I received Special Recognition for Outstanding Reviews for CHI2024, UIST2022 and CHI2021.
Full Paper Publications
|
DreamStruct: Understanding Slides and UIs via Synthetic Data Generation
Yi-Hao Peng,
Faria Huq,
Yue Jiang,
Jason Wu,
Xin Yue Li,
Jeffrey Bigham,
Amy Pavel
European Conference on Computer Vision (ECCV2024)
|
|
Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces
Yue Jiang,
Changkong Zhou,
Vikas Garg*,
Antti Oulasvirta*
ACM CHI Conference on Human Factors in Computing Systems (CHI2024)
|
|
AXNav: Replaying Accessibility Tests from Natural Language
Maryam Taeb,
Amanda Swearngin,
Eldon Schoop,
Ruijia Cheng,
Yue Jiang,
Jeffrey Nichols
ACM CHI Conference on Human Factors in Computing Systems (CHI2024)
Paper |
Video |
Video Preview
|
|
FlexDoc: Flexible Document Adaptation through Optimizing both Content and Layout
Yue Jiang,
Christof Lutteroth,
Rajiv Jain,
Christopher Tensmeyer,
Varun Manjunatha,
Wolfgang Stuerzlinger,
Vlad Morariu
IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC2024)
Paper |
Video
|
|
VisRecall++: Analysing and Predicting Visualisation Recallability from Gaze Bahaviour
Yao Wang,
Yue Jiang,
Zhiming Hu,
Constantin Ruhdorfer,
Mihai Bรขce,
Andreas Bulling
ACM Symposium on Eye Tracking Research and Applications (ETRA2024)
|
|
Impact of Design Decisions in Scanpath Modeling
Parvin Emami,
Yue Jiang,
Zixin Guo,
Luis A. Leiva
ACM Symposium on Eye Tracking Research and Applications (ETRA2024)
|
|
UEyes: Understanding Visual Saliency across User Interface Types
Yue Jiang,
Luis A. Leiva,
Hamed R. Tavakoli,
Paul R. B. Houssel,
Julia Kylmala,
Antti Oulasvirta
ACM CHI Conference on Human Factors in Computing Systems (CHI2023)
|
|
HiFECap: Monocular High-Fidelity and Expressive Capture of Human Performances
Yue Jiang,
Marc Habermann,
Vladislav Golyanik,
Christian Theobalt
British Machine Vision Conference (BMVC2022)
|
|
Pretty Princess vs. Successful Leader: Gender Roles in Greeting Card Messages
Jiao Sun,
Tongshuang Wu,
Yue Jiang,
Ronil Awalegaonkar,
Xi Victoria Lin,
Diyi Yang
ACM CHI Conference on Human Factors in Computing Systems (CHI2022) (Best Paper Honorable Mention)
|
|
ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints
Yue Jiang,
Wolfgang Stuerzlinger,
Christof Lutteroth
ACM CHI Conference on Human Factors in Computing Systems (CHI2021)
|
|
Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition
Karan Ahuja,
Yue Jiang,
Mayank Goel,
Chris Harrison
ACM CHI Conference on Human Factors in Computing Systems (CHI2021)
|
|
โPositive Energyโ: Perceptions and Attitudes Towards COVID-19 Information on Social Media in China
Zhicong Lu,
Yue Jiang,
Chenxinran Shen,
Margaret C Jack,
Daniel Wigdor,
Mor Naaman
ACM CHI Computer-Supported Cooperative Work and Social Computing (CSCW2021)
|
|
SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization
Yue Jiang,
Dantong Ji,
Zhizhong Han,
Matthias Zwicker
IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR2020) (Oral Top 5%)
|
|
ORCSolver: An Efficient Solver for Adaptive GUI Layout with OR-Constraints
Yue Jiang,
Wolfgang Stuerzlinger,
Matthias Zwicker,
Christof Lutteroth
ACM CHI Conference on Human Factors in Computing Systems (CHI2020)
|
|
The Government's Dividend: Complex Perceptions of Social Media Misinformation in China
Zhicong Lu,
Yue Jiang,
Cheng Lu,
Mor Naaman,
Daniel Wigdor
ACM CHI Conference on Human Factors in Computing Systems (CHI2020)
|
|
ORC Layout: Adaptive GUI Layout with OR-Constraints
Yue Jiang,
Ruofei Du,
Christof Lutteroth, Wolfgang Stuerzlinger
ACM CHI Conference on Human Factors in Computing Systems (CHI2019)
|
Workshop Papers, Extended Abstracts, and Doctoral Consortium
-
Computational Representations for Graphical User Interfaces
Yue Jiang
ACM CHI Conference on Human Factors in Computing Systems (CHI2024 Doctoral Consortium)
[PDF]
-
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations
Yue Jiang,
Eldon Schoop,
Amanda Swearngin,
Jeffrey Nichols
ACM CHI Conference on Human Factors in Computing Systems (CHI2024 Workshop Paper)
[PDF]
-
Suggesting Colors for UI Designs with Graph Neural Networks
Busra Asan,
Lena Hegemann,
Yue Jiang (project advisor),
Antti Oulasvirta
ACM CHI Conference on Human Factors in Computing Systems (CHI2024 Workshop Paper)
-
Computational Methodologies for Understanding, Automating, and Evaluating User Interfaces
Yue Jiang*,
Yuwen Lu*,
Tiffany Knearem,
Clara Kliman-Silver,
Christof Lutteroth,
Toby Jia-Jun Li,
Jeffrey Nichols,
Wolfgang Stuerzlinger
ACM CHI Conference on Human Factors in Computing Systems (CHI2024 Extended Abstract)
[PDF]
-
UEyes: An Eye-Tracking Dataset across User Interface Types
Yue Jiang,
Luis A. Leiva,
Hamed R. Tavakoli,
Paul R. B. Houssel,
Julia Kylmala,
Antti Oulasvirta
ACM CHI Conference on Human Factors in Computing Systems (CHI2023 Workshop Paper)
[PDF]
-
Computational Assistance for User Interface Design: Smarter Generation and Evaluation of Design Ideas
Lena Hegemann,
Yue Jiang,
Joon-Gi Shin,
Yi-Chi Liao,
Markku Laine,
Antti Oulasvirta
ACM CHI Conference on Human Factors in Computing Systems (CHI2023 Interactivity)
[PDF]
-
The Future of Computational Approaches for Understanding and Adapting User Interfaces
Yue Jiang,
Yuwen Lu,
Christof Lutteroth,
Toby Jia-Jun Li,
Jeffrey Nichols,
Wolfgang Stuerzlinger
ACM CHI Conference on Human Factors in Computing Systems (CHI2023 Extended Abstract)
[PDF]
-
Pervasive Eye Tracking and Mobile Eye-based Interaction (PETMEI)
Yao Wang,
Ludwig Sidenmark,
Teresa Hirzle,
Yue Jiang,
Andreas Bulling
ACM Symposium on Eye Tracking Research and Applications (ETRA2023 Extended Abstract)
[PDF]
-
Computational Approaches for Understanding, Generating, and Adapting User Interfaces
Yue Jiang*,
Yuwen Lu*,
Jeffrey Nichols,
Wolfgang Stuerzlinger,
Chun Yu,
Christof Lutteroth,
Yang Li,
Ranjitha Kumar,
Toby Jia-Jun Li
ACM CHI Conference on Human Factors in Computing Systems (CHI2022 Extended Abstract)
[PDF]
-
Disability Semantics for Academic Writing
Garreth Tigwell, Yue Jiang
AccessSIGCHI
-
Designer-in-the-Loop Layout Autocompletion with Graph Neural Networks
Yue Jiang,
Vikas Garg,
Antti Oulasvirta
Finnish Center for Artificial Intelligence AI Day Workshop, 2022
-
DocShop: Bringing Document Content to Life
Yue Jiang
Adobe Research Project Expo, 2020
Workshop Organization
|
Computational Methodologies for Understanding, Automating, and Evaluating User Interfaces
Yue Jiang*,
Yuwen Lu* (Equal Contribution),
Tiffany Knearem,
Clara Kliman-Silver,
Christof Lutteroth,
Toby Jia-Jun Li,
Jeffrey Nichols,
Wolfgang Stuerzlinger
ACM CHI Conference on Human Factors in Computing Systems (CHI2024)
|
|
The Future of Computational Approaches for Understanding and Adapting User Interfaces
Yue Jiang,
Yuwen Lu,
Christof Lutteroth,
Toby Jia-Jun Li,
Jeffrey Nichols,
Wolfgang Stuerzlinger
ACM CHI Conference on Human Factors in Computing Systems (CHI2023)
|
|
Pervasive Eye Tracking and Mobile Eye-based Interaction (PETMEI)
Yao Wang,
Ludwig Sidenmark,
Teresa Hirzle,
Yue Jiang,
Andreas Bulling
ACM Symposium on Eye Tracking Research and Applications (ETRA2023)
|
|
Computational Approaches for Understanding, Generating, and Adapting User Interfaces
Yue Jiang*,
Yuwen Lu*,
Jeffrey Nichols,
Wolfgang Stuerzlinger,
Chun Yu,
Christof Lutteroth,
Yang Li,
Ranjitha Kumar,
Toby Jia-Jun Li
ACM CHI Conference on Human Factors in Computing Systems (CHI2022)
|
Education
-
Honors Bachelor of Science in
Computer Science
Specialist &
Mathematics
Major (High Distinction) @
University of Toronto ๐จ๐ฆ
Sept. 2014 - Apr. 2017 (granted in 2018)
Supervisor:
Prof. Gerald Penn
|
Selected Professional Experience and Internship
|
Academic Service
-
Associate Chair (AC), CHI2025 Full Papers
2025
-
Accessibility Co-Chair x 2, CHI2023, CHI2024 FAQ
2023, 2024
-
Program Committee Member, ASSETS2024 Experience Reports
2024
-
Associate Chair (AC) x 5, CHI Late Breaking Works
2020, 2021, 2022, 2023, 2024
-
Session Chair, Gaze Interaction in Immersive Environments, CHI2024
2024
-
Session Chair, AI and UI Design, CHI2024
2024
-
HCI Master Program Admission Committee Member, Aalto University, Finland
2024
-
Ph.D. Admission Committee Member, Finnish AI Center, Finland
2024
-
Program Committee Member, AI and HCI Workshop, ICML2023
2023
-
Workshop Juror, CHI2023
2023
-
Program Committee Member, IUI2023
2023
-
Session Chair, Human AI Collaboration, CHI2023
2023
-
Session Chair, Text Input and Textual Communication, CHI2023
2023
-
Co-Organizer & Poster Chair, HelsinCHI 2023 HelsinCHI 2023 Webpage
2023
-
Organizing Committee Member, European Lab for Learning & Intelligent Systems (ELLIS) Doctoral Symposium 2023
2023
-
Co-Organizer, Seminar of HCI for Chinese HCI Researchers around the World
2021 - Now
-
Committee Member, ACM SIGGRAPH Research Development Committee --- DEI and Accessibility
2021 - 2023
-
Ph.D. Admission Committee Member, Finnish AI Center, Finland
2023
-
Co-Host, DIS2022 Ask-Me-Anything Session
2022
-
Program Committee Member, IUI2021 Demos and Posters
2021
-
Session Chair, MobileHCI
2021
-
Ph.D. & Master Admission Committee Member x 3, University of Maryland, College Park, USA
2019, 2020, 2021
-
Ph.D Student Panel Leader for Prospective Students, University of Maryland, College Park, USA
2019
-
Graduate Mentor for Technica Research Bootcamp x 2, USA
2018, 2019
-
Department of Mathematics Ambassador, University of Toronto, Canada
2016
|
Selected Honors and Grants
-
Nokia Scholarship
2024
-
Heidelberg Laureate Forum Young Researcher
2024
-
Meta Research PhD Fellowship
2023
-
Google Europe Students with Disabilities Scholarship
2022
-
Special Recognitions for Outstanding Reviews x 3 (CHI2021, UIST2022, CHI2024)
2021, 2022, 2024
-
Best Paper Honorable Mention, CHI2022
2022
-
Finnish Center for Artificial Intelligence (FCAI) Research Visit Funding (6 Months)
2023 - 2024
-
Finnish Center for Artificial Intelligence (FCAI) PhD Grant
2022 - 2025
-
ACM SIGCHI Gary Marsden Travel Award
2022
-
ACM SIGCHI Gary Marsden Travel Award (Virtual)
2021
-
Graduate Dean's Scholarship
, University of Maryland, College Park
2018 - 2019, 2019 - 2020
-
CVPR2020 Oral Paper Top 5%
2020
-
Adobe Research Gift Grant for Document Intelligent Research
Apr. 2020
-
Adobe Research Gift Grant for Document Intelligent Research
Sept. 2020
-
Jacob K. Goldhaber Travel Grant Award
2019
-
Department of Computer Science Travel Grant Award
, University of Maryland
2019
-
Grace Hopper Scholarship
2018
-
Dean's List Scholar x 4
, University of Toronto
2015, 2016, 2017, 2018
-
New College Council In-Course Scholarship x 2
, University of Toronto
2014 - 2015, 2015 - 2016
-
Award for Outstanding Achievement in Mathematical Expression and Reasoning for Computer Science
, University of Toronto
2015
-
National Olympiad in Biology in Provinces, First Prize
2012
-
China Adolescents Science and Technology Innovation Contest, First Prize
2011
|
Selected Teaching Experience
-
Guest Lecturer on the topic of Visual Rerception
Human Factors Engineering (Aalto University) ๐ซ๐ฎ
May 2023
-
Research Project Mentor,
Research Project in Human-Computer Interaction (Aalto University) ๐ซ๐ฎ
Jan. 2023 - May 2023
-
Guest Lecturer on the topic of Visual Saliency
Computational Design and Interaction (Aalto University) ๐ซ๐ฎ
Nov. 2022
-
Co-Instructor,
Seminar: Computing the User and their Interface (Saarland University) ๐ฉ๐ช
Oct. 2021 - Feb. 2022
-
Course Co-Supervisor,
Seminar: Computer Vision and Machine Learning for Computer Graphics (MPI) ๐ฉ๐ช
Apr. 2021 - Aug. 2021
-
Teaching Assistant,
CMSC740 Advanced Computer Graphics (UMD) ๐บ๐ธ
Jan. 2020 - May. 2020
-
Teaching Assistant,
CMSC427 Computer Graphics (UMD) ๐บ๐ธ
Aug. 2019 - Dec. 2019
-
Teaching Assistant,
CMSC427 Computer Graphics (UMD) ๐บ๐ธ
Aug. 2018 - Dec. 2018
-
Teaching Assistant,
CSC263 Data Structures and Analysis (UofT) ๐จ๐ฆ
Jan. 2017 - Apr. 2017
-
Teaching Assistant,
CSC263 Data Structures and Analysis (UofT) ๐จ๐ฆ
Sept. 2016 - Dec. 2016
-
Teaching Assistant,
CSC165: Mathematical Expression and Reasoning for Computer Science (UofT) ๐จ๐ฆ
Jan. 2016 - Apr. 2016
-
Peer Tutor,
Calculus, Linear Algebra I, Linear Algebra II (UofT) ๐จ๐ฆ
Sept. 2015 - Apr. 2016
-
Teaching Volunteer,
Elementary School and Junior High School Mathematics and Physics, Qinyuan, Shanxi, China ๐จ๐ณ
2014
|
My Name
My first name is a combination of my mother's last name and "the Moon", which has the same pronunciation as "the Moon" in Chinese. My last name is "river". So my full name is "the Moon over the river".
In China, "the Moon over the river" is widely used in poems to express the emotion that although we cannot live together with our family, we can still share the same Moon at night, and the Moon over the river can always send the best wishes to the ones we love!
I hope all the families can have more chances to reunite. If it cannot be realized, I hope the Moon over the river can help us send the best wishes.
|
Acknowledgment
I appreciate all the people who have helped or encouraged me throughout my life! I am an unlucky but also super lucky person. I may have gone through a little bit more ups and downs in my life than most people, but I always received countless support from so many people, no matter what situation I was in.
My goal is to be a good mentor in the future since some unforgettable professors and mentors always supported me and served as examples for me.
Prof. Wolfgang Stuerzlinger is the most important person in my academic journey. I am grateful for the opportunity he gave me to work on exciting HCI projects, even though we had never met in person until two years later.
He was the person who transformed me from a novice student to a researcher. He continuously taught me the skills to conduct high-quality research and has been an unwavering source of encouragement and support both academically and emotionally throughout these years.
It was also an unforgettable moment when Wolfgang squatted down to my level the first time we met, as he is tall.
Together with Prof. Christof Lutteroth, they made a significant impact on my academic journey. The second project I worked on with them remains my favorite project to date. I thoroughly enjoyed the topic and the experience of working with both of them. Their support and encouragement meant a great deal to me and made me feel happy every day. I was inspired by their hard work and dedication, especially considering I was just a remote student.
Prof. Faith Ellen was my idol during my undergrad years. She encouraged me to speak English, as I almost could not speak English when I went to Canada, and let me fall in love with data structure and computer theory. She taught me that persistence is much more important than intelligence.
Prof. Gerald Penn was my undergraduate advisor in the Mathematical Linguistics group. He has some illusion about me that he often told other people that he would rank me as the best undergraduate student he has ever worked with. I really thank him for overpraising me.
I was a data structure TA under Prof. Sam Toueg and Prof. Francois Pitt. I appreciate them giving me the chance to teach students data structure proofs for two hours every week. I broke all the chalks in the classroom during my first time teaching since I was too nervous. I still remember Sam and I worked past midnight to do the grading.
My graduate advisors, Prof. Antti Oulasvirta and Prof. Matthias Zwicker have been instrumental in advancing my research and teaching abilities. Through their guidance and support, they have helped me to make meaningful progress in these areas.
|
|