Microsoft researchers unveil InvisibleMentor system for learning better workflows from screen recordings

Microsoft Research has published work on InvisibleMentor, a system that uses vision-language and language models to analyse screen recordings of users completing tasks. The researchers say it can identify repetitive or inefficient actions and recommend more effective workflows without requiring the user to first describe the problem in a prompt. The work was presented as research rather than a general consumer product launch.