From experiment to launch: how data shaped a new comments experience

Over the past two months, we’ve rolled out a number of updates to comments in Figma. As always, insights about our users and how they design together informed these improvements. The data science team spearheaded experiments to learn how to encourage feedback and collaboration directly in the file. Here, Data Scientist Emily Jia shares the hypotheses, tests, and learnings that lead to this cross-functional, data-driven product launch.

Before I joined Figma last year, the data science team started thinking about re-building comments and how we might use data to improve collaboration across teams. Making data-driven product decisions means testing and validating (or debunking!) hypotheses at different levels of scale—sometimes that means iterating quickly based on user insights, and other times it’s about conducting large-scale experiments.

Now that new and improved comments

are available to everyone, I’m excited to share how we got here. In this case, a series of smaller experiments ballooned into a full-fledged product launch. While this experiment was part of a larger redesign spanning product, marketing, and design, I’ll share the data science lens of bringing an updated comments experience to all of our users.

Unresolved comments

Two years ago, a working group across Figma decided to restructure the (now-updated) Starter plan

to help new users experience the full potential of multiplayer on our free tier. As part of this workstream, my teammates Clancy and Wendy started to think about experiences in Figma that unlock collaboration. They found that although multiplayer editing is the most apparent way to design together in Figma, engaging the entire team—editors and viewers alike—is key to helping a team grow with Figma.

The working group started pondering magic moments—those experiences in Figma that spark joy and bring more people into the design process. Since editors and viewers have access to share feedback on Figma files, we predicted that comments would be the front door to better collaboration across a team. While the data validated that comments are a strong indicator of team growth and engagement, we were surprised to learn that comments weren’t widely used. Knowing that giving and receiving feedback is central to design work, we ran a series of experiments to understand why the insights from the data didn’t match the user behavior we observed.

Our discoverability hypothesis

Previously, users entered comments mode by clicking on an icon in the upper left corner of the Figma editor. A series of research sessions showed that, though comments provided value, users were struggling to find it within the product.

To confirm whether low discoverability inhibited comments usage, we ran an experiment that prompted users to leave a comment. This initial test targeted developers with view-only access to a file—these users rely on close collaboration with designers in Figma, but were historically the least likely to interact with comments. After two weeks of experimentation at a 50/50 split, comment creation in the test group was 45% higher—without affecting the rate at which users returned to comments the following week. Simply making comments easier to discover had substantial impact on comments usage.

Unexpected results, and a path forward

After validating that comments were valuable but hard to find, we set out to help the product team find a more discoverable solution. For our second experiment, we thought that moving the entry point for comments from the left to the right side of the Figma menu bar would make it easier to find. While tools on the left offer core creation and canvas functionality for designers, we hypothesized that cross-functional partners might find themselves more frequently using the right side of the editor, where many of the collaboration and viewing features live. We ran the experiment with a 50/50 split on new users and tracked the percent of these users discovering comments within seven days of signing up. The results surprised us: we saw a 20% drop in comments discoverability across the board.

This experiment was a helpful reminder that small changes can have a meaningful impact on the user experience, and in fact, we should expect most of our product hypotheses to be wrong. In 2009, the experiment platform team at Microsoft revealed that around one third of ideas at Microsoft improve the metrics they were designed to improve, and other large tech companies report similar success rates. The best way to catch regressions is through extensive experimentation and testing. While the data demonstrated the opportunity in making comments more discoverable, our second experiment showed that fixing discoverability issues was not as simple as moving comments mode to a new location.

After continued experimentation (both across Figma users and teams at Figma!) we decided to launch a number of changes

, not only to the discoverability, but also to improve how users manage, sort, and interact with comments.

A new launch paradigm

As our product designer, Ryhan Hassan, iterated on comment designs, the data, product, and go-to-market teams kicked off a conversation about how to extend the data-driven approach to our launch strategy as Figma scales. To minimize disruption amidst a substantial UI refresh, we ultimately decided to incrementally release comments to our users, rolling out the updates over several weeks.

Meanwhile, the go-to-market teams would publicly announce the launch at the beginning of the rollout to give users a heads up that the updates were coming. This approach allowed us to quickly address any bugs or user experience issues before expanding to a broader population. We sized the rollout proportion to be large enough to detect statistically significant changes in user behavior, while also honoring our plan to make these changes incrementally.

Since comments are an inherently collaborative feature, we randomized the experiment on the Professional and Organization team levels—we wouldn’t want collaborators to have different commenting experiences! However, comments usage among teammates is correlated, so the experiment setup needed a larger test group than usual to maintain the same power.

Reflections

Overall, users responded positively to the new comments experience. We saw a significant lift in all users leaving comments, and this increase was especially pronounced for non-designers. This was the case internally as well—comments usage among Figma employees has 3x-ed since the beginning of 2021!

The data also indicated a more dynamic collaboration process. Not only were users who left comments more likely to get a response to their feedback, response time actually went down. While the median comment response time is 1-2 hours, the 75th percentile response times have decreased by a few hours. One of our goals is to make the design process more approachable to entire product teams with varying levels of design experience. By bringing comments into focus whenever a user opens a file, we’ve made comments a feature that encourages more voices in the design process.

For me, it has been rewarding to reflect on the influence that the data science team has on such foundational improvements to the Figma experience. In this case, our approach became a playbook that will guide major launches in the future, but our work goes beyond product strategy. From setting metric goals with leadership, to collaborating with partner teams on product and design decisions, I’m excited about all the ways our team can use data to help influence the future of design.

If you’re interested in helping us make data-driven product decisions, learn more about Figma and check out our open roles here.